I hear this frustration from customers ALL the time. This is not the practice of ALL contractors, but so many are this ill mannered that it has given the entire profession this hard to shake stigma.
There are contractors that are out there that do respect your time and do understand that this is a big enconvience to you and you do want this fixed as quick as possible and correct the first time.
The best way to find a good contractor is by word of mouth or from friends that have had good experiences. BBB is another good outlet.
But, promptness and on time is something that all contractors run into problems with. They could be coming from another job and believe me when a contractor is in a home, the homeowner will inindate him/her with all sorts of other projects other than what he/she came there for originally. Sometime its really hard to leave a customer when you need to to make the next meeting on time.
This may sound rude and I don't want you to feel that way, but have some slack or slide time ready it he/she is not there promptly. The worst thing that you want to do is to make a contractor upset about promptness when he/she is working on your home. You also need to be extra flexible too.