Dave, you are on the right track--sort of. The problem with asbestos is asbestosis, where the fibers fill up the small spaces in the lungs and you can't breathe. There probably is some potential of cancer but asbestosis was a much more common problem with asbestos workers. In any case, it's not good to breathe the particles and old, deteriorating heating and piping insulation in old houses and other buildings is a prime source since the asbestos is not bound up with other materials. This, in my opinion, must be dealt with immediately when it is found.
Since the asbestos in shingles, tile or tile cements and other solid materials is fairly well encapsulated, it definately should be removed and the removal done in such a way as to minimize the generation of dust. But the total volume of asbestos in these is much less that the pure asbestos, usually found in pretty deteriorated condition, in old piping and heating duct insulation. You can just look at some of this stuff and the dust falls off.
But I do agree, if there is a chance that you have asbestos, read the EPA reports and your state regulations on how to get rid of it and get rid of it according to the regulations, no matter how much it costs.