Actually, I wouldn't be worried about the asbestos in the drywall itself. I work for BPB, the world's largest manufacturer of gypsum products, and can tell you that it is highly unlikely that there is asbestos in any of the drywall you've ever encountered. Asbestos WAS common in joint compound and fillers for a while, but hasn't been in there for about 25 years.
The fibres you have seen in the board you removed could be one of a number of other things. We commonly add clay, strands of fibreglass, and even paper fibre to core mixture, depending on the product. The most visible of these is fibreglass, and I'll wager that's what you've encountered.