If you 'mixed' the grout yourself, perhaps you didn't mix it properly? Or, the grout dried too fast? Or, you used the tub too soon and didn't let the appropriate drying time to take place.
If there's a 'problem' with scheduling the use of the bathroom, you could always send everyone away for a day to the in-laws, or something. In my house, I schedule this type of work (unless it's an emergency) for when the Mrs. and the 5 year old 'go away' (on a Mom+Daughter weekend, or something.) This way, I have plenty of time to 'screw up' if I need it.
Re-read the directions all the way around, and see if anything comes to mind. In any event, be sure they're followed, including the curing time. My best to ya and hope this helps.