You need a grout tool. It's a hand-held 'file' of sorts that you run in between the tiles in the grout to remove it. Once you've taken out the grout with the tool, you should be OK to knock out the bad tiles. Even though the tile is coming out in pieces, at least it's coming out and it won't affect the surrounding tiles. Just tap lightly and be careful to angle the chisel AWAY from the good tiles. In other words, don't chisel towards the good tiles, and tap lightly. It's good to hear that they're on as well as you say.