the plastic spacers are meant to be left in place. As long as the grout covered them up completely, you should be ok. You didn't say how thick the tiles were. Since this is not a floor, and there is no significant loading. The % area the spacers take up compared to the total area the grout takes up is insignificant. Also, sanded grout is much stronger than unsanded. No Problem!