The longer it sits, the more difficult it will be to remove. If it's not too bad there are products designed specifically to remove the haze caused from a poor grout application. Some are solutions designed to dissolve the haze, some are mechanical and literally scrub it away.
Again...the longer it sits, the more difficult it is to remove. The last resort is to use an acid solution to remove it, which is not good for the grout.