Have you considered the possibility of having an insulating company inject the walls with a low expansion type foam? It needs to be done carefully to prevent bowing of the walls, but would be very effective. If you consider this, find a reputable company to do the job.
Otherwise, I would pull off the drywall as you suggest, install R13 fiberglass unfaced batts, cover the walls with 6 mil plastic, and put up new drywall. The R13 fiberglass will give slightly less insulation than rigid foam, but would be MUCH easier to install. Either way, you will see a tremendous difference in how the room feels.