If the drywall's paper is coming off, that could spell problems. If you don't have a smooth surface, you're gonna see the 'mistakes' that were left behind. You need to either 'cut out' the damaged paper, and skim-coat some 'mud' on the wall, or remove the sheet and re-tape, and go from there. If you're not comfortable doing this, you might want to hire a Pro to do the compounding and sanding. If you do the work, when you think your finished, get the room as dark as you can, and hold a light along the wall and let the 'shadows' tell you how well you did. This way, you'll see what mistakes are left behind that need more fixing. (YOu can do this after each layer dries ...)
As for the part where you have glue left on the wall, see this recent post. Removing wallpaper glue - It will talk about removing wallpaper glue.
My best to ya and hope this helps. Oh, if you didn't use the DIF solution (or something along the same lines), then that might explain why the drywall paper is coming off!