Very good that you removed the wallpaper. Sorry to hear that you inadvertently removed some drywall paper too. You seem to have the latter part under control.
Just for your benefit and for that of others, before you PRIME and PAINT, you want to wash the walls of the glue. DIF by Zinsser is good for this. After this, go ahead and use Joint Compound to smooth out an areas in need of this, especially the areas where the drywall paper came off. Then, you need to sand and respackle if necessary. There's no need to remove the latex paint unless it's peeling off by hand. If you're thinking/planning to sand off the Latex paint, there's no need for that. In other words, if the paint is REALLY stuck on there, leave it alone. Wherever there are any 'imperfections', say ... where you removed a piece of loose paint, just 'fix' that area with Joint Compound (and sand and respackle) until it's smooth with the 'surrounding' wall. The end result you want is simply a smooth wall. It doesn't all have to be devoid of latex paint (again, unless the paint is loose.)
Once you're satisfied with the repairs, apply 2 coats of a GOOD Latex Primer/Sealer. Look in the YELLOW PAGES under PAINT - RETAIL for a store. (Remember, you get what you pay for.) I recommend 2 coats because the 1st coat will DEFINITELY be absorbed by the dry Joint Compound. The 2nd coat is insurance. After that, go ahead and apply your topcoat. Here, too, I suggest 2 coats. While you're at the Paint Retail Store, feel free to describe to them what you're 'up against' at home. They'll be sure to give you the right paint Supplies too. Again, you get what you pay for. Spend some of that hard-saved $$$ on good material and tools. If you don't, you may be back at this in as little as a few years!
My best to ya and hope this helps.
Jay J -Moderator
PS: Oh, there is no 'drawback' to leaving areas of 'old paint' on the wall (as long as it's not loose and falling/peeling off.) It's actually a benefit! I'd lightly sand the wall too before you paint since there may be some 'paper fibers' on the wall from the already removed wallpaper. One Trick: To see how well you've done BEFORE you PRIME, turn off all the lights and hold a flashlight against the wall whereas the light is directed parallel to the wall. NOT perpendicular; parallel. You'll see all the 'bumps and bruises' that way! Since in this trick light travels in a straight line, don't get too hung up on getting the job perfect. If you are endlessly trying, you'll never get to PRIME!