First, vinyl siding is not recommended for use down south. It doesn't hold up well and that foam board insulation acts as a vapor barrier. Since down south we use no vapor barriers in our homes, the foam board traps moisture in the walls and leads to a multitude of problems. Not the least of which is rotting wood, mold/mildew growth in walls, sick building syndrome, and termites that are hard to eliminate.
If you're up north where it is common practice to install interior vapor barriers then you won't have those problems above. I agree with you, I'd remove at least the first layer of siding and take a look at what's behind it. For all you know you could have rotting wood behind the asphalt. Your trim will fit better, too. And the very slight amount of insulation that the asphalt layer provides would be insignificant.
One last thing - have you considered using a cement-based siding like Hardiplank? It should be much cheaper than vinyl, won't ever rot, burn, or be affected by insects. And it should hold a good paint job for 15-20 years.