If your siding is a DARK color, you have to be careful that whatever you use you DON'T rub so hard, or use the wrong 'stuff', that the siding is ruined. The next thing you know you'll be telling us what happened and now you want to PAINT it. Soooo, tread carefully, and try your method in the inconspicious places FIRST.
Get some cotton balls and try some Skin So Soft. Again, rub gently, then when you're finished, wash the affected area with mild dish detergent (NOT dishwasher or laundry detergent. You know, use Joy, Dove, Palmolive ...) If SSS doesn't work, try a little Automobile compound on a cotton cloth. Again, rub gently in a circular motion. Wash and rinse as I stated when you're finished.
If these 2 methods don't work, call the Mfgr.'s phone number on the can for help. I guess next time, you'll know to 'cover' the wall with plastic before you spray. Oh, not that you need to be 'beaten up' any more, be sure you don't spray on a breezy/windy day because, even if the wall is covered, the stain will get carried down wind. I know a guy that did 'all the right things' until he saw his car in the driveway (about 20 yards away) had a thin film on it from the blown deck stain!!!