First off, if the 'specs' were dirt, the power washer would have removed them right away. If it's mold/mildes, then you'll need more than a power washing.
I see you tried Clorox. Have you tried using it w/a scrubbing brush??? The vinyl has 'grooves' in it so you'll need something w/bristles to get in there. Use either a nylon or thatch bristled brush. (No metal or otherwise ...)
Now, wash from the BOTTOM, up; NOT from the top, down. Rinse the ENTIRE 'verticle sections' you've already washed + the section you just washed. The reason for this approach is to avoid streaking. You can't streak a 'clean wall' with cleaner. DO realize that the stronger the mix, the MORE 'hazardous' it will be to your plants and grass and such. Fall is a better time to clean because the grass is going dormant. You should still saturate the ground after cleaning to dilute the Clorox.
Make sure you're using gloves too. If this doesn't work, visit the Home Center and pay $4.50 for some Vinyl Siding Cleaner. I used it and a bristle brush to clean about 8 squares of siding by hand. It took about 1 hour. That stuff should DEFINITELY be used w/gloves. Read the Instructions at the store so you have everything you need. All you do is rinse. (I didn't use it w/a power washer. Again, I washed and rinsed by hand ...)
As an aside, mold/mildew is most prevelant on the NORTH side of the house. It's the shadiest and 'dampest' side. Trim bushes and trees and plants BACK AWAY from the foundation to allow air back there. This will help w/drying up any excess moisture.