This is interesting. The house is on a slab, and you didn't mention anything about having a humidifier or dehumidifier, and you didn't mention anything about bathroom humidity or 'cooking/boiling' humidity, and you didn't mention anything about the dryer vent, well ..., I have to assume that the house is finally showing signs of 'tightness' now that you see the mold / mildew.
Just as a test, get yourself a squirt bottle or something to 'spray' on the wall and make a mixture of 1 cup household bleach with 1 gallon of water. If you want, you can go heavy on the bleach. (Obviously, you don't need this much of a mixture but it's just the ratio to use.) Anyways, spray some on the mold/mildew. If the mold/mildew turns 'clear' after, say, 5 minutes, it's mold/mildew. If it doesn't change color, then it's dirt. This is simply a TEST that you want to do to be SURE you're attacking the right problem. You may be sure it's not dirt but I'd want to be ABSOLUTELY sure it's not dirt.
OK - Where does that leave us? W/O pulling off the drywall to see what's behind the wall, I guess you'll never know if there's a moisture problem behind the wall. However, it's unlikely that there's water back there because unless the drywall is soft and 'mushy', then I'd have to figure it's not wet back there. If it's dry back there, you wouldn't see any growth AND your drywall would be 'soft'. Remember, in order for mold and mildew to grow, there needs to be 1) a CONSTANT souce of moisture (ie., humidity), AND 2) there needs to be 70+ temperature (optimally, although it can 'thrive' at lower temps but it does so at a slooooooow rate.) If you can reduce either or both of these 2 conditions, then you might solve your problem. If the carpet/floor isn't wet (which you didn't say you checked) and your drywall isn't 'mushy', then the source of the humidity has to be coming from inside the house itself.
One way to test out whatever you do is to 1) clean the 'affected' area(s), then 2) set up a temporary dehumidifier in the area(s) in question. IF, after cleaning everything up and running the dehumidifier for a while, you don't see it come back, then you know that the room is, basically, very high in humidity. Also, you didn't mention if you have a ventless heating system. Maybe you could re-read my/our past Follow-ups and individually 'address' each 'item' mentioned and perhaps say something about it. (Like, "No, we don't have that." Or, "Yes, but that was fixed 3 years ago." ANd so on, kinda like you mentioned about the windows.) Tell us more.