My suggestion is to first clean the mildew off the walls. There are a variety of solutions for this so I'll leave them up to you to investigate. Anyway, there are a couple of approaches.
One, you need to seal the stain to the wall once it's been cleaned up. You can use a number of stain sealers for this task. A product such as Stain Kilz is an oil-based primer+sealer. It's VERY toxic, and if you're working in a close, you could get severe headaches. So, if you're gonna use this stuff, place a fan in the window and have it SUCK OUT the air in the home. If you have the fan operating in the opposite manner, it will blow all over the house. I like it in spite of its toxicity because it does the job. You should go to a reputable paint store such as MAB, Sherwin Williams, Finneran & Haley, Duron, just to name a few. There, you'll get comporables to Stain Kilz, and this leads me to my next approach.
Two, once the stain is sealed, you need to paint. Don't skimp on cheap paint. Go to the aforementioned type of store and pay $20, give or take, per gallon. You get what you pay for. Make sure you get a paint that has a mildewcide mixed right in it. Since your closet already has had a mildew problem, you need a paint that resists this. Plus, it makes it easier to clean should the problem begin again. One thing I've failed to mention is that you need to be sure that your closet walls (et al.) are 'sound'. If they're damaged, you ought to correct them. I'd put the fans in first to see how much of the problem is alleviated. Proper fan sizing and venting to the outside is paramount. Measure the L x W x H of your bathroom(s) and take this info to the store w/you when you go to buy the fans. I suggest a fan that operates with at least humidity controls. I'm gonna comment beyond the scope of your post but you didn't mention where you're running the ductwork (if any) for your bathroom fans. If it's into the attic, make sure you end up outside, and it's properly insulated. Otherwise, you'll end up with a new set of problems with moisture!
Other than that, I don't have anything else to suggest.