I have a musty smell too... in fact it has gotten so bad that my clothes are smelling!!
I don't have a crawl space as I am on a slab foundation. I haven't seen any leaks in the walls either upstairs or downstairs and the heaviest concentration seems to be near my downstairs bedroom.
A couple months back I had a company come out and professionally clean my vents and they were HORRIBLE - however that didn't solve the problem. Now I'm trying to figure it out and then about to throw out my wardrobe.