Now that you're talking about the inside, a few new possibilities come to mind.
Yes, a damp crawl space could be the source of your problems. Is the rug rotting anywhere in the room that has the mold? If you have or don't have a rug, you can lay a piece of plastic down on the floor in the area where the mold is. If in a couple of days or so you see condensation on the underside of the plastic, you have a moisture leak from the crawlspace. When you lay the plastic down, use as large a piece as possible because the moisture may not come up right where the mold is. Also, do it right when it rains. If you have access in the crawl space, lay a large sheet in there too to see if there's moisture accumulating on the underside. CHeck for rotting on the floor joists. Again, if moisture is collecting on the underside of the plastic either inside the house or in the crawlspace, you've got moisture. You need to vent the crawlspace, and there's a formula for calculating how much venting you need. If you have a dirt crawlspace, it's highly probably that your moisture is coming right on up. There are ways to mitigate this. Before I or anyone else gets too far, let me (us) know about the moisture test.