What a lot of people aren't aware of is the fact that the ground moves. Usually no more than a millimeter or so a year, but it definatly moves. Look at an old fence line sometime, I assure you that, as a person who spent his high school years running fences (not by choice), fences are usually straight as arrows when they're new.Old ones usually look like someone was drunk.
Steve is probably right on the money here. The combination of moving ground, fluxuations in temperature and humidity are exagerated by a bad foundation. If you have a basement or crawl space, take a flashlight and inspect closely around the perimiter of the basement or crawl space. Your looking for cracks in the floor of the slab or, in the case of a basement, in the walls. Cracks are normal. Cracks that do not line up flush to one another are not. It would probably behoove you to seek the advice of a mason.