Make sure you turn off the power to the fan. Then remove the decorative cover from the ceiling. There is usually a single set screw holding the entire fan and light assembly into the metal housing box. Undo that set screw (it's usually a silver hex-head), and drop the fan/light assembly out of the housing box (be careful of the wiring).
Where the ductwork comes into the fan housing, there SHOULD be a plastic piece that has a self-closing, one-way flap (exiting out of the box, of course). This is what should be preventing incoming air in the ductwork from entering your fan unit, and ultimately your room.
Sometimes, in the haste of installation, I have seen this one-way flap left off the housing. Of course, that results in a gaping hole that's simply allowing the air to come into the unit, out of your ceiling and into the room.
If the flap on your unit is missing, you can probably go on-line and order one from the manufacturer for your model. Of course, then you'll have to install it, which may mean a trip into the crawl space or attic, because it's attached from outside the housing.