Jim isolated the problem.
to test the theory, open the damper and get a small flashlight, turn it one and place it on the smoke shelf or up and over the damper.
Close the damper and see how much light you can see from the cracks in the damper.
you will of course see some, these areas are not as tight as they should be, but you should not see allot. The more light you see the more wind that will come in.
Get a putty scraper and scrape the damper works. This should help. Be ware, this is a very dirty job.
A chimney sweep will do a bang up job for about $40.
Check it out.
© 2016 Renovate Your World LLC