Air being sucked down the chimney may be a sign of air being sucked out in another area. Probably the first thing to check is your air management system (Heating & A/C). If the system is out of balence, it could be creating a negative pressure by the fireplace, thus creating a downdraft. If there is sufficient air leakage out of your thermal envelope, this could also be creating such a downdraft. If there is insufficient allowance for make-up air for all exhaust appliances (Bath & kitchen fans; dryer; furnace & waterheater drawing it's combustion air from the house), the same situation can exist.
Be mindful of any other barametric venting appliances (waterheater; furnace). If a backdraft can be created in one place, it can happen in another.