Real Estate agents never see any problems, that's not what they get paid to do!
Most Home inspectors do not have to pass any test or be licenced. So, their knowledge of this problem may be zero.
I agree with Harold H., this is a problem.
Think of the chimney as a funnel. Water that gets behind the chimney, will rot out the house.
How long has this been exposed? Out 2 to 3", TOO long!
What kind of pest do you think would like to live in a place that is dark, wet, and has rotting wood? Can you say Termites? Sure you can.
If you truly love the rest of the house, I would have a brick mason or chimney service give you an estimate to repair it, (take it down and rebuild it on a stabilized foundation), and get a rough quote from a carpenter on what you can expect to pay to repair the rot, and finaly, get the price of pest control. Have it deducted from the sale price or move on to the next house.
Remember unless you are using a buyers broker, All agents work for the seller. If you are working with an agent and he/she shows you a house listed by another agent, your agent is still working for the seller, not you!
Also, Home inspectors that find problems that messes up a sale, don't get referals from agents. If this inspector did not see this as a problem, what else was not a problem? And what else did the inspector not even see?
I know there are good home inspectors out there but, this is not one of them!
It raises my blood pressure when they call themseves inspectors. They don't inspect for any Building, Plumbing, Electrical, or Life Safety Codes. They should called "Just My Opinons".
Phew... I'm sorry, I'm better now!
I would move on to the next house with a different broker and look for a new, "Just My Opinion" person. (ask the local Building official if you can get three names of people that know somthing.)