The picture is clearer here. He should not have anything outside the setback area, the county right of way, and your property. He is clearly in violation of subdivision regulations regarding setbacks and is trespassing on county and your property.
The right of way is the same as for a city street. For example, a city will get a typical right of way of 60' for a residential street (collector and arterial streets have more for extra lanes, etc.) which they then put in a street of let's say (they vary) 31' back of curb to back of curb(aka "back to back")(or may be identified by road width of 29' or 30' without adding in the curbs). The remaining 29' in this example ends up being part of the front yards on both sides. They try to run the street down the middle but sometimes they move over and all they actually have to be is inside the right of way. Assuming in this example that the street is in the middle, 14.5' of a person's front yard on either side actually belongs to the city even though the homeowner mows it and is required to mow it and take care of it. Even so, it is still city property although some people believe they own to the road. The homeowners property pins should be the 14.5' back from the street in this example. The setback then starts from the property line and nothing is allowed inside the setback so people don't build right up to the property line. They also have side yard and rear yard setbacks. A property may also have a utility easement which is not supposed to be built on either which are also identified on plats. Just a little subdivision lesson.
Your neighbor is in open and notorious use of yours and the county property. If you don't contest it, then he has a shot at getting it by adverse possession. See if the county will force him to move. A visit from the sheriff may help on a possible criminal trespass violation. You and the county can file a trespass action asking for an order for him to remove his property and compensate you for him using it. Where I'm at, the local governments don't have a staff attorney but use local attorneys on an hourly rate which means they are less inclined to get involved in expensive litigation. If they won't do it, maybe they will do a cost share with you on it.
Contact the county zoning and subdivision administrator, file a criminal trespass complaint with the sheriff (after sending a letter to your neighbor to remove his stuff from your property because he doesn't have your permission to have it there). Ask the county commission to do the same and see about sharing legal costs if they won't do anything on their own.
Forget about being neighborly cause if he was such a good neighbor himself, he wouldn't be on your property as well as the county's and wouldn't be aggressive/uncooperative. Sounds like he is already a sour individual so the relationship is already sour and indecent. I assume you did the very first and neighborly thing with an introduction with him(not his wife), hand shake, invitation to coffee, other friendly gestures, and a visit about beautifying the neighborhood and that you moved in and the survey showed he was on your property and you would sure help him move it (inside the sideyard setback), etc. If your friendly gestures are rebuffed then you know where you stand and make your demand to stop trespassing along with just compensation for using it(maybe the compensation will cover your legal fees and maybe you can get them anyway with an intentional tort of trespass). Good luck.