In some homes, this type of 'damage' is Chalked Off As Character. In others, it's considered damage. So whether to repair it or not is your call.
If the damage is from scratches and not gouges, then perhaps a deep sanding is the answer. BUT, you may already be at the point where if you continue to sand, you'll hit the nails. If you had replacement boards, then you could refinish them and replace the gouged ones. Like you said, I wouldn't fill the gouges. As far as leaving them alone and pollying, that's what I'd do but at the same time, I'd poly the damaged board/area by hand making sure the poly doesn't pool in the gouge.
Now, if the gouge is 1/8" wide or less, then I don't see the need to do anything. If it's wider, then poly by hand when you come upon it.