... If you're referring to the hardwood, itself, being bumpy, you have an aestetic 'problem'. If you're referring to the poly being bumpy, you want to get it level. You should screen it until you get it as level as you can.
Start with an 80 grit screen followed by 100 grit. I say it's better then a small, sanding tool. Be sure to vacuum up between each screening. As you're already aware, screen between each coat of poly you apply with a 180, or 220, grit screen. Then, vacuum. If you're using a water-based poly, you might notice that when you screen, there's a propensity for the poly to 'gum up' on the screen. Clean the screen every 5, or so, minutes.