Joe is totally right that there are many posts on this subject, so many probably that you may have a problem figuring out which one is correct for your particular problem. You are lucky that you planned to refinish the floor anyway.
Your home center should have a scraper with a sharp, flat edge on the front that will lift most of the glue and rubber. Use this somewhat carefully since you do not want to put deep scratches or gouges in the floor. Since you already planned on sanding, this will make the scraping easier. The floor sander will remove any moderate amounts that you missed, but at the price of using more sanding belts.
It is ALWAYS worth it to get the article in Family Handyman that is referenced in the earlier posts. The usual safety warnings for this type of project apply. You should see many of those as you look back. Another way to check the index for possible answers is to click on Edit, probably near the top left of any page and then click on F.