I wouldn't call 3-5" small because if there's enough hydrostatic pressure, you could get some water in there!
This may sound silly but if you can ctc the previous owners to find out if they, or the owners previous to them, know anything about the holes, do so. Otherwise, who knows. The holes may have been used for, say, an indoor clothes line pole!? I don't know. Maybe someone had nothing else better to do. IF, and I say if, the holes are NOT symetrical, I think you can fill them w/o any problem. You'll need a hydraulic / expanding cement. You'll need to chisel out the inside of the hole a little so the cement has something to 'grip' on to. (Wear eye protection and gloves when you work in the hole and with the cement.)
Here's a link to help expand your knowledge. Hydraulic Cement. There are others. Be sure to read the packaging AT THE STORE, and BEFORE you cut into it at home. You want to be sure it's for your situation. And get the right tools while you're at the store.