I don't know how practical it would be to try to limit yourself to "household" items. How did those items get into the household in the first place? They were bought at a store. The only thing I could think of to use that you might have lying around is: A. run them through the dishwasher, it might work; or B. Try turpentine or any other paint thinner you have lying around; C. maybe lemon juice?
The stuff you need to buy at the store that will wisk off the labels is "Goof-Off" or "OOPS!". One of these will definitely take the labels right off. Different labels have different adhesives, so it is impossible to be sure which one will positively work. I have never found a label that didn't submit to one of these special purpose solvent mixtures. You can find either one, or similar at hardware stores for a couple bucks.