IMO, I would. If you don't, you'll see any imperfections left on the wall from the glue. It may not be too noticible until it's too late. You may simply see 'swirling' marks or even 'lumps'.
Washing the walls isn't too hard. You can buy the 2 foot wide plastic that's in the paint isle in the Home Centers. It's a few $$$. Measure the perimeter of the rooms and buy that much plastic stripping (not the sheet plastic). Then put down a layer or 2 of newspaper on top. Then buy and use a product called DIF to wash the walls. I stronly urge you to read the directions on the container 1) to see how much you need, and 2) if you don't follow them right, it won't do the job. If I'm not mistaken, keeping the water boiling hot is the key. I used plastic gloves and one of those big sponges that you use to wash a car (buy a NEW sponge). The stuff works great! Give it time to do its job as the directions say. You can work ahead, and come back to 'rinse'. (Again, read the bottle in the store.) There is a home remedy that uses vinigar but I don't know the mixture nor how well it works. (I'll leave that up to you.)