but, I once had to remove brown cork board to a wall that a previous owner had installed to hang stuff and really looked dark and dingy.
the new owners of course did not like it. The previous owners used liquid nails to adher the stuff to the walls. It would not come off of course. I had to scrape off the big globs and re hang new rock. what a mess.
In answer to your question, depends on the type of glue used. What ever you do, do not scrape off the outer skin of the rock or you will have to re hang also. Remember anything too abrasive or too chemical will only compromise the drywall's ability to hold paint and or texture.