You didn't say what you're removing it from. Concrete? Hardwood? Sub-flooring? Vinyl tile? Lauan???
Depending, try an Ice Chopper. Yes, the kind you use to remove ice from the ground outside in Winter. (That's assuming you live where there is a use for such a tool ...) You can try a Border Spade Shovel too.
In a job I did, I actually removed the Lauan Underlayment with the Parquett still attached! It was quicker, cheaper (in labor costs), and just as well, to do it that way. If I didn't, at worst, the glue was STILL 'stuck' to the underlayment. Unless you could smooth that out, you might as well remove the underlayment. You may find that to be the case in your situation too.