I would not use the pry bar. Most sliding doors are aluminum and the ridge that the door slides on is also aluminum. A pry bar will and could bend this ridge or edge and cause more problems.
The door should just lift up and over with very little effort. Make your plans and have a place ready to lay the door upon its removal.
Many times the wheel has not damaged, it just needs re adjusting so that more pressure is exerted on the track. This can be done by adjusting the screw to do this. Then a good cleaning of the track with WD-40 and steel wool and a vacume will do the trick without the removal.
Try this before you remove the door and see what happens.