You either have to take off the jam on both sides of the pocket or have to have access to the rollers/tracking 'outside' of the pocket.
Newer doors can be 'lifted' off the tracking after a minor 'unadjustment' of the door. Then, it's EASY to replace the worn wheel. If your door isn't of this type, again, the jam needs to be taken off so you can slide the door off of the tracking. My 'newest' pocket is only 6 years old, and if I need to do anything to it, I have to take off the jam.
Sorry, it's one way or the other. My best to ya and hope this helps.