One of the biggest problems is how to conceal the 1/8" - 1/4" gaps all the way around the door edge. I have done this in some really high end houses by creating a panelized trim detail where there are 1/8" gaps between 1x trim "styles and rails" throughout the room, including one horizontal one going all the way around the room at the height of the top of the door. This gets pretty elaborate though and is next to impossible to describe in writing.
Another way to hide the door would be to create an opening in the wall that you could cover with something, like a mirror that was mounted to the door. I would carry the baseboard across the door opening so that you would have to step up, over it. Otherwise, you have obvious seams in your baseboard. Then I would mount the door using hidden hinges, so that the door opened toward you, into the main room. Use these hinges:
The big ones that measure 1'' Wide x 4-5/8'' Long. I would put regular door casing (or even some nicer picture framing) on the door itself, just barely overhanging all sides so that it covers the gap all the way around the door. The drywall on the wall surrounding the opening would have to be finished smoothly right up to the edge of the opening. You might want to use corner bead where one side is nailed right onto the framing, inside the opening. This way the edge of the drywall won't get all ratty. So, you would end up with what looks like a mirror that is "picture framed". Now the only problem is having something to grab onto to be able to pull the door open. You could carve out a depression in the top edge of the top piece of casing sort of like a finger pull. Or, you could fashion a piece of stiff, thin wire as a "grabber" tool. To secure the door in place I would use a magnetic lock: http://www.rockler.com/findit.cfm?page=1611 and/or bullet catches: http://www.rockler.com/findit.cfm?page=146
With the magnetic lock the total door thickness needs to be between 3/4" and 1 1/8" thick so you might need to carve out the back to make part of the swinging edge of the doo thinner where the lock is mounted. These things aren't terribly strong, but you can't see them.
Other random thoughts... If the closet is deep and you can settle for only hiding part of it.. Leave it "as is", so that if you open the closet door you see a shallow closet jammed with clothes etc. Then, have a false back on it with an access panel. The panel would be hidden by the clothes.