That nifty little device is there to save your life. In the old days, water heaters had a nasty habit of BLOWING UP (ie right throught the roof of the house).
When you heat water, it expands. Pressure builds at different rates, and sometimes, builds faster than it can be absorbed by the system safely. The prv is supposed to be rated for just a little higher pressure than your water pressure, so that the tank doesn't turn into a boiler.
Unfortunately, sometimes you can get a little rock in the valve, or sediment, etc. And, they can wear out.
If you adjust the water pressure in your home above the pressure rating on the valve, then it will leak all the time. Hopefully, yours is vented to the outside.
If yours is leaking, try opening it up and letting it snap back. Usually, that will do the trick. If the leak won't go away, get a plumber to replace it. (about $85)