The drip edge is very cheap and if you have to remove it, then so be it. the gutters need to be only cradled by a set of slings attached to the soffetts that will keep from bowing the gutters and only get them out of the way until you repair the facia.
Remove the bad facia with a pry bar and either replace the entire board from end to the boards end or cut a 20 degree angle cut over the rafter tail so that you can angle attach the new facia there at a tail. You might have to bend the drip edge up a bit to remove the facia, but it will bend back down and behind the gutter anyway.
I always find it much easier to remove facia from cut to cut even if you remove good facia. That angle cut is hard to make when its attached to the tails. Usually a good board will be about 12 or 16 foot in length. Just replace it all instead of two or three foot sections. Be sure to caulk it up good and prime it before painting.