My suggestions are purely speculative, since I cannot see the area of concern.
I paint many homes built in the 50's that used the siding called R-105 or tounge and scoop, or ship lap. In frequent cases, the paint just peels off this wood. Even after many years.
The newer wood does this on a regular basis.
I was told by a old timer painter whom I admire for his knowledge and panacea for problems with paint. He contends that this lumber has not properly dried and continues to leach sap and liquids which cause the paint and primer and just everything to peel off.
How new or old is your freeze board or facia board? Do you feel and rosin or sap on the board? Is the paint, prior to painting, bubble up and fester up before peeling?
I would suggest to wipe down the sanded wood that is down to the bare wood with a cleaning fluid and then prime with a good oil based primer.
Not to offend you in any way, but 95% of paint failure is lack of proper prep.
Then apply a good (big 3) brands on the surface. use an exterior latex.