Joy......in reading your post - I went back some -and saw some scary things - so I replied to them - please read what follows:
Joy your particular issue is likely to be just what happens to oil paint. Your contractor was partially correct that oil is more durable. It gets very hard (oil paints cure by oxidation - so they cure indefinitely - as long as they are exposed to oxygen) - so it is very scrubbable and scratch resistant - HOWEVER - this also makes it brittle - much more likely to chip and crack than latex paints - which are more flexible.
I would clean you trim - sand it lightly. Dust off and apply one coat of Glidden Ultra-Hide Gripper - and two coats of Glidden Ultra-Hide/Evermore or Ralph Lauren Satin or Semi-Gloss Latex finish topcoat - colour of your choice.
These products are available at The Home Depot!
PEOPLE! I have read some scary things on here in regards to latex topcoats and oil primer .....here is how it works for decorative house paints - interior or exterior.
It depends more on what you are priming than what you are topcoating it with.
Oil primers are designed for wood, metal, or stain/odour sealing only.
Wood primers - oil/alkyd undercoaters they are sometimes called - are mainly used on wood. They are fairly slow drying (12-24 hr) and are great for raw wood and hard to stick to previously painted surfaces - like Calsomine ceilings. Their binder is oil/alkyd - the binder is the part of the paint that is left on the wall after it is dry - there is no "oil" left behind that won't mix with latex, or will fight with an oil? That makes no sense at all???
These can be topcoated with a latex or an oil finish coat.
Alkyd metal primers are designed for ferrous (rusting) metals - they cannot be used on galvanized - only latex there. These are usually relatively fast drying (2-6) hours - and generally can be agian topcoated with latex or oil finish coats.
Stain/Odour sealing Primers - B-I-N and Kilz are just two examples of these. These are specialty primers that are designed only for sealing stains and odours. They dry very quickly (1 hr recoat) and they dry very hard. They cannot be used on large surfaces outside - they dry so hard that they are not flexible to use over large surfaces outside.
These can be topcoated with latex or oil finish coats as well.
HOWEVER - These types of products do have what is called a "critical recoat time" - this means that they must be topcoated - with either latex or oil - within a certain time frame (usually 24 hours) or they become too hard for EITHER TYPE OF TOPCOAT TO ADHERE TOO!
Usually the only way around this is to sand the primer - recoat it with itself - and then topcoat again before the critical recoat time is past.
This has been a really long explanation - it could have been even longer.........bottom line is - READ THE BACK OF THE CAN!! The manufacturer of the primer knows what can go over it!