It is unusual that a gas-fired boiler has a tanbkless coil option, so it is likely that the oil boiler was converted to a gas conversion burner. When converted, the burner may have been sized to serve the heat load of the house and not the hot water load. When converted to gas such boilers lose the radiant heat of the oil fire and depend upon the burned gas in the passages, so the boilers lose some of the capacity to heat and burner input is reduced.
The smallest tankless coils are about 2-1/2 gallon per minute capacity. This means they have to have at least 125,000 btuh input minimum to serve that much hot water. If the burner input is reduced to half that to serve a large house for heating, the boiler runs out of hot water quickly even if the water temperature is kept to at least 180F by the controls. The answer is to increase the input to the amount needed by the hot water, not the house,