How about installing a 6" sleeve over your 4" pipe? I'm visualizing a chimney vent, where there is an inner and an outer tube. If you protect the 4" with a larger sleeve, I would say of either PVC or tin, I would guess you could insulate right up to it. That's the theory around insulated recessed light cans, too. Just a thought and certainly not an educated recommendation. Good luck.