You never said what was up with the other end of the 4-wire in the switch box. You described white tied to white (fine) ground to ground (fine) and black (from the line) going to the switch, and then to the red in the load wire. What you didn't mention is where is the black wire from the load?
It sounds like you need to have (at the fan) white white, green green, red blue, and black black.
and (at the switch) white white green green black (line) to wire nut black (4-wire) to wire nut red or black pigtail from wire nut to switch (line) red (4-wire) to switch (load)
You have to allow power to the fan all the time, so it has to have a return path back to ground. It doesn't sound like black (4-wire) is connected to anything on the switch end.
I'm writing this at midnight, and my advice is free. I am not a licensed electrician, so follow this at your own risk. Do some more research before you "charge" ahead.