First off, whatever you try should be tested first to make sure that marker is all it removes.
Lacquer thinner will remove it, but I'm not certain that it won't damage the ladder in the process. If you have nail polish in the house, try that first. Lighter Fluid (the kind for zippo lighters, not charcoal) may also work. There are latex paint spatter removers that I'm fairly certain will remove it as well, and probably wont damage anything.