let the ivy die in the sun, remove dried twigs with a pressure washer, then try muratic acid on the brick to remove any organic fibers left behind (mild solution of acid at first, then try stronger mixes). That should do it, just guessing though, heh heh heh. No warrantee implied or expressed. heh heh heh.
IMO, Ivy looks great on brick buildings, you can't live with the bugs and potential damage, right?