This is the most common cause of water dripping back into the bathroom.
Insulate your ductwork and see what happens. Right now, warm moist air is condensing-out on the cold, dry ductwork. You might want to consider replacing the ductwork first since the one you have now may be a 'mess' inside. Some folks are adverse to using the pre-insulated flexible ductwork for this application but I'm not. Since your run is short, you should be just fine. AT worst, use the rigid ductwork and insulate.