Yes, it does sound ridiculous, but there is a good reason: balance. Lets look at a typical room, say, the one Iím sitting in right now. It has 3 exterior walls, with windows in each, with the fourth wall interior. All the ducts are exactly where you would predict; along the exterior walls under the windows.
Your reasoning for placing the ducts away from windows would be valid if, instead of windows and doors, I just had openings. If the ducts were under these openings, the air would just get sucked out.. But I do have windows and doors, and assuming they are fairly airtight, the actual air I pay to heat or cool remains in the house. What Iím actually loosing is the energy the air contains. Itís important to realize that point, and to also realize that we are going to loose this energy through windows and the like no matter where the ducts are.
Now weíre back in my room, and letís say itís winter time. Given what Iíve said, we would expect the areas of the room near the windows would loose heat faster. If the vents were located away from the windows, those places in the room would be much warmer than the window areas, so the temperature in the room overall, would feel uneven. If the whole house were set up as you suggest, the house would, in fact feel colder because only the center areas of the house would feel like the thermostat setting, and one would tend to turn up the heat to try to compensate. Generally, the house would tend to not feel comfortable.
So itís a matter of balance. Since windows and doors are an unavoidable, and since itís a fact that they loose energy, contractors place the vents there to achieve a more balanced room temperature. Likewise, in houses that have baseboard or radiator heat, the same rules apply., exterior walls and windows. The expense to heat or cool the house may be more, but the amount is insignificant in the long run, if the house feels comfortable.