1) baseboards, as LJ stated, this would be your best choice. Although I HATE with a passion that spray foam. Its messy, it will never come off your hands and it does get away from you when using. But, its the best cure for your situation.
The drywallers knew that base molding would cover that small area so they stopped there. It would be terribly difficult to cut a two inch piece of rock for that small area.
I would carefully use the foam and control its spray into the area. Or go to a home center and get that backer rods or cords to push in there. This is foam that comes in three different diamenters that you just push into the area and it will definately do the job and NO MESS. Backer Rods come in a plastic bag all currled up, very cheap and you need the one inch size. Where the void is larger, use two.
2) Windows, I always caulk and weather strip and all from the outside first. This is your main line of defense. If your leak is still severe after a thorough caulking from the exterior and you still have air coming in, then remove the trim around the window from the inside and re insulate and place it thoroughly. Seal it back up.