I finished off a walk out basement in a former house and had the same problem. Windows and doors are mounted to the outside of an 8" concrete block, and I am adding a 2x4 wall on top. My solution was to frame around the window and door openings and run drywall out to the windows. It looks a lot like a 12" shaft to a sky light, but if the windows are large, it is not so noticeable. I built sills with hardwood plywood with a screen bead on the exposed edge.
It will be interesting to hear how others have dealt with this.