Hi Chris I too have rafters that end flush with the outside brick wall. I can't imagine how ugly it would look with no facia board....exposed rafter ends???? Not to mention the insects that will just love to fly up into your attic and call it home!!
The facia board (usually a 1X8/10/12 is nailed to the rafter ends and comes down over your top siding board (lap siding?). If you have a ridge vent or a well ventillated attic, you'll want to leave a small gap just under the roof shingles (1/4" or so should be ok). I put a screen over that opening to keep flying insects out. Be sure to prime both sides of the facia board before putting it up.