Josh, there are basically 2 kinds of "primers". The more common are just thinned paints. Many interior and exterior wall paints fit this category. Usually, better products are available to aid in specific application situations. This would include such things as shellac based primers that are also good sealers and stain killers for interior walls and automotive primers with a lot of filler for good sandability. It would help a lot to know what specific "primer" is on your brick.