Hi, Josh. First, I'm pretty sure that you didn't shake the can anywhere near well enough. The white globs are probably diatomacious earth or talcum powder. These make the primer have a very flat finish--no shine when dry--and sandable. But if you were spraying them out in clumps, the primer was not mixed well. The toluene and xylene are solvents to make the primer liquid. One aside is that Rustoleum has a fish oil base to retard rust. You will be a lot better off to stick with Rustoleum products on future repaintings.
Now to your real answer. Just about any paint stripper will remove the white blobs. Citristrip is somewhat safer to use, in my opinion, and still works well. For outside like this, and if you keep kids and pets well away, any of the many methyl chloride based stripper will also work. Try not to work in direct sun or high heat conditions as the stripper will tend to dry before it releases the primer that's holding the blobs together. You want to glob the stripper on the blobs only and thickly to keep the active chemical working on the blob and not evaporating. The more that you brush back over the stripper, the more that evaporates. When the surface of the blob gets real wrinkly, you can use a metal putty knife to remove the loosed primer. It will probably take a couple of hours to work but there are so many variables it is impossible to predict exactly how long this will take. If the stripper itself shows signs of getting crusty, too much of the active chemical has evaporated so you will want to get real busy with the putty knife. If the stripper dries completely, it can be as bad to remove as the paint.
Depending on the thickness of the blobs, you may have to apply and remove the stripper a couple of times. Depending on your brick, you will probably want an X-acto knife with #11 blades to get the primer out of indentations in the brick. Citristrip's Paint Remover Wash will not be needed for this job, if you are wondering about buying one of their stripping kits.
The good news is that this should not really be too much of a job. The bad news is that you will really clean the brick where the stripper was. Depending on the age and amount of dust, dirt, grime and pollution on the brick, the clean spots and dirty spots could leave a polka dot pattern. If this is real objectional, pressure washing will help clean everything equally but don't use so high a pressure that you remove the mortar between the bricks. A lot less is better that a little too much with the pressure washer.
Be sure that you read, understand and follow the usage and safety instructions on all the products that you use. Don't even think of skimping on the gloves and eye protection when using or removing the stripper and the same with the pressure washer. Don't use the thin latex gloves either. Use heavy dishwashing gloves or organic solvent gloves.