IN MOST cases, think as mortar as s bonding agent to hold things together or act as a filler. Mortar is used between bricks to be pointed and make it look better, it would be very difficult to point bricks when they have aggregate or rocks in the mix.
Cement is used alone for streets and sidewalks, it can be screed to be smooth and the rocks will sink for a nice surface, but you cannot screed the mortar in between the bricks.
The only exception that I know of is that in making a cement flooring for a shower pan, you do use mortar. Here its used like cement, but cement is not normally used. I guess its because of the construction of the "bowl" shape, cement would be more difficult again because of the aggregate.