I say I wouldn't do this because afterwards you're wanting to install sheet vinyl or vinyl tiles; both are very unforgiving floors when it comes to bringing attention to every flaw there may be in the surface they are attached to & will "telegraph" each one to the surface. If you try & use a leveling patch to fill the gaps, I doubt you'd ever quite get there. The high spots in one area will vary from the next & I think what you'll likely end up with is a very wavy looking floor. It may feel & seem level to the touch after filling, but if you laid a long straightedge across the surface it wouldn't be. Cover this with a glossy-finish material & it will really show up.
Since this is a concrete floor, even though it may seem tough, you probably have an easier job in cleaning than if it were a wood subfloor.
Look into renting a vibrating scraper if the job is too daunting to finish by hand.