How "Sloppy" is the wall? If it's level and plumb, then cutting furring straps and attaching should solve the problem. Shimming may be considered.
PL400 or a product like this will glue the straps to the wall. (Requires work) Or you may use a "Remington" fastening device that shoots an anchor into the strap and wall. (Noisy)
Either way, check the wall for level and plumb, work from there.
© 2017 Renovate Your World LLC