You don't have to replace the drywall (from what you've stated.) Use thin joint compound. Apply a little at a time and sand it between each coat. Wait 1 day between each coat. 2 to 3 coats with sanding should do well. Try to get a look at the work with natural light. It shows the 'imperfections' the best. Prime the area with a Primer SEALER, then paint.