If the 'dirt' is dirt, then wet the wall, apply a mixture of household kitchen soap mixed in w/water (as you would to wash your car), and apply w/a light, bristle brush. Work your way from the bottom UP; NOT the top down. Rinse the just-cleaned area (as well as the area just below the just-cleaned area to get the soap off.) IF you do the job from the top, down ..., you'll get streaks.
Now, if the dirt is Mold/Mildew, you'll need a mixture of 1 cup of bleach per gallon of water (or stronger if the mold/mildew is bad/heavy.) Apply and rinse in the SAME way as described above. Just note that w/a bleach solution, you'll probably kill whatever plants and such are growing in the garden (at the base of the wall, assuming you use a fairly heavy solution.) The basic mixture should be applied and rinsed sparingly anyways to as to minimize any possible damage to vegetation. Again, apply and rinse as described above (bottom - up.)