If it's white siding, first, pull the panel off with a Zip Tool. Then, put a piece of tape behind the hole. Put the siding back together. Then, put a piece of Painter's Masking Tape OVER the hole and cut it out a liiiitle bigger than the hole, like 1/8" bigger around. With the tape in place, put on some silicone caulk. Wet your finger in some soapy dishwater and smooth it out flush with the tape. Let it dry overnight. Then, rip off the masking tape VERY CAREFULLY. If the tape is too stuck to the caulk, cut it with a utility knife VERY CAREFULLY.
You won't notice it much at all. IF the vinyl is a different color, or it's white and you don't like the idea, then try and find a panel in a less-conspicious part of the house that's the SAME SIZE as the one you want to replace. Switch them using the Zip Tool. THEN, do the patch job I described. (No one will see it if it's in the back of the house or high-up on the house.