Hi -- I just dealt with a very similar problem, and did solve it with caulking. Part of the problem with the old caulking is that due to the size of the gap, alot of caulk had to be used. I found aproduct that is called Caulk Saver that can be found in a hardware store where the weather striping materials are --- it is basically a long "rope" of foam material that you can sqeeze in to the lenth of the gap -- you would press it down so there is only a small space on top for caulking and then caulk over it. It worked well in my situation. You can also purchase rolls of adhesive backed stripping (usually found where the caulking and tub/tile things are at your hardware store) that is designed to fit along edges of counters or you could get shoe molding (quarter round) like used along floors and caulk/glue it in place over the gap -- paint it first.