Marion. It is glazing falling out of your windows - not caulking.
Glazing helps hold the glass into the frame - ane seals the glass and wood from the elements.
You need to remove the old glaze and then put in some new.
Heat the old stuff with a heat gun - be careful - you will break the glass if you get it too hot. - you just want to get the glaze soft.
You can remove the glaze - and install new with a glazing knife.
Home Depot has glaze, knives, etc.
Please don't put caulking in there - that is not what caulking is for - it will make a mess.
© 2016 Renovate Your World LLC