You say "after the construction" and "He has done it with all the allowances, but did not state it in the contract as such."
I read that the job is done!
I think this IS the hill I would want to die on!
If it was not stated in the contract, and you used less money than what you had for an allowance, then that is your money!
It reminds me of "Can I have my change back please." (Those of youy with kids will understand.)
It is not up to the contractor to come up with reasons why you owe him more, because you spent less.
On the other hand, if you were told this in advance, it would be different. But, I don't think this was the case.
Stick to your guns and see where it leads.
How much are we talking on all the allowances?
How much over on the whole house did you go?
Ps. This is how contractors learn, what to put into thier contracts. Surprises, now matter how small, leave a bad taste in thier customers mouths and they will tell others if they got nickled and dimed.