I don’t think writing to the courts is a bad thing, just make sure you keep all the evidence.
Unfortunately I think this will go to court. You just need to make sure you are seen to be doing the right thing, and you can prove it.
I.e offered him the money back following an inspection etc. He is the one being unreasonable not you. The judge won’t care who replied to who or who hasn’t read letters etc. The judge will just look at the facts. You have offered a refund, however following a suspicious set of events you want an inspection first. That’s perfectly reasonable.