I am kinda on the other side of this. If the client wants to know when and who you talked to at the IRS, just tell them. By acting like you have something to hide, you fuel their concerns. I would "dummy down" the notes if they are very detailed. Change to something like "Spoke to Mr. West ID#1000XXXXX on xx/xx/2020 concerning client 1040X. Mr. West confirmed that IRS is still in backlog and that interest and penalties will clear up when 1040X processed". And that is the truth, so why not put it in writing. What is he going to do, sue you? Wait till he sees the backlog at the courts because of Covid. He will get his refund, and the next 3 or 4 refunds, long before he gets a hearing.
Sometimes, people just don't trust anybody and they are looking for a reason to be upset, especially now. If you give him the notes, or the summary version of the notes, the problem goes away.
My humble opinion. Your practice, your decision on how to proceed.
Tom
Modesto, CA