Some great points have been made on both sides. As I see it, it comes down to your second sentence: "This client has issues anyway." What would make it worth, to YOU, to deal with those issues again? How much is it worth, to YOU, NOT to be dealing with those issues?
Simple return, grovelling before the IRS on the OIC, OK... it can all be done. How much would you have to charge him, up front and in cash, to deal? Feel free to say $10,000 if you were that glad to see the back side of him walk away. $500? $750? More? Less? If he'll pay that, up front, in cash, happily - or at least without verbal grumbling anywhere you can hear him, take him back. But he gets ONE chance to accept, and ONE opportunity to pay in full up front - or out he goes. If he doesn't kiss your boots (metaphorically) and happily hand over full cash right away, then he'll be nothing but trouble from the minute the cash hits your hand.
Should you take him back, what you want is a happy compliant client who does what you tell him to do and who never fusses about your fee. You do not want the client equivalent of chronic lower back pain!