Just transfer the records as long as you have the authorization. The client wants to go to someone else, so let them. They won't be happy if they're forced to stay with you because you won't let their records go. Clients move on for all sorts of reasons--maybe they found someone closer to them, think your prices are too high, decided to try someone a friend or colleague swears by, whatever.
I get really angry when a new Sch C or E client can't get depreciation schedules from their former accountant. Why make their lives (and mine) miserable? They've already left you and aren't coming back. Be the grown up, act like a professional, and help with the transition. If that's how you conduct yourself, new clients will soon take the place of the former ones. I never withhold depreciation schedules, NOL worksheets, claim of rights histories, or anything except my own research and notes. What's the point?