On another forum, the standard advice was to double their bill every year until they left or you were happy. One guy wrote back, saying he'd inherited a "problem client" (acctg and tax) who drove him NUTS always calling with questions. Annual fee was $1250. So he double it to $2500. Still drove him nuts. Doubled again to $5,000 and they were still annoying. Doubled it yet *again* to $10,000 - and now they're some of his favorite clients, with whom he is always happy to chat! The fee just wasn't commensurate with what they needed and wanted. How many times do we put up with stuff we should be charging for, and resent it, when clients might be happy to pay more? (Or walk away, with us waving energetically behind them to hurry them along.)
Make sure the ones you are firing are really PITA's that you don't want at any price. Else try hiking the price to the level they annoy you, and see. One guy said some years ago that if you double your price and lose half the clients, you're making the same money for half the work.