It depends on the purpose of the payment. There is a lot of case law in regards to gifts to employees vs compensation.
Since the pastor is a former employee, there should be a clear line as to whether the payment was compensation for services; or an act of charity.
As Pacun mentioned, the intend of the church is the key. The payment could be classified as compensation If it was an enticement for future services or if he had not been fully compensated in the past. The reason for his termination could shed some light on the case.
So for example, if he had fallen upon hard times, the church could make a gift out of concern for his welfare. In defining gifts, courts have used terms like affection, pity, generosity....