For follow-up discussion and general commentary on the topic. Comments are sorted chronologically.
Catholics, Orthodox and Anglicans have Canon Laws to handle such cases. I am interested to hear how the Protestant communities handle these.
All we have is read the Word of God to answer the question. 1 Timothy 3;1-7 " This is a true saying, If a man desire the office of a bishop, he desires a good work. A bishop must be blameless. the HUSBAND of one wife, vigilant, sober, of good behavior, given to hospitality, apt to teach. Not given to wine, no striker, not greedy of filthy lucre, but patient, not a brawler, not covetous. One that rule well his own house, having his children in subjetion with all gravity; For if a man know not how to rule his own house, how will he take care of the Church of God? Not a novice lest being lifted up with pride he fall into the condemnation of the devil. More ever he must have a Good report of them that are without lest he fall into reproach and the snare of the devil" if he does not qualify; he should not be a pastor.. God can not be mocked.....
I agree with just about everything in the three answers listed when I just read them. There is, however, one thing I would like to add that I believe is relevant to the question.
Pastors (or elders), like all believers, still have a sin nature and wrestle with temptation. They do sin (Romans 7:14-24; 1 John 1:8-10). When this happens, they ought to confess their sin to God, and accept his forgiveness. If they do not do so, and others become aware of their sin, the steps outlined in Matthew 18:15-17 should be followed: (1) speak to him privately, (2) if he refuses to listen to you, involve one or two others, and (3) if he still refuses to listen, it needs to become a matter for the entire church.
The one other thing I would add is this: 1 Timothy 5:19-20 says, "Do not entertain an accusation against an elder unless it is brought by two or three witnesses. Those who sin are to be rebuked publicly, so that the others may take warning." A person who is in such a respected position should not be destroyed by the accusation of one person. But if there are two or three who agree with the accusation, the matter should be investigated. If it is true, the person with the public church ministry should be rebuked before the whole church.
This action should be taken normally by the other church leaders (normally the elders).