If any vote fails clearly - not because afk, but enough !n-votes - the vote, the first argument, the voters and the duration (a half hour) should be stored in database.
if someone repeat it while the majority of !n-voters is still in this room, repeat the vote - if it fails again, alert the caller, that a new recall in the next 1/2 hour will ban him.
A third try will ban him.
---
If this is either not possible or not good in a way I forgot to think about, just disable this player to start a new vote in the next minute (other ppls get the chance to call a vote)
If someone get kicked, disable this player's votes for 5 minutes.
If someone with disabled votes get kicked, disable this troll's votes for 30 minutes.
If nobody of the devs will complain and enough peoples want it, I will try to figure out how to implement my last though.