I don't think this would be a horrible system at all. I believe It should be your choice to choose who you would like to play with in any game you host. I dont believe this kind of system can be abused. if you are going to keep kicking a player you do not like, then it would be easier to simply blacklist them and their would be no reason to kick them repeatedly. So the boot Feature would simply be used just to remove players you would not like to play with at that Particular game.
Then just don't add any open seats in the first place...!? Once you make that decision, I think everyone who satisfies your filters (and is not on your blacklist) should be treated equally.