That would quickly become a slippery slope, potentially to justify silencing views you don't agree with, and restricting freedom of speech.
Banning someone for violating the Terms of Service is not a 'free speech' violation.
Banning someone for violating the Terms of Service is not a 'free speech' violation.