Problems with our server

We have been having problems with our server where whenever we get over 30 players, the server(not client) FPS drops down to 10 and the game is very unplayable for our players. We tried doing a stress test with a vanilla server with no mods that is just running sandbox and we still get under 20 FPS with 30+ players in it.

Server specs:
Centos 5
2xQuad cores running @ 3.0GHz
8GB of ram
1gbps connection speed

You would soccer, you would