Gmod server FPS lag

After many people join the server, usually about 30+, my fps gets really really low. This is a picture after being on the server for about an hour with about 35 people on

I am getting around 20 fps

Here is a picture, in the same location on the map, after a reconnect and there is still about 35 people on

Now I am getting around 130 fps

Here is a picture in town with a lot of people

Getting around 40 fps

Server rates:

sv_maxrate 30000
sv_minrate 1500
sv_maxupdaterate 33
sv_maxcmdrate 33
fps_max 300

NFO is hosting the server

More info:

Your server is currently being hosted on a Intel Xeon E5-1660 (Sandy Bridge-EP 3.3ghz) node with 6 total processor cores and 64 gigabytes of RAM. The database code for this node is seattle-hexa33; its OS is Windows. 

These are the current stats on your entire machine (not just your game server), based on the last 5 minute period:
CPU usage: 30.58% 
Free physical memory: 51425mb 
Machine uptime: 307 days, 17 hours, 44 minutes, 6 seconds

Server CPU usage is around 80 percent with 35 players on right now.

Server doesn’t have many addons.


When I join other servers with a lot of people, they too have low fps, but it isn’t as choppy as my server.
For example:
On my server, when I have around 30 fps, if I look at a door, strafe back and forth and keep my aim on the door, it is very choppy.
On other servers, If I do the same thing, it is still smooth, even with 30 fps.
Is this a problem with my rates? should I make fps_max higher?