Server Frame Drops with Net Graph etc

Im about to publish a DarkRP Server and got all ready but I dont know what to do with the frame drops.
It cant be the server as you can see here . This is the Net Graph when I have such Frame Drops:
As I can see there its client side but I have a pretty decent PC so I hope anyone can explain it to me.
Best regards Azoni

this is the collection:
+some gmodstore addons like VCMod + ELS, Arcbank etc

Do you happen to have a lot of entities such as chess boards, arcade machines, etc out and setup? That contibutes to a lot of fps lag on my server