my garrys mod server lags help

Hey i just had a question about my server i have 2 40 slot servers and they run fine until there gets to be over 20 clients in the server then everyone starts chopping around and it gets worse as clients keep joining. when the server slows down back to 20 or less it runs fine again? i know its not my server box because i have had 40 people in my 1st server and joined my second one with 15 people and there is no lag and it runs smooth! i set the server tick to 33 and i also added some commands other people on forums said would help but i still have had no luck in eliminating the lag. i added fps_max 500, sv_stats 0, sv_minrate 20000, sv_maxrate 25000. i dont know what im missing but if you do please let me know i want to fix this lag!