Ping going up and down (Only Garry's Mod)

Hello. Like it says in the title my ping is always going up and down. Even on servers close to me I have about 100 ping ingame. This is ONLY on Garry’s Mod and not any other game.
If I do a speedtest.net I will have good ping like in other games like CS:GO where I have about 5 ping all the time.
A GIF how it looks (I spam refresh): https://gyazo.com/562aa78bedbe5e8e3e2838e0078143c3

Any help? Thanks.

Do you have any addons? If so how many?
Have you tried verifying integrity of game cashe via steam?
Have you tried reinstalling the game?

I recently did a complete reinstall so yes, I basically just reinstalled it.
I have about 5 addons which are just some tools like precision (Dunno if even needed anymore, might already be in the game.)

Do you have any programs running in the background?

Nothing out of the general. Can’t find any reason why it would be only in Gmod??

Assuming your addons are not interfering, there are no troublesome programs, and you are running a clean install of garrysmod, I cant think of any other culprit other than your internet connection.

Are you playing over wifi or are you wired?

what are you refreshing to get this number

Wired.
Like I said, this is not about my internet connection. I have about 5 ping on Swedish servers on every other steam game. Like in CS:GO I have 5 ping on Swedish servers, maybe 20 - 30 on UK servers.
I’ve tested several servers on Gmod and they all are on the 70 - 90 level. I even tried Swedish servers in gmod and still around the same level.

View Game Info then I just click refresh.
Don’t worry, that’s not the problem. If I go in-game and look at the scoreboard and the net_graph I have the same ping, if not higher.

Does the ping readout actually effect your gameplay? Are you lagging/stuttering at all?

Yes, it’s noticeable but not that bad that I can’t play a server without enjoying it