GMod server high ping. ISP issue?

[doggydude]

I have a very small GMod server that never seems to get less than 150ms ping. A speedtest shows 10ms ping to speedtest's server, although I know that's not the whole story. The upload speed is only 7megabits/s, but I am only running this server with a max of maybe 4 people, usually just me and one other person.

Server specs:
Windows Server 2012 R2 Datacenter
3.10 ghz quad core Xeon
16gb RAM
gigabit networking equipment all the way to the modem, comcast is my ISP. speedtests are usually 90megabit/s download 7 megabit/s upload.

Normally this server is a SpaceBuild server, but to test this ping problem I removed EVERY addon, so it is completely stock. All ping tests I did were with only me on the server connecting over a 50megabit/s download 5 megabit/s upload connection. Normally I am connected over LAN to the server and have no ping problems. I have had three friends test their connection and they range from 140ms to 170ms ping. I know this isn't a server performance issue, but I tested it either way by spawning a bunch of props and complicated dupes and the base ping speed never increased (unless I spawned a bunch of ragdolls. F*** ragdolls). The server CPU usage never goes over 30%. The tickrate is currently 66 but I have experimented with 33 as well as all the way up to 512 with no noticeable difference. What I haven't messed with is rates in the server.cfg. Currently they are default as I have no idea how to optimize them.

Here is a net_graph 3 of me walking around for a couple minutes: [url]http://imgur.com/UOswUSe[/url]
If I ping my router from command prompt I get about 30ms average.
I have joined a couple other GMod servers just to test and I usually get between 40 ms and 80ms.

Thanks for any help anyone gives, as well as thanks even if nobody can help me. I love the GMod community, and this forum has helped me so much as a player and as a small server manager. Happy New Year guys, and here is to many more years of GMod to come!