• My server is being hit by some pretty powerful DDoS attacks.
    27 replies, posted
And yes, it's definitely a DDoS attack. Our host (NFOServers) has confirmed that we're being attacked by a botnet. This has happened four times over one and a half weeks, and each time we've been nullrouted by NFOServers since it was a 30Gbp/s attack and would've affected other servers in the area. To accompany that, we've been getting hit by smaller scale attacks that NFO was able to successfully filter. Here's the message we've been getting for the DDoS attacks that have been causing us to be null routed; [quote]A very large DDoS attack against your IP address [server IP here] overloaded our connection speed and/or our router's filtering capacity at your location a short while ago and caused location-wide problems. We were forced to null-route the IP address as an emergency response measure. This null-route will need to remain in place for at least 8 hours. [URL="http://www.nfoservers.com/forums/viewtopic.php?f=25&t=11456"]We talk more about what a null-route means in our knowledge base[/URL]. We are always upgrading our infrastructure to make sure that null-routes remain a rare, emergency measure, and we investigate every null-route to explore what we and Internap can do to filter it better.[/quote] And here's one of the messages we get when a smaller scale attack is successfully filtered; [quote]Our system detected a (D)DoS against your service at this time, described as 'a UDP flood of length-378 packets', and added a filter to our router to block it for about 60 minutes. Depending on the size and characteristics of the attack, and the nature of your software, you may or may not have seen effects from the attack before it was filtered. Most attacks are spoofed (use random fake IPs). This means that it is not possible to examine the traffic and determine the attacker. We will likely not be able to provide further information on this attack.[/quote] Or; [quote]Our system detected a (D)DoS against your service at this time, described as 'a UDP flood containing the string "Tr0x"', and added a filter to our router to block it for about 13 days. Depending on the size and characteristics of the attack, and the nature of your software, you may or may not have seen effects from the attack before it was filtered. Most attacks are spoofed (use random fake IPs). This means that it is not possible to examine the traffic and determine the attacker. We will likely not be able to provide further information on this attack.[/quote] Needless to say, it's quite annoying, since we dropped from the 32nd most popular server on Gametracker to the 211~th in a matter of a week thanks to the downtime. I'm willing to bet this is from a competitor, too... Is there anything I can do to figure out who might be launching these attacks, or block them more efficiently? For reference, we're running on a virtual machine/VPS in Managed mode, running Linux.
You have several options here. You can: 1) Make up with the party who feels they are wronged (unless you get unlucky and this is just someone attacking you for a laugh or is a competitor). You should probably go with this option as it will cost a lot less in the long run.. 2) Wait for the attacks to stop. It can take several days to several months depending on how determined this person is. 3) Pay for third party DDoS protection. This can be tricky because if the scrubbing center is too far from the game server then latency is going to be high. 4) Colocate/Rent a dedicated server with someone like CNServers.. you're probably still going to have to pay out the *** for protection capable of blocking 30Gbps+ [editline]28th September 2014[/editline] Or you could just go with a host like OVH and hope the attack is common and VAC is preconfigured to block it.
[QUOTE=darksoul69;46101667]You have several options here. You can: 1) Make up with the party who feels they are wronged (unless you get unlucky and this is just someone attacking you for a laugh or is a competitor). You should probably go with this option as it will cost a lot less in the long run..[/quote] I have no idea who it is. [quote]2) Wait for the attacks to stop. It can take several days to several months depending on how determined this person is.[/quote] We decided to do this for the time being. [quote]3) Pay for third party DDoS protection. This can be tricky because if the scrubbing center is too far from the game server then latency is going to be high.[/quote] What's a good one in the Chicago area? [quote]4) Colocate/Rent a dedicated server with someone like CNServers.. you're probably still going to have to pay out the *** for protection capable of blocking 30Gbps+[/quote] I've been looking around, and OVH sticks out. [quote]Or you could just go with a host like OVH and hope the attack is common and VAC is preconfigured to block it.[/QUOTE] I'm looking at SoYouStart for once they get back in stock, in particular, [url=http://www.soyoustart.com/us/offers/e3-sat-1.xml]this package[/url] or [url=http://www.soyoustart.com/us/offers/e3-sat-3.xml]this package[/url]. They're a subset of OVH and offer the same protection, minus the API.
So you're the guy NFO has been sending messages about. NFO keeps messaging me saying another customer has been getting powerfully DDOS'ed and that it may cause lag on other servers. [editline]28th September 2014[/editline] To add on, personally, you might have to get a new IP and new community name. That's my only ideas.
Ah, good ol' Garry's Mod Gmod server owners are the worst
[QUOTE=Luni;46101894]Ah, good ol' Garry's Mod Gmod server owners are the worst[/QUOTE] How so? I'm a pretty decent guy.
There isn't much you can do to stop DDoS attacks besides hopefully mitigate it, which is why it's so powerful for how much effort it takes to do it.
I was referring to the people who are obsessed with donations and jealously attack any of their "competitors"
[QUOTE=Luni;46101919]I was referring to the people who are obsessed with donations and jealously attack any of their "competitors"[/QUOTE] Ah, yeah, those types are very bad indeed.
[QUOTE=Luni;46101919]I was referring to the people who are obsessed with donations and jealously attack any of their "competitors"[/QUOTE] I feel bad for them, they obviously have some sort of r3tardation... (irony)
Anyhow, unless we get significantly more donations somehow so we can handle the payment, we'll be buying a server from OVH. The long-term plan is to buy it from SoYouStart once they get more back in stock, but if this keeps happening frequently we'll need to buy from OVH instead of SoYouStart.
[QUOTE=WitheredPyre;46102133]Anyhow, unless we get significantly more donations somehow so we can handle the payment, we'll be buying a server from OVH. The long-term plan is to buy it from SoYouStart once they get more back in stock, but if this keeps happening frequently we'll need to buy from OVH instead of SoYouStart.[/QUOTE] How much do you need? ;D
[QUOTE=SwikCoder;46102233]How much do you need? ;D[/QUOTE] Well, the main issue here is, we have $300 or so in donations sitting around. This would be enough to pay off the setup fee ($127 or so) and the first month for the $109 server, but we like to have enough money to last two months or more in advance just in case disaster strikes. I'd love you if you're actually willing to pay at least $100 of that, although I seriously doubt anyone is kind enough to throw around that kind of money to someone they don't even know (and if you are, bless you). This is why we want to wait for the mid-tier hosting to be restocked (one of the two that I linked above),Caine we can afford that and it's somewhat cheaper/equal to our current hosting plan with NFO, in terms of price. To be completely honest, we can probably garner up the money by explaining the situation in full to our players and making the message noticeable, but anything that can accelerate this process if we don't get things sorted out and have to switch is very helpful and generous. I'm not going to ask you to send us money, of course. Only if you really, really want to.
Hey, I am experiencing the EXACT same thing as you are. However, we've managed to sustain stability and uptime despite heavy lag spikes. I'll show you some of the DDoS logs: [IMG]http://i.imgur.com/8oArNIU.png[/IMG] We're getting erratic player counts because of this :( [url]https://www.gametracker.com/server_info/play.auroraen.com:27015/[/url]
[QUOTE=Retribute;46102654]Hey, I am experiencing the EXACT same thing as you are. However, we've managed to sustain stability and uptime despite heavy lag spikes. I'll show you some of the DDoS logs: [IMG]http://i.imgur.com/8oArNIU.png[/IMG] We're getting erratic player counts because of this :( [url]https://www.gametracker.com/server_info/play.auroraen.com:27015/[/url][/QUOTE] Someone clearly doesn't like us. It's just leads me to put more faith in the idea that it's a competing server owner.
Ah the server list. The only place where making your server better means DDoS'ing the fuck out every other server so you're the only one left!
I've had similar experiences having my servers go in and out of the top 50... As soon as you rank in the top 50, you will start to see frequent ddos attacks. SoYouStart has DDoS protection, but there is a delay before it activates, in which time your players may time out from the attack. OVH has the option to keep the protection on permanently.
[QUOTE=WitheredPyre;46102419]Well, the main issue here is, we have $300 or so in donations sitting around. This would be enough to pay off the setup fee ($127 or so) and the first month for the $109 server, but we like to have enough money to last two months or more in advance just in case disaster strikes. I'd love you if you're actually willing to pay at least $100 of that, although I seriously doubt anyone is kind enough to throw around that kind of money to someone they don't even know (and if you are, bless you). This is why we want to wait for the mid-tier hosting to be restocked (one of the two that I linked above),Caine we can afford that and it's somewhat cheaper/equal to our current hosting plan with NFO, in terms of price. To be completely honest, we can probably garner up the money by explaining the situation in full to our players and making the message noticeable, but anything that can accelerate this process if we don't get things sorted out and have to switch is very helpful and generous. I'm not going to ask you to send us money, of course. Only if you really, really want to.[/QUOTE] Post your Paypal bud :) Ps. GMC Hosting is good
I've setup such a infrastructure to deal with DDoS and SRCDS based DoS attacks using OVH plus KAD. The problem with OVH is that they are in Montreal, Quebec. And Steam master servers geo-locate so the OVH IPs only show up to people close to that area. I've observed mostly Canadian, some north eastern USA, and EU traffic. With the occasional connection from Africa (idk). If your still interest in a server that will remain online you can PM me.
I would suggest blocking GameTracker from tracking your server. Part of the community I work for has been doing this for a long time and I believe it has helped in reducing the amount of DDoS attacks we receive. It creates an unnecessary popularity contest which I doubt any actual players look at.
This sounds eerily reminiscent of a Call of Duty DRDoS attack, whereby the attacker queries a whole bunch of servers, and then forges the return IP so that the servers collectively return packets to the attack target instead of the initiating attacker's IP(s). I forget the name of the tool used, despite searching for it a little bit. Basically, you might be fucked for a while.
[QUOTE=wickedplayer494;46109098]This sounds eerily reminiscent of a Call of Duty DRDoS attack, whereby the attacker queries a whole bunch of servers, and then forges the return IP so that the servers collectively return packets to the attack target instead of the initiating attacker's IP(s). I forget the name of the tool used, despite searching for it a little bit. Basically, you might be fucked for a while.[/QUOTE] lots of tools used it, you're probably thinking of devnul which isn't around any more - generally most amplification ddoses work the same way, they are effectively the same thing and like 90% of all skiddy attacks use some sort of amp anyway
Most DDoS attacks are DRDoS. Which has been DNS and NTP mostly. But SNMP as well. Easy to filter using OVH's Firewall rules. Most DoS attacks on SRCDS itself are normally attacks designed to get the server to generate a challenge. As that seems to cause SRCDS some issues. These normally appear as A2S_CHALLENGE requests. A2S_RULES or A2S_PLAYER requests with 0 as the challenge (srcds see these requests the same as A2S_CHALLENGE). Or q packet spam (again another challenge generation attack). The q packet being used during joining so if you attempt to filter it then people cannot join your server. I designed KAD to block all forms of these attacks.
[QUOTE=Smt;46109193]lots of tools used it, you're probably thinking of devnul which isn't around any more - generally most amplification ddoses work the same way, they are effectively the same thing and like 90% of all skiddy attacks use some sort of amp anyway[/QUOTE] Watch out for Gmod Dosers, they have LOIC cannons!!
[QUOTE=Kigen;46109456]Most DDoS attacks are DRDoS. Which has been DNS and NTP mostly. But SNMP as well. Easy to filter using OVH's Firewall rules. Most DoS attacks on SRCDS itself are normally attacks designed to get the server to generate a challenge. As that seems to cause SRCDS some issues. These normally appear as A2S_CHALLENGE requests. A2S_RULES or A2S_PLAYER requests with 0 as the challenge (srcds see these requests the same as A2S_CHALLENGE). Or q packet spam (again another challenge generation attack). The q packet being used during joining so if you attempt to filter it then people cannot join your server. I designed KAD to block all forms of these attacks.[/QUOTE] it's mostly no more easy to filter than any other common & modern attack type, and if the volume is simply too big for protection to handle then there's nothing you can do, though i would assume ovh's set up should be sufficient for the majority of attacks on gmod servers
The DrDoS attacks rely on publicly available servers that don't have proper security settings. Most of these servers are setup on default ports. So with OVH's firewall all I have to do is block the source port of DNS, NTP and SNMP. Its pretty easy to do. Plus it appears that enabling the Firewall on OVH causes their mitigation system to trigger much faster during an attack. Mainly due to the fact that it routes you through "Vaccum" already by enabling the Firewall. You can see this by "tracert voice.hellsgamers.com". As far as the DoS exploits against SRCDS. They normally only use one compromised box to initiate those attacks since it doesn't take much to cause SRCDS to choke on it. The whole reason I developed Kigen's Anti-DoS is because some dude decided it'd be fun to send 10-100Mbps of random Source query packets for almost two months straight to HG's PERP server back in 2012. I actually found that Windows 2008 R2 didn't like that very much and it would actually cause problems with the OS freezing on interrupts. So I got fed up with it and bypassed Windows by having KAD read directly from the NIC card. As the interrupt freezing issue only happens with two conditions. The first is that the port must be open and accepting connections. The second is that the attack has to be randomized IPs. My theory is that Windows was trying to track all the different connections for whatever reason and thus was choking on the massive amount of random IPs. For the protection of SRCDS from the random queries I developed KAD to act like a query cache. It caching all responses to INFO, PLAYER, and RULES. And it returns those replies when the correct conditions are met. For those interested, 192.95.44.10:27015 is a server protected by the above systems. While the server isn't populated due to the aforementioned issue with Steam master servers geo-locating. It has never had an issue from an attack when we attempted to move the population to this server.
[QUOTE=Kigen;46108924]The problem with OVH is that they are in Montreal, Quebec. And Steam master servers geo-locate so the OVH IPs only show up to people close to that area. I've observed mostly Canadian, some north eastern USA, and EU traffic. With the occasional connection from Africa (idk).[/QUOTE] There's a convar to force a certain region identification.
[QUOTE=WitheredPyre;46110684]There's a convar to force a certain region identification.[/QUOTE] That does not work because GMod is broken on its query. [quote]Fletcher Dunn <fletcherd@valvesoftware.com> to me Aug 4 The server browser in Garry’s mod probably still has a bug with US-East, which is sv_region 0. Specifically, if you select that region, it actually doesn’t do any region filtering. We fixed it in TF but it doesn’t look like he has it. If you search for a particular region (besides US-East) and you get fewer than 4000 servers or so, my expectation is that: · You get all of your servers that are set to that region. · You do not get any servers set to the “world” region. That is by design. When somebody is looking for a particular region, they only want those servers. “World” region matches any region when SEARCHING. Setting your server to “world” region does not mean, “Match any search”. Can you do that test and confirm that it behaves as expected? That will help confirm what the problem is. But let’s not get to distracted by the region filter. My only purpose in bringing up region filtering at all is to confirm that the problem is that the server browser is retuning the max number of servers, and your server is just falling off the bottom of the list due to geolocation. I would expect that somebody geographically near your server would see it. If the problem is just that there are more than 5000 servers, and Garry’s mod is not doing any sort of filtering at all, and the user experience is to query and then ping 5000 servers and then locate the server by visually locating the one you want, with no filtering by the application, then I think that may be something Valve should discuss with Garry the best way to solve. Alternatively, it might be that geolocation is not working well. Geolocation is not exact, but it should work approximately. Do you have any indication that servers hosted, for example, in Australia, are showing up in a list of 5000 servers, while your server in Dallas is not? And we’ll reach out to Garry to see if he can bring over the bug specific to the US-East region.[/quote] [editline]29th September 2014[/editline] Also, to point out. [quote]Fletcher Dunn <fletcherd@valvesoftware.com> Aug 4 to me How many servers are you seeing? The master server is only going to return the closest couple thousand servers according to (approximate) geolocation. If you set the server to a particular region and narrow the search, does it show up?[/quote] You cannot force the server to appear to everyone.
Sorry, you need to Log In to post a reply to this thread.