So. I’ve been having issues with my server going offline multiple times in the last few days. I’ve been trying to work with my host to help them figure out what the issue is. After 5 days of consecutive server crashes my host has finally figured this out and decided that I am the one that should be suspended. This was their reply. I’m pretty unhappy with them at this point. I seem to be the one being punished for what seems like a disgruntled player.
Unfortunately, your server has been the target of a severe DDoS attack. Due to the nature of this attack, we have been unable to filter it, and in order to protect the stability of our network, we have had to temporarily suspend this subscription.
Should your server continue to be targeted with such attacks, we may be forced to stop providing this service to you.
We apologize for having to take this measure, but the attack is affecting other customers on our network. These types of suspensions will normally remain in place for 24-48 hours. We will provide you with another update here once the suspension is removed."
What kind of response were you looking from your GSP? From what I can see on the Gameservers website, they aren’t advertised as a DDoS-protected GSP, in fact, the only mention of DDoS protection I can find is for their Enterprise hosting.
This is how it works in the hosting industry. If you become subject to a DDoS attack, the IP should, and usually will, get null routed to protect their network and the individual node which your server is running on.
I’m not entirely sure why you think Gameservers have done something wrong here - do you understand how the traffic is routed from the Internet to your GSP? Not only will the DDoS attack be affecting the node your server is running on, but it may also be causing issues for dozens / hundreds of other servers if it’s saturating other routes which are being used to route the attack to you.
Let’s assume every node runs 10 game servers, and the router being used to route to your rack is handling 20 other nodes. 20 (nodes) * 10 (game servers) = potentially causing issues for 200 other game servers through network saturation.
Find a GSP which actually advertises DDoS protection, then you have some leg to stand on when they suspend your services due to a DDoS attack. I hear NFO is pretty good depending on where you need it.
Bandwidth is expensive. Sure, you can get FIOS at home for $100 / month and get a pretty decent chunk of bandwidth, but that won’t cut it for a commercial hosting operation. Not even close. When you’re hosting for money, you’re looking at a MINIMUM of a full gigabit cross connect. 10 Gb links are common. And you can’t have just one carrier. You’re going to want direct hops to the biggest ISPs, plus a few carrier-neutral links. At a bare minimum, you’ll probably want bandwidth from Verizon, Comcast, plus a few tier-1 carriers like Zayo, Cogent, and Level(3). Gaming servers require some careful network planning and tuning for latency, so it’s not just a matter of randomly picking out a few carriers.
Most carriers sell bandwidth at the 95th percentile model. For example, you might buy “50 megabits burstable to one gigabit”. Multiply that by 3-10 carriers, and you’ve got about 50 TB of usable bandwidth per month with a maximum throughput of around 3 Gb/s. I’m greatly simplifying things here, but this is the gist of it. And if you don’t think a mid-size host can burn through a few dozen terabytes in 30 days, then stop crying about networks because you’ve obviously never worked in a real datacenter.
All of that bandwidth gets damn expensive real quick. Depending on your negotiating skills, carrier blend, and infrastructure, you’ll easily drop a few thousand a month or even tens of thousands per month. That’s on bandwidth alone and does not include datacenter lease space, hardware costs, staffing costs, electricity, or anything else.
We’re keeping this simple.
So now you’ve got your bandwidth in play. You’ve priced your servers to cover your costs and turn a profit. But guess what? Along comes Captain Dickbag and his L33T H4X0R CR3W and turns up a 15 Gb DDOS against one of your players. This consumes roughly 6-7 terabytes PER HOUR. You’ll blow through your entire 50 TB monthly allocation in just a few hours. And just like those old cell phone plans, once you go over your limits, the overage charges are killer. Expect to pay anywhere from 2x - 10x your commit price for bandwidth. If the attack is severe enough, the carrier may simply drop your connection until the attack dies down. Now you’re REALLY fucked because (1) that DDOS alone is going to cost you $$$ - $$$$ in additional bandwidth charges, and (2) you’re getting even more customers knocked offline and/or seeing huge lag.
So if you expect your host to weather a DDOS attack to protect your $50 / month gaming server, you’re insane. Why should they drop hundreds or thousands of dollars to protect you? If they spend $500 on additional bandwidth charges just to protect your $50 / month plan, they need you to remain a customer for AT LEAST ten months just to turn a single penny of profit – and that’s assuming you never get DDOS’d again, or have any support requests.
There are providers who offer DDOS protection. If you need protected, go with one of them. You’ll pay more because it costs more. But don’t bash your host because they didn’t provide a service that they don’t even offer.
And if you don’t think 15 gigabit attacks are common, or that a host can burn through 50 TB of bandwidth in a month, or that 10 Gb connects are common, or that commercial-grade multi-homed bandwidth is expensive, then stop whining about getting DDOS’d. Spend some time working in an enterprise-grade datacenter or large hosting company and your opinion will change REAL quick.
It sucks that it happened to you, but it happens. Either find a protected host or stop playing.
I would hardly consider the OP ‘whining’ and find it perfectly reasonable that someone who doesn’t know how the industry works would find it odd that he’s getting punished for the actions of others.
To the OP, I hope they are reimbursing you a prorated amount for the time your server is suspended. While they are just protecting themselves by disabling it, they shouldn’t also expect for you to pay for the service you aren’t able to use.
Fair enough. But knowing about DDOS protection is part of shopping around for a server these days. Being unaware of the potential is like shopping for a car without a concept of what scheduled maintenance is going to be. If you buy a server without understanding what a DDOS is or what happens when they roll in, that’s on you.
Fair enough. I’m not going to sit here and pretend I am thoroughly knowledgeable in all the subject matter you are talking about but I do get it. I know enough to fully understand it all.
I’m also going to be honest here. It may be ignorance on my part but, before this happened to me, I had no idea what a DDoS attack was. I knew the whole concept, but I wasn’t in on the exact terminology. I don’t work in the industry.
Perhaps it IS my bad for not knowing these things while shopping around for my VERY FIRST game server. But it’s a learning process right? Shit happens, and we learn. It is however admitted in the support ticket that gameservers.com DOES have a filter for DDoS attacks, but, it was so severe they were not able to handle it. It took them 5 days to figure out what was happening.
The way I see it is this. I don’t really care what kind of service or goods you are providing via internet or not, if I (a paying customer) purchase that service or goods, I expect them to be of the highest quality. I don’t think an example is really needed here, I’m sure you understand what I’m talking about.
Also, maybe this is more ignorance on my part but is there not other options? Is it crazy for me to have the idea to migrate my server to a different ip and asking me to rename it as an option?
My biggest issue here with gameservers is their complete lack of motivation to work with me and their latency to figure out what was happening. They should have figured it out sooner and warned me of the consequences to follow if it continued. Am I pissed some punk I banned from my server is attacking it? Fuck yes I’m mad. But that’s not what we’re talking about here. That’s another issue itself. We are talking about the level of service provided to me by my host. I understand all you explained and the costs involved but as far as I look at it that’s the business they are in.
If the attacks do continue and they do suspend my service I fully expect to be reimbursed the cost for the rest of my subscription.
DOS attacks run the gamut from basic ICMP floods to full throttle custom-crafted packet floods from millions of spoofed IPs. Even one bored player can take down or severely lag a typical server just by using DNS amplification. There are different levels of DOS protection to go along with this. If you program a server to only accept one ICMP request per second per IP, yes, technically you’ve now put up a defense against some ICMP floods – and technically, you are now “DDOS protected”.
But IMHO, this is REALLY stretching the limit on what customers expect when you say “DDOS protected”. When I see a “DDOS protected” server, I’m expecting it to filter x million packets per second at y gigabits per second. Companies like Black Lotus and CloudFlare provide services exactly like this. But because packet filtering and inspection adds latency, what I would call “true DDOS protection” is rarely used in game servers.
I’m not saying either of these apply to gameservers, I’m just making a general observation. I have no idea how their infrastructure is set up.
I do understand. Unfortunately your expectation is not at all realistic.
Imagine if you just bought a brand new Bentley Continental GT V8. One of the finest automobiles on the planet. Along comes some road raging douchebag with a 4-ton 1975 Ford pickup truck. He T-bones your car at 70 MPH, sending it over the side of a cliff. Should Bentley buy you a new car here? Of course not. Even though they built and sold the car, even though they provide the warranty on the car, heck even though you bought the extended service plan on the car, it’s not their fault someone else ruined your fun. While they’ve taken all reasonable measures to protect the car and its occupants in a collision, you just can’t protect against everything. So in this case, you’ve lost your car.
“Aha,” you say. “I have insurance! They’ll pay for everything!” And you would be correct. But what is insurance? It’s an additional service (on top of the price of the car) that you pay for yourself. Your insurance carrier calculates the odds of your car being destroyed or damaged, puts you in a pool with other drivers, and figures out how much money they have to collect from each driver to spread the risk around. Bentley isn’t providing that insurance – you are. Bentley won’t write you a check for the replacement value of the car – your insurance company will.
It’s the same with buying a server. It’s not the provider’s fault your server got DDOS’d. They may have built the server to stand up to some basic attacks, but if the severity of the attack goes past those limits, the only thing they can do is null route your IP in order to protect the rest of their customers. If you want “insurance”, find a GSP offering REAL DDOS protection on their servers. You’ll pay more (probably at least an extra $50 / month / IP), but you’ll be better protected. Is it worth it? That’s entirely up to you.
No, there are no other options short of the GSP introducing network- and server-level DDOS protection. Since quality DDOS scrubbing costs a fortune, they’re going to charge you extra for that. Either they’ll make DDOS protection an additional optional feature (probably by keeping a handful of DDOS-protected servers in DDOS-protected IP blocks), or they’ll spread the cost by raising everyone’s rates.
A third option might be to look at getting your own server. It will be a lot more expensive: expect to pay around $150 - $250 / month for a Xeon E3-1230v3 (about on par with a mid-range Haswell i7, if not just a TINY bit faster) with 10Gb DDOS protection on a GbE port in a reliable, well-connected tier 3 datacenter. You could also take your chances at someone like OVH, but their infrastructure is notoriously hit-or-miss.
Most providers know immediately when a DDOS is occurring by watching tools like Netflow for a sudden surge in traffic. My employer has a fairly robust semi-automated network management system that regularly polls the switches and does some neat tricks when specific network anomalies occur (such as a sudden spike in bandwidth to a specific IP). If that fails to mitigate the event, automated alerts get sent out and we deal with it by hand (which may include null routing the IP until it passes). It can be fun watching a few hundred thousand IPs sending SYNs to a server at the same time for hours on end, and even an automated script on a 10G network is going to take time to deal with that. But this level of service is very expensive, which is why we don’t have any GSPs on our network. If GSPs were to start offering this level of advanced DDOS mitigation for their clients, that $20 / month Rust server would jump up to $100 / month in no time.
Now, should they have noticed faster? There’s no reason for a service provider not to notice a DDOS attack after 2-3 minutes. Those automated tools a VERY easy to work with, and any python / perl / bash monkey can slap together something in no time. I don’t know what their infrastructure looks like so I can’t tell. But from their website, they don’t offer DDOS protection anyway, so there’s nothing they could have done anyway. Your server would have gone down whether you were notified 30 minutes later or 30 seconds later. You have a valid point, and you’re right to feel frustrated, it’s just that I don’t think it’s fair to bash on a provider for failing to provide something they don’t even claim to provide.
[editline]12th February 2015[/editline]
I’ve re-read my post from earlier today, and I may have come across as a little harsh. You didn’t deserve that, OP. That was a knee-jerk reaction to what looked like yet another thread complaining about yet another DDOS attack, and I let my annoyance spill through in my explanation of what’s happening behind the scenes. There have been plenty of “guys my GSP got DDOS’d and null routed me so now I will sue them!111!!!1one!!” threads and yours was not one of them. Sorry about that!