Man accidentally 'deletes his entire company' with one line of bad code

[Wowza!]

[QUOTE]A man appears to have deleted his entire company with one mistaken piece of code.
By accidentally telling his computer to delete everything in his servers, hosting provider Marco Marsala has seemingly removed all trace of his company and the websites that he looks after for his customers.
Mr Marsala wrote on a forum for server experts [URL="http://serverfault.com/questions/769357/recovering-from-a-rm-rf"]called Server Fault[/URL] that he was now stuck after having accidentally run destructive code on his own computers. But far from advising them how to fix it, most experts informed him that he had just accidentally deleted the data of his company and its clients, and in so doing had probably destroyed his entire company with just one line of code.
The problem command was "rm -rf": a basic piece of code that will delete everything it is told to. The “rm” tells the computer to remove; the r deletes everything within a given directory; and the f stands for “force”, telling the computer to ignore the usual warnings that come when deleting files.

Together, the code deleted everything on the computer, including Mr Masarla’s customers' websites, he wrote. Mr Masarla runs a web hosting company, which looks after the servers and internet connections on which the files for websites are stored.
That piece of code is so famously destructive that it has become a joke within some computing circles.
Normally, that code would wipe out all of the specific parts of the computer that it was pointed at. But because of an error in the way it was written, the code didn’t actually specify anywhere – and so removed everything on the computer.

“I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers,” wrote Marco Marsala. “Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.”[/QUOTE]
[url]http://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html[/url]

[MatheusMCardoso]

[quote]rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.[/quote]

Damn. Now I'll remember to never write code without pre conditions.

[LoneWolf_Recon]

How do you fuck up that bad, especially as someone with near the responsibility of a SysOp.

And going to StackExchange to seek advice just totally discredits your responsibility and trustworthiness as a web host.

Also where the fuck are any sort of backups?

[Swiket]

[QUOTE=LoneWolf_Recon;50133661]Also where the fuck are any sort of backups?[/QUOTE]

[quote]Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too.

“All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).”[/quote]

[bitches]

The man in question claims to have found a solution, after paying a company to look at [I]one[/I] customer's hard drive, who claims that all of the data is still recoverable.

[PGunnii]

Holy shit what an absolutely perfect fuckup. This is like sticking the landing in the olympics of deleting shit. 10/fucking 10

[Swiket]

Also, I love the attached image to this article.

[t]http://static.independent.co.uk/s3fs-public/styles/story_large/public/thumbnails/image/2016/04/14/16/laptop-fire.jpg[/t]

[LoneWolf_Recon]

[QUOTE=Swiket;50133679][QUOTE]Mr Marsala confirmed that the code had even deleted all of the backups that he had taken in case of catastrophe. Because the drives that were backing up the computers were mounted to it, the computer managed to wipe all of those, too.

“All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).”[/QUOTE][/QUOTE]

Horrible luck, but honestly that should've been well thought before the system was setup. Mounting the offsite backups in the same script that does any sort of file removal or cleaning (I assume this script, just by the name, does clean up work of temp files/etc) is asking for some sort of trouble.

The whole point of offsite backups is to be isolated from the primary systems in case of any sort of catastrophic failure.

[Rockman1811]

I saw this on r/programmerhumor and r/softwaregore. I would feel bad for him but, damn...

[AtomicSans]

I was expecting something complex and out of my depth to be the culprit but nope. rm -rf. Fucking unbelievable.

[No_0ne]

why would you ever rm -rf

it's like the linux equivalent of committing suicide

[neesu]

Considering the fact that --no-preserve-root wasn't used and the full command was 'rm -rf /' and not 'rm -rf /*', it's impossible that this could've worked.
This sounds like a troll.

[Niteshifter]

Someone suggested using dd in the serverfault thread and his response:
[quote]I swapped if and of while doing dd. What to do now? – bleemboy Apr 11 at 7:02[/quote]
(For those who don't know, he basically made recovery a lot harder or potentially impossible)

Probably a troll, but if not, that guy is likely not going to have an IT job ever again.

[Nomad]

Why would anyone even type that command if its the equivalent of shooting your head? Or is it something you need to type before other commands to get it to run?

[Saxon]

[QUOTE=bitches;50133688]The man in question claims to have found a solution, after paying a company to look at [I]one[/I] customer's hard drive, who claims that all of the data is still recoverable.[/QUOTE]

It typically costs an arm and a leg though, pretty expensive mistake to make

I hope his insurance covers it for his sake

[HumanAbyss]

[QUOTE=Saxon;50134385]It typically costs an arm and a leg though, pretty expensive mistake to make

I hope his insurance covers it for his sake[/QUOTE]

A lot of insurance policys for data centres have pretty comprehensive protection policies in place in the event of data loss.

However, this guy seems significantly smaller scale, and may not have such a policy as those policies are very pricey from my understanding of that part of the industry.

[meppers]

[QUOTE=Sgt. Lulz;50134140][IMG]http://i.imgur.com/xoeZp4c.png[/IMG]
I'm not sure I buy this. For the command to succeed you would've had to run it as 'rm -rf /*' or 'rm -rf / --no-preserve-root'. It couldn't have been '/*' if the 'bar' variable in 'rm -rf {foo}/{bar}' was undefined like the question said it was, and you'd have to be really, really dumb to use --no-preserve-root, which has been part of the standard since 2006.
As much as I hate being a buzzkill, I felt like that was worth clarifying.[/QUOTE]

if a server works fine and newer operating systems don't let the servers work faster companies don't spend the time to update them

[neesu]

[QUOTE=meppers;50134552]if a server works fine and newer operating systems don't let the servers work faster companies don't spend the time to update them[/QUOTE]

CentOS 7 isn't that old, it definitely has --no-preserve-root implemented

[Curls]

what even is rm -rf

[londonbeckett]

[QUOTE=J!NX;50134705]what even is rm -rf[/QUOTE]

Deletes everything in the hard drive recursively until the OS crashes.

[Perrine]

[QUOTE=J!NX;50134705]what even is rm -rf[/QUOTE]

it looks like it's a basic piece of code that will delete everything it is told to. The “rm” tells the computer to remove; the r deletes everything within a given directory; and the f stands for “force”, telling the computer to ignore the usual warnings that come when deleting files.

[Curls]

[QUOTE=Combin0wnage;50134729]Deletes everything in the hard drive recursively until the OS crashes.[/QUOTE]

this sounds like something anyone with basic sense that understands linux would just... know without needing to be told

like one of those things that you should just... KNOW as a programmer/anyone dealing with that sort of thing

it sounds like the type of thing, like, you know how guns can kill you? Don't pretend to shoot yourself with one. That type of thing :v:

[Glitchvid]

[QUOTE=J!NX;50134774]it sounds like the type of thing, like, you know how guns can kill you? Don't pretend to shoot yourself with one. That type of thing :v:[/QUOTE]
Hence
[QUOTE=KingKombat;50133693]Holy shit what an absolutely perfect fuckup. This is like sticking the landing in the olympics of deleting shit. 10/fucking 10[/QUOTE]

[editline]later[/editline]

In actuality, unless the drives were zeroed, you should be able to recover the data very easily. In my experience it'd been doable, at least on NTFS. Not sure about EXT4, or if he was using ZFS.

[thatonejeff]

Remember when Pixar almost lost Toy Story 2 because of a similar incident?

[Tamschi]

[QUOTE=Combin0wnage;50134729]Deletes everything in the hard drive recursively until the OS crashes.[/QUOTE]

[I]If[/I] the OS crashes. Linux file system behaviour usually means you can even "delete" (actually detach, which is why the data is still mostly recoverable) things currently in use, and the programs in question will keep running just fine until they try to open an additional file.

[Socram]

A shell command is hardly a "line of code", but I guess that kind of nitpicking is on par with ye olde clip vs magazine discussion.

Also, anyone who uses a shell for more than a week learns that command and its potential side effects, this guy is either a successful troll, the greatest actor ever for getting any clients at all, or somehow executed it in an EXTREMELY convoluted manner.

[Comfy]

[quote]
and I use Ansible to automate some operations to be run on all servers
[/quote]

Aparently Ansible aborts script execution when a variable is uninitialized by default, So that part of the story wouldn't fit together, unless he did circumvent it somehow.

Also goes for the "rm -rf /" command, which does nothing on most Linux Systems released in the last ~5 or more years, as you also need to explicitly pass "--no-preserve-root", or runt he command as "rm -rf /*", which will work.

[quote]
Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.
[/quote]

How the hell does he not know about "set -u"?

If he ran the thing as

bash -u script.sh

Then the script would have stopped as soon as it realized that some variables couldn't be initialized.
hell he could have even includes "set -u" at the top of the file.

Also having both live websites and off-site backups mounted (What the fuck), on the same OS is reckless as hell.

The entire thing smells a bit on the Bullshit side, and everything seems to be aligned in the most convenient way.

[Ogris]

[QUOTE=Socram;50135202]A shell command is hardly a "line of code", but I guess that kind of nitpicking is on par with ye olde clip vs magazine discussion.

Also, anyone who uses a shell for more than a week learns that command and its potential side effects, this guy is either a successful troll, the greatest actor ever for getting any clients at all, or somehow executed it in an EXTREMELY convoluted manner.[/QUOTE]

Probably a troll, this is the internet after all-

[londonbeckett]

[QUOTE=Tamschi;50135054][I]If[/I] the OS crashes. Linux file system behaviour usually means you can even "delete" (actually detach, which is why the data is still mostly recoverable) things currently in use, and the programs in question will keep running just fine until they try to open an additional file.[/QUOTE]

I just assumed the command would royally fuck over the OS to the point of crashing completely. Linux is a beast.

I never tried it myself, so I wouldn't know, haha.

[dai]

[QUOTE=ThatoneJeff;50134975]Remember when Pixar almost lost Toy Story 2 because of a similar incident?[/QUOTE]

remember food fight