Tired GitLab.com sysadmin deletes 300gb worth of data - Polidicks

[quote] [IMG]https://regmedia.co.uk/2016/12/14/shutterstock_mega_face_palm.jpg?x=648&y=348&crop=1[/IMG] [B]Source-code hub GitLab.com is in meltdown after experiencing data loss as a result of what it has suddenly discovered are ineffectual backups.[/B] On Tuesday evening, Pacific Time, the startup issued a sobering series of tweets we've listed below. Behind the scenes, a tired sysadmin, working late at night in the Netherlands, had accidentally deleted a directory on the wrong server during a frustrating database replication process: he wiped a folder containing 300GB of live production data that was due to be replicated. Just 4.5GB remained by the time he canceled the rm -rf command. The last potentially viable backup was taken six hours beforehand. [/quote] Source: [url]https://www.theregister.co.uk/2017/02/01/gitlab_data_loss/[/url] The entire article is an amazing read.

I'm kind of sick of gitlab's shit, tbh. There's so much downtime compared to github, and now there's this.

So a versioning control software failed to take usable backups for when something got deleted? Oh the irony

[url]https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub[/url] A pretty funny to read update timeline

[QUOTE]So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.[/QUOTE] This is just hilarious. I can't even begin to comprehend how they can fuck up so much. :v:

well this is fuckin shitty, some of my stuff was on there good thing I have a few local backups

[QUOTE=mchapra;51760257]well this is fuckin shitty, some of my stuff was on there good thing I have a few local backups[/QUOTE] The repositories aren't gone, only website specific stuff (like pull requests for example). Those are being restored as we speak. [video=youtube;nc0hPGerSd4]https://www.youtube.com/watch?v=nc0hPGerSd4[/video]

[QUOTE=mchapra;51760257]well this is fuckin shitty, some of my stuff was on there good thing I have a few local backups[/QUOTE] [quote]The last potentially viable backup was taken six hours beforehand.[/quote] Good thing you kept a local copy of your work in the last [i]six hours[/i] before the incident. Right ?

[QUOTE=Skipcast;51760270]The repositories aren't gone, only website specific stuff (like pull requests for example). Those are being restored as we speak. [video=youtube;nc0hPGerSd4]https://www.youtube.com/watch?v=nc0hPGerSd4[/video][/QUOTE] oh phew, I missed that bit, my b. [QUOTE=Nabile13;51760276]Good thing you kept a local copy of your work in the last six hours before the incident. Right ?[/QUOTE] ....yeah?

[QUOTE=mchapra;51760277]oh phew, I missed that bit, my b. ....yeah?[/QUOTE] By that I mean it's thankfully not as catastrophic as a full, complete loss of data.

Say what you want about gitlab, but they are transparent and open as hell. Other companies would prolly hide the fact that they fucked up and would just say it's routine maintenance or something or "An issue affecting a small percentage of their userbase"

[quote=GitLab] 6. Our backups to S3 apparently don’t work either: the bucket is empty 7. We don’t have solid alerting/paging for when backups fails, we are seeing this in the dev host too now. So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. => we're now restoring a backup from 6 hours ago that worked [/quote] This is funny, they didn't test their backup options.

[img]http://i.imgur.com/UBnbMIr.png[/img] Fuck the admin for theregister was tired too

Shitty tech companies want you to work loads of overtime and then wonder why you make mistakes. It's common. Crap industry to work in for that reason

[quote]So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.[/quote] You know, this fuck-up might actually be a good thing. If you have 5 backup systems in place and none of them are functioning, you are opening yourself up for a much, [I]much[/I] larger problem somewhere down the line.

speaking as someone who works with databases and the engineering of backup solutions, this is an insanely common problem database backup processes are usually shite, and their problem with the out of step binaries is common as fuck for some reason

[QUOTE=Cloak Raider;51760492]speaking as someone who works with databases and the engineering of backup solutions, this is an insanely common problem database backup processes are usually shite, and their problem with the out of step binaries is common as fuck for some reason[/QUOTE] I once accidentally wiped a production database when going live with it for the first time Hilarity ensued

I panicked just reading that, imagine the moment he realized too late what he was doing

[QUOTE=Flubbman;51760528]I panicked just reading that, imagine the moment he realized too late what he was doing[/QUOTE] Oh man I can imagine the instant vertigo, field of view change, dilated pupils and that feeling where you can sense everything around you including seeing into the past and the future where you're going "Oh fuck oh fuck oh fuck..." The palming of your face... Running the fingers through your hair, grabbing it and [B][Screaming internally][/B]

[QUOTE=sannys;51760226]I'm kind of sick of gitlab's shit, tbh. There's so much downtime compared to github, and now there's this.[/QUOTE] I think they're both mediocre at best BitBucket is my choice, I trust Atlassian not to fuck me over

[QUOTE=NotMeh;51760730]I think they're both mediocre at best BitBucket is my choice, I trust Atlassian not to fuck me over[/QUOTE] Boy do I have some bad news for you...

[QUOTE=kaukassus;51760828]Boy do I have some bad news for you...[/QUOTE] go ahead

[QUOTE=kaukassus;51760828]Boy do I have some bad news for you...[/QUOTE] Oh snap, I use bitbucket. What's wrong with them?

[QUOTE=Shadow801;51760454]Shitty tech companies want you to work loads of overtime and then wonder why you make mistakes. It's common. Crap industry to work in for that reason[/QUOTE] I mean its really the nature of the job, they have no other choice but to do it at the time of least usage I.E. the middle of the night.