this post was submitted on 18 May 2024
360 points (95.5% liked)

Technology

71223 readers
4083 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


founded 2 years ago
MODERATORS
 
  • Google Cloud accidentally deleted UniSuper's account and backups, causing a major data loss and downtime for the company.
  • UniSuper was able to recover data from backups with a different provider after the incident.
  • The incident highlighted the importance of having safeguards in place for cloud service providers to prevent such catastrophic events from occurring.
top 44 comments
sorted by: hot top controversial new old
[–] Gormadt@lemmy.blahaj.zone 51 points 1 year ago (5 children)

As the saying goes: if you only have one backup you have zero backups.

How the fuck does Google of all companies manage to accidentally delete that‽

[–] smokinliver@sopuli.xyz 21 points 1 year ago (1 children)

If this is the thing I heard of a few days ago then google had multiple backups on different sites but they managed to delete all of them

[–] brbposting@sh.itjust.works 5 points 1 year ago (1 children)

I guess they weren’t paying quite enough to have offline backups? I believe financial institutions can keep stuff stored in caves (think records of all the mortgages a bank wants to be repaid for - data loss isn’t an option).

[–] T156@lemmy.world 9 points 1 year ago

From the sounds of it, they did, since they were able to recover the data from elsewhere.

They just lost the data they kept and stored with Google.

[–] RedditRefugee69@lemmy.world 10 points 1 year ago

I’m betting job cuts and someone was in a hurry

[–] T156@lemmy.world 8 points 1 year ago (2 children)

Backups all tied to the same Google account that got mistakenly terminated, and automation did the rest?

It didn't matter that they might have had backups on different services, since it was all centralised through Google, it was all blown away simultaneously.

[–] erwan@lemmy.ml 2 points 1 year ago (1 children)

It's weird that backups got deleted immediately. I would imagine they get marked for deletion but really deleted something like a month later to prevent this kind of issue.

[–] Hootz@lemmy.ca 1 points 1 year ago (1 children)

That's when accounts are closed or payments missed, I think in this case they just deleted the sub itself which just bypassed everything for instant deletion.

[–] asdfasdfasdf@lemmy.world 1 points 1 year ago* (last edited 1 year ago) (1 children)

I don't see why it matters that it was a subscription. Anything which deletes data should be a soft delete.

[–] lolcatnip@reddthat.com 1 points 1 year ago

Sometimes it has to be a hard delete to comply with a user's request to remove data.

[–] ricdeh@lemmy.world 2 points 1 year ago

UniSuper was able to recover data from backups with a different provider after the incident.

[–] elucubra@sopuli.xyz 5 points 1 year ago (1 children)

My first job was in a Big Iron shop in the late 80's, where I was in charge of backups. We kept Three sets of backups, on two different media, one on hand, one in a different location in the main building, in a water and fireproof safe, and one offsite. We had a major failure one day, and had to do a restore.

Both inhouse copies failed to restore. Thankfully the offsite copy worked. We were in panic. That taught me to keep all my important data on three sets. As the old saying goes: Data loss is not an if question, but a when question. Also, remember that "the cloud" simply means someone else's remote servers over which you have no control.

[–] Diplomjodler3@lemmy.world 9 points 1 year ago (1 children)

And had you ever tested the restore process?

[–] elucubra@sopuli.xyz 1 points 1 year ago

In a big iron shop?everything gets tested, dry run, etc, but shit happens, hence backups

[–] Hootz@lemmy.ca 3 points 1 year ago (2 children)

Everything is tied to the subscriptions, they deleted the sub and that automatically deleted all backups.

[–] linearchaos@lemmy.world 3 points 1 year ago

That sounds like a pretty trashy backup scheme. I don't care what your subscription status is I'm keeping those backups until retension's over.

[–] capital@lemmy.world 2 points 1 year ago

Very stupid.

AWS has a holding period after account deletion where nothing is actually deleted, just inaccessible and access can be regained without data loss.

Since first hearing about this I’m wondering how TF Google Cloud doesn’t have a similar SOP.

[–] possiblylinux127@lemmy.zip 35 points 1 year ago (2 children)

"an unprecedented sequence of events"

[–] Nurse_Robot@lemmy.world 7 points 1 year ago (1 children)

Yeah? It was, what's your point?

[–] possiblylinux127@lemmy.zip 23 points 1 year ago (2 children)

It sounds similar to "a unscheduled pen test" and other corporate speak

[–] andrew@lemmy.stuart.fun 15 points 1 year ago (1 children)

These situations are almost always self-inflicted. If someone else hacked Google Cloud this badly then you'd likely have heard it from them first. And they probably would have done something significantly more destructive if their goal was harming Google reputation.

[–] possiblylinux127@lemmy.zip 3 points 1 year ago

I just found the phasing to be kind of funny

[–] skittle07crusher@sh.itjust.works 5 points 1 year ago (1 children)

“rapid unscheduled disassembly” 🙄

[–] possiblylinux127@lemmy.zip 1 points 1 year ago

Boeing is opening the doors to the ~~future~~ plane mid flight

[–] 1984@lemmy.today 3 points 1 year ago* (last edited 1 year ago) (1 children)

Sounds really dramatic in a news item though. Click bait. :)

But yeah, I recently moved away from these cloud services and have a Nas at home now. Only encrypted backups in cloud. Because fuck Google.

[–] brbposting@sh.itjust.works 3 points 1 year ago

“Unprecedented” is kinda hot right now. Tries to mitigate too much blame being heaped on: “obviously we prepare for the usual and even the unexpected, but this has literally never happened before (give us another shot pls)”.

So it’s interesting for the news that it takes on a different context when said breathlessly: “UNPRECEDENTED failure!”

[–] riodoro1@lemmy.world 35 points 1 year ago (2 children)

Second week with this story.

[–] I_Miss_Daniel@lemmy.world 7 points 1 year ago (1 children)
[–] umbrella@lemmy.ml 7 points 1 year ago* (last edited 1 year ago) (2 children)

reading about google fucking up and bringing other corporations down with it never gets old.

[–] applepie@kbin.social 6 points 1 year ago

Sudar the creep will keep your nudes around for rest of eternity but can't provide proper enterprise product...

Cheers

[–] Aux@lemmy.world 1 points 1 year ago

Reading about Google fucking up got old ten years ago.

[–] TonyTonyChopper@mander.xyz 4 points 1 year ago

I was gonna ask if this happened again lol

[–] nieceandtows@programming.dev 14 points 1 year ago (2 children)

They're talking like it's some global celestial event

[–] andrew@lemmy.stuart.fun 19 points 1 year ago (1 children)

Unprecedented only means there's no precedent. This just hasn't happened before at this scale.

[–] nieceandtows@programming.dev 20 points 1 year ago

I was only commenting on the phrasing. 'unprecedented cloud event' sounds like some global scale meteorological event.

[–] FiniteBanjo@lemmy.today 3 points 1 year ago

The headline does say "Customer account" as in singular.

[–] ipkpjersi@lemmy.ml 14 points 1 year ago (1 children)

Follow the 3-2-1 rule for your important data, ideally 4-3-2 or better. Remember, if you only have one copy of your data, you actually have zero copies of your data.

[–] ohwhatfollyisman@lemmy.world 5 points 1 year ago (1 children)

what are these rules? i genuinely am not aware of them.

[–] LastElemental@lemmy.world 13 points 1 year ago (1 children)

3 separate backups on 2 different media (ie 2 backups on 2 separate HDDs plus one on DVDs) with At least 1 offsite (ie a satellite office or your parents house for personal stuff)

[–] ohwhatfollyisman@lemmy.world 5 points 1 year ago

cheers mate.

[–] corsicanguppy@lemmy.ca 7 points 1 year ago

If you didn't put Google's name in there I would've assumed a different company facepalming. Hint: it's the one whose name sounds like 'unsure'.

[–] ad_on_is@lemmy.world 4 points 1 year ago

Always follow the 3-2-1 rule, Google. Always!

[–] fxt_ryknow@lemmy.world 4 points 1 year ago

At the end of the day... Cloud storage is just using someone else's computer.

[–] _sideffect@lemmy.world 3 points 1 year ago

Imagine if YouTube lost all its videos