PSA - Pay attention when making changes to your disks

Lounge
#1

So…I needed to destroy and recreate the zpool which holds all my root and home datasets for all my OSes.

Since I maintain hourly replicas of those on another machine, I thought it would be pretty clean. I tested to make sure they were valid. Everything seemed OK and low risk. In the worst case, I could reinstall since all my important config is in a different zpool.

I did a test run in a VM to ensure that everything would be smooth and I knew what I was doing.

Then, when I went to create the new zpool, I used the wrong partition and overwrote one of my other zpools. You know, the zpool with all the config mentioned above that I wasn’t prepared to lose. One mistake and “poof”, all data gone.

That zpool held, my pictures, my vmware config, my firefox profile, my thunderbird profile, my steam data and all my most important VMs.

The good news is that my pictures are safe since I have a replica I update nightly and I take daily borg backups.

Most of what is left is inconvenient to recreate, but not the end of the world.

The biggest loss is my Windows VM. I know, many of you probably don’t see that as a loss but I had some data in it that I hadn’t yet copied out. Most notably, my most recent years tax documentation.

So, it is going to be a long day/week/month trying to reassemble everything.

In the end, I guess the learning is the same as always, be careful with your data and have backups.

9 Likes
#2

Ouch. Yeah, at some point I think everyone no matter how much they understand the idea of being careful in concept makes that mistake and learns just how important it is the HARD way…

3 Likes
#3

Ouch! Got a bottle or two ready? :tumbler_glass:

3 Likes
#4

I used to guard against this kind of thing by unplugging (physically) the connection to my backup drive - anything accessible can be borked!

Of course, even that didn’t work ALL the time. Even if recovery is straightforward, the ‘grrrr’ sounds you make will be distracting! Concentrate - and good luck!

2 Likes
#5

Bad luck.

Like a friend of mine (years ago) accidentally deleted his dissertation by mixing the original and backup disk in a fatal way.

There’s no way to fully prevent accidents. One can only try make things less prone to human errors.

1 Like
#6

Well, the good news is that my original task seems to have worked well. I am back in my Fedora install which was the more important one to restore since it is so difficult to install Fedora on zfs. I also restored my Arch install but I probably won’t boot into it until I get everything configured and working since the two installs share all the config that was lost.

Been working on Firefox for the last hour. :neutral_face:

4 Likes
#7

My dad was a carpenter for 50 years, and he said that it was often the most experienced guys who hurt themselves worst on the jobsite, because they were the most confident that they knew what they were doing. He mentioned this after dropping a 30lb. circular saw on his head that he’d left on a floor joist above him, which had been fine until he stepped on the dangling power cord.

6 Likes
#8

Oof… that sounded painful. One backup is no backup. I’ve long since learned to triple-load. Online, offline, and air-gapped. It’s easier to back up everything than have to try and figure out what’s worth keeping. I try and avoid running any commands without at least two coffees inside me… :slight_smile:

1 Like
#9

I agree but I have too much data for that so I have to be a bit more selective. I did, however, add my vmware, firefox and thunderbird profiles to my backups. Backing up all my VMs would be costly and not have much benefit. I have(had) over 100 VMs.

#10

Well that’s no in any way indicative for a normal backup strategy… :rofl: I do not envy you…

1 Like
#11

Soooooo…It’s a win-win situation then? :upside_down_face:

But yeah, i feel your pain. Good thing you still have most of other data backed :+1:

1 Like
#12

It’s a feeling all too familiar to me. You type in the command, triple check everything, hit Enter… and then realise what you have just done. The cold sweat, the hands shaking… and all you can utter is: “oh, :poop:”…

I’ve been there on more occasions than I’m willing to admit.

3 Likes
#13

Although it’s not @dalto’s specific case, but…

To avoid that feeling :hot_face: :cold_face: i always use 1 partition = 1 physical disk scheme, and if / when i reinstall OS i always physically disable disks with non-OS data :sweat_smile:

Call me paranoid…

Still, even with only one disk to mess up (and all the backups in place) - i have that feeling, it never goes away…

2 Likes
#14

Data I mustn’t lose under any circumstance (category I), things like photos of my late father, I keep burnt on multiple DVDs shared to other, trustworthy people (as well as locally). Other precious data (category II) I have multiple copies of on several different computers, and at least one on an external, unplugged drive. Important files, like config files, current projects (category III) I keep a single backup of on a home server and some of it is copied on other computers. The other two categories are files that would be inconvenient to lose, and files that are expendable. I sometimes make copies of the former.

1 Like
#15

I exactly know how that feels, when the cold sweat starts to cover your skin when you realize what you did. You have my sympathy.

2 Likes
#16

RIP @dalto ‘s data. Sorry to hear of your loss, bro’.

1 Like