This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
unix:lvm_recovery [2012/02/11 23:33] robm [Finding the ext4 file-system] |
unix:lvm_recovery [2013/05/28 09:28] robm [Finding the ext4 file-system] |
||
---|---|---|---|
Line 505: | Line 505: | ||
Success! | Success! | ||
- | **Update**: I ended up writing a Python script, [[find_ext3.py|https:// | + | **Update**: I ended up writing a Python script, [[https:// |
+ | |||
+ | < | ||
+ | root@skuld:/ | ||
+ | -c | sort -rn | head | ||
+ | 17 origin -134282240 | ||
+ | 16 origin -134514176 | ||
+ | 1 origin 8382976 | ||
+ | 1 origin 8268288 | ||
+ | 1 origin 8256000 | ||
+ | 1 origin 8215040 | ||
+ | 1 origin 8145408 | ||
+ | 1 origin 8133120 | ||
+ | 1 origin 8043008 | ||
+ | 1 origin 8030720 | ||
+ | </ | ||
+ | |||
+ | From this I conclude that my original (working) installation was not actually using the Logical Volume! This may explain why updating Ubuntu to a version which has LVM support by default made my system unable to find the file-system. | ||
+ | |||
+ | So from this point on I'll ignore the Logical Volumes in my disk image ('' | ||
====== Getting my data back ====== | ====== Getting my data back ====== | ||
Line 545: | Line 564: | ||
Error writing block 3226742528 (Invalid argument). | Error writing block 3226742528 (Invalid argument). | ||
+ | </ | ||
+ | |||
+ | Using my [[https:// | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # head vg_scratch-snap.log | ||
+ | OK / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | BAD / | ||
+ | </ | ||
+ | |||
+ | Obviously the " | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # cut -d' ' -f9- ./ | ||
+ | 22 origin 231936 | ||
+ | 19 origin 0 | ||
+ | 1 origin 880602744320 | ||
+ | 1 origin 8802304 | ||
+ | 1 origin 8790016 | ||
+ | 1 origin 8679424 | ||
+ | 1 origin 8667136 | ||
+ | 1 origin 8556544 | ||
+ | 1 origin 8544256 | ||
+ | 1 origin 8433664 | ||
+ | </ | ||
+ | |||
+ | As above, I mounted this on a loop back device and ran '' | ||
+ | |||
+ | On the advice of [[http:// | ||
+ | |||
+ | < | ||
+ | mke2fs -S /dev/loop0 | ||
+ | </ | ||
+ | |||
+ | followed by '' | ||
+ | |||
+ | ===== Lowering IO and CPU scheduling priority of fsck ===== | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # ps fo pid, | ||
+ | PID %MEM %CPU CMD | ||
+ | 19540 0.0 0.0 -bash | ||
+ | 19666 0.0 0.0 \_ sudo su | ||
+ | 19667 0.0 0.0 \_ su | ||
+ | 19675 0.0 0.0 \_ bash | ||
+ | 20181 0.0 0.0 \_ fsck /dev/loop0 -y | ||
+ | 20182 49.4 86.8 \_ fsck.ext2 -y /dev/loop0 | ||
+ | </ | ||
+ | |||
+ | To keep my system usable I lowered the IO and CPU priority of '' | ||
+ | |||
+ | < | ||
+ | root@Ikari: | ||
+ | </ | ||
+ | |||
+ | Then raise the " | ||
+ | |||
+ | < | ||
+ | root@Ikari: | ||
+ | 20182 (process ID) old priority 0, new priority 10 | ||
+ | </ | ||
+ | |||
+ | ===== Adding more swap to my system ===== | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # free -m | ||
+ | | ||
+ | Mem: 5969 | ||
+ | -/+ buffers/ | ||
+ | Swap: | ||
+ | </ | ||
+ | |||
+ | I was concerned to see that 50% of my swap was in use. I don't know how long '' | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # lsblk | ||
+ | NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT | ||
+ | loop0 | ||
+ | sda | ||
+ | ├─sda1 | ||
+ | ├─sda2 | ||
+ | └─sda5 | ||
+ | sdb | ||
+ | ├─sdb1 | ||
+ | ├─sdb2 | ||
+ | └─sdb5 | ||
+ | sr0 11:0 1 1024M 0 rom | ||
+ | sr1 11:1 1 1024M 0 rom | ||
+ | sde | ||
+ | └─sde1 | ||
+ | ├─vg_scratch-home_zfs (dm-0) | ||
+ | ├─vg_scratch-lv_scratch-real (dm-3) 252:3 0 894.3G | ||
+ | │ ├─vg_scratch-lv_scratch (dm-2) | ||
+ | │ └─vg_scratch-snap (dm-1) | ||
+ | ├─vg_scratch-snap-cow (dm-4) | ||
+ | │ └─vg_scratch-snap (dm-1) | ||
+ | └─vg_scratch-photorec (dm-5) | ||
+ | </ | ||
+ | |||
+ | It's probably ''/ | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # file -Ls /dev/sdb5 | ||
+ | /dev/sdb5: no read permission | ||
+ | |||
+ | # meermanr@Ikari:/ | ||
+ | # sudo !! | ||
+ | sudo file -Ls /dev/sdb5 | ||
+ | [sudo] password for meermanr: | ||
+ | /dev/sdb5: Linux/i386 swap file (new style), version 1 (4K pages), size 1048063 pages, no label, UUID=d0bbff73-a09a-47f6-8387-e27268cdc9fc | ||
+ | </ | ||
+ | |||
+ | Great! Let's enable it! | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # sudo swapon /dev/sdb5 | ||
+ | </ | ||
+ | |||
+ | And verify: | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # lsblk | grep SWAP | ||
+ | └─sda5 | ||
+ | └─sdb5 | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # free -m | ||
+ | | ||
+ | Mem: 5969 | ||
+ | -/+ buffers/ | ||
+ | Swap: 10234 | ||
+ | </ | ||
+ | |||
+ | ===== Extending the snapshot volume ===== | ||
+ | |||
+ | < | ||
+ | # lvdisplay / | ||
+ | |||
+ | --- Logical volume --- | ||
+ | LV Name / | ||
+ | VG Name vg_scratch | ||
+ | LV UUID 4EFJ8Y-bzWT-aif4-MlT9-4234-aS1d-qcipq0 | ||
+ | LV Write Access | ||
+ | LV snapshot status | ||
+ | LV Status | ||
+ | # open 1 | ||
+ | LV Size 894.27 GiB | ||
+ | Current LE | ||
+ | COW-table size 9.63 GiB | ||
+ | COW-table LE 2335 | ||
+ | Allocated to snapshot | ||
+ | Snapshot chunk size 4.00 KiB | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 256 | ||
+ | Block device | ||
+ | </ | ||
+ | |||
+ | As it happens, I had not allocated all of the volume group: | ||
+ | |||
+ | < | ||
+ | root@Ikari: | ||
+ | VG #PV #LV #SN Attr VSize VFree | ||
+ | vg_scratch | ||
+ | </ | ||
+ | |||
+ | So extending the snapshot is easy: | ||
+ | |||
+ | < | ||
+ | root@Ikari: | ||
+ | </ | ||
+ | |||
+ | Verify: | ||
+ | |||
+ | < | ||
+ | # lvdisplay / | ||
+ | --- Logical volume --- | ||
+ | LV Name / | ||
+ | VG Name vg_scratch | ||
+ | LV UUID 4EFJ8Y-bzWT-aif4-MlT9-4234-aS1d-qcipq0 | ||
+ | LV Write Access | ||
+ | LV snapshot status | ||
+ | LV Status | ||
+ | # open 1 | ||
+ | LV Size 894.27 GiB | ||
+ | Current LE | ||
+ | COW-table size 18.63 GiB <-- Has increased | ||
+ | COW-table LE | ||
+ | Allocated to snapshot | ||
+ | Snapshot chunk size 4.00 KiB | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 256 | ||
+ | Block device | ||
+ | </ | ||
+ | |||
+ | ===== ... 2 weeks later ===== | ||
+ | |||
+ | It has been two weeks since I started '' | ||
+ | |||
+ | Now: | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # date | ||
+ | Mon May 27 14:53:34 BST 2013 | ||
+ | </ | ||
+ | |||
+ | Size of block device: | ||
+ | |||
+ | < | ||
+ | root@Ikari:/ | ||
+ | Python 2.7.3 (default, Aug 1 2012, 05: | ||
+ | [GCC 4.6.3] on linux2 | ||
+ | Type " | ||
+ | >>> | ||
+ | >>> | ||
+ | >>> | ||
+ | 960218560000 | ||
+ | >>> | ||
+ | ' | ||
+ | >>> | ||
+ | </ | ||
+ | |||
+ | Offsets of '' | ||
+ | |||
+ | < | ||
+ | Every 2.0s: lsof / | ||
+ | |||
+ | COMMAND | ||
+ | fsck.ext2 20182 root 4u | ||
+ | python | ||
+ | </ | ||
+ | |||
+ | That's approximately 28%. :-( | ||
+ | |||
+ | < | ||
+ | | ||
+ | May 13 0.0 0.0 364 su | ||
+ | May 13 0.0 0.0 | ||
+ | May 13 0.0 0.0 1320 \_ watch lvdisplay / | ||
+ | May 13 0.0 0.0 364 su | ||
+ | May 13 0.0 0.0 | ||
+ | May 13 0.0 0.0 | ||
+ | May 13 99.1 24.3 1488468 | ||
+ | May 14 0.0 0.0 1456 watch lsof /dev/loop0 | ||
+ | </ | ||
+ | |||
+ | So '' | ||
+ | |||
+ | < | ||
+ | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND | ||
+ | 20182 root 30 10 4696m 1.4g 784 R 94 24.3 19564:51 fsck.ext2 -y / | ||
+ | </ | ||
+ | |||
+ | Total system memory: | ||
+ | |||
+ | < | ||
+ | # meermanr@Ikari:/ | ||
+ | # free -m | ||
+ | | ||
+ | Mem: 5969 | ||
+ | -/+ buffers/ | ||
+ | Swap: 10234 | ||
+ | </ | ||
+ | |||
+ | < | ||
+ | Every 2.0s: lvdisplay / | ||
+ | |||
+ | File descriptor 4 (pipe: | ||
+ | --- Logical volume --- | ||
+ | LV Name / | ||
+ | VG Name vg_scratch | ||
+ | LV UUID 4EFJ8Y-bzWT-aif4-MlT9-4234-aS1d-qcipq0 | ||
+ | LV Write Access | ||
+ | LV snapshot status | ||
+ | LV Status | ||
+ | # open 1 | ||
+ | LV Size 894.27 GiB | ||
+ | Current LE | ||
+ | COW-table size 84.89 GiB | ||
+ | COW-table LE 21733 | ||
+ | Allocated to snapshot | ||
+ | Snapshot chunk size 4.00 KiB | ||
+ | Segments | ||
+ | Allocation | ||
+ | Read ahead sectors | ||
+ | - currently set to 256 | ||
+ | Block device | ||
+ | </ | ||
+ | |||
+ | Output from '' | ||
+ | |||
+ | < | ||
+ | File ... (inode #9791282, mod time Thu Oct 5 01:40:26 2006) | ||
+ | has 11143 multiply-claimed block(s), shared with 5 file(s): | ||
+ | < | ||
+ | ... (inode #9791794, mod time Thu Oct 5 01:40:26 2006) | ||
+ | ... (inode #4115835, mod time Thu Aug 20 03:31:06 2009) | ||
+ | ... (inode #4130006, mod time Mon Nov 29 16:38:10 2010) | ||
+ | ... (inode #4784754, mod time Tue Jul 26 06:01:10 2005) | ||
+ | Clone multiply-claimed blocks? yes | ||
</ | </ |