RobMeerman.co.uk

User Tools

Site Tools

You are here: RobMeerman.co.uk » Unixy Stuff » Recovering my lost data: LVM and RAID
Trace:
unix:lvm_recovery

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
unix:lvm_recovery [2013/05/14 10:36]
robm [Getting my data back] 		unix:lvm_recovery [2013/05/28 09:28]
robm [Finding the ext4 file-system]
Line 505: Line 505:
 Success! Success!
  
-**Update**: I ended up writing a Python script, [[https://github.com/meermanr/ext3_recovery|find_ext3.py]], to help me locate ''ext3'' superblocks, and automatically check their validity using ''dumpe2fs''.+**Update**: I ended up writing a Python script, [[https://github.com/meermanr/ext3_recovery|find_ext3.py]], to help me locate ''ext3'' superblocks, and automatically check their validity using ''dumpe2fs''. This showed that the most common file-system origin is actually _before_ the start of the LVM logical volume: 
 + 
 +<code> 
 +root@skuld:/home/meermanr/projects/find_ext3# cut -d' ' -f9- store_vg-store_lv.find_ext3.log | sort | uniq 
 + -c | sort -rn | head 
 +     17 origin -134282240 
 +     16 origin -134514176 
 +      1 origin 8382976 
 +      1 origin 8268288 
 +      1 origin 8256000 
 +      1 origin 8215040 
 +      1 origin 8145408 
 +      1 origin 8133120 
 +      1 origin 8043008 
 +      1 origin 8030720 
 +</code> 
 + 
 +From this I conclude that my original (working) installation was not actually using the Logical Volume! This may explain why updating Ubuntu to a version which has LVM support by default made my system unable to find the file-system. 
 + 
 +So from this point on I'll ignore the Logical Volumes in my disk image (''store_vg/store_lv''), and instead look for a file-system in the raw image.
  
 ====== Getting my data back ====== ====== Getting my data back ======
Line 761: Line 780:
 </code> </code>
  
 +===== ... 2 weeks later =====
 +
 +It has been two weeks since I started ''fsck'', and it is still running. During this time I've not been able to use my desktop PC for gaming, and so I've decided to hit ^C and move it to another machine. Here are the forensics:
 +
 +Now:
 +
 +<code>
 +# meermanr@Ikari:/home/meermanr (master *)
 +# date
 +Mon May 27 14:53:34 BST 2013
 +</code>
 +
 +Size of block device:
 +
 +<code>
 +root@Ikari:/home/meermanr/projects/find_ext3# python
 +Python 2.7.3 (default, Aug  1 2012, 05:14:39) 
 +[GCC 4.6.3] on linux2
 +Type "help", "copyright", "credits" or "license" for more information.
 +>>> f = open("/dev/loop0")
 +>>> f.seek(0, 2)
 +>>> f.tell()
 +960218560000
 +>>> hex(f.tell())
 +'0xdf917c7600'
 +>>> 
 +</code>
 +
 +Offsets of ''fsck'' and ''python'':
 +
 +<code>
 +Every 2.0s: lsof /dev/loop0                                                       Mon May 27 14:56:18 2013
 +
 +COMMAND     PID USER   FD   TYPE DEVICE     SIZE/OFF NODE NAME
 +fsck.ext2 20182 root    4u   BLK    7,0 0x3f903ef000 5941 /dev/loop0
 +python    23598 root    3r   BLK    7,0 0xdf917c7600 5941 /dev/loop0
 +</code>
 +
 +That's approximately 28%. :-(
 +
 +<code>
 + STARTED %CPU %MEM   RSS CMD
 +  May 13  0.0  0.0   364 su
 +  May 13  0.0  0.0   528  \_ bash
 +  May 13  0.0  0.0  1320      \_ watch lvdisplay /dev/vg_scratch/snap
 +  May 13  0.0  0.0   364 su
 +  May 13  0.0  0.0   536  \_ bash
 +  May 13  0.0  0.0   464      \_ fsck /dev/loop0 -y
 +  May 13 99.1 24.3 1488468          \_ fsck.ext2 -y /dev/loop0
 +  May 14  0.0  0.0  1456 watch lsof /dev/loop0
 +</code>
 +
 +So ''fsck'' has about 1.4GiB of resident memory (and 4696MiB of virtual memory according to top):
 +
 +<code>
 +  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                     
 +20182 root      30  10 4696m 1.4g  784 R   94 24.3  19564:51 fsck.ext2 -y /dev/loop0  
 +</code>
 +
 +Total system memory:
 +
 +<code>
 +# meermanr@Ikari:/home/meermanr (master *)
 +# free -m
 +             total       used       free     shared    buffers     cached
 +Mem:          5969       5036        933          0         87        157
 +-/+ buffers/cache:       4791       1178
 +Swap:        10234       4764       5470
 +</code>
 +
 +<code>
 +Every 2.0s: lvdisplay /dev/vg_scratch/snap                                       Mon May 27 14:54:06 2013
 +
 +File descriptor 4 (pipe:[72594953]) leaked on lvdisplay invocation. Parent PID 23517: sh
 +  --- Logical volume ---
 +  LV Name                /dev/vg_scratch/snap
 +  VG Name                vg_scratch
 +  LV UUID                4EFJ8Y-bzWT-aif4-MlT9-4234-aS1d-qcipq0
 +  LV Write Access        read/write
 +  LV snapshot status     active destination for /dev/vg_scratch/lv_scratch
 +  LV Status              available
 +  # open                 1
 +  LV Size                894.27 GiB
 +  Current LE             228934
 +  COW-table size         84.89 GiB
 +  COW-table LE           21733
 +  Allocated to snapshot  43.48%
 +  Snapshot chunk size    4.00 KiB
 +  Segments               2
 +  Allocation             inherit
 +  Read ahead sectors     auto
 +  - currently set to     256
 +  Block device           252:1
 +</code>
 +
 +Output from ''fsck'' itself:
 +
 +<code>
 +File ... (inode #9791282, mod time Thu Oct  5 01:40:26 2006) 
 +  has 11143 multiply-claimed block(s), shared with 5 file(s):
 + <filesystem metadata>
 + ... (inode #9791794, mod time Thu Oct  5 01:40:26 2006)
 + ... (inode #4115835, mod time Thu Aug 20 03:31:06 2009)
 + ... (inode #4130006, mod time Mon Nov 29 16:38:10 2010)
 + ... (inode #4784754, mod time Tue Jul 26 06:01:10 2005)
 +Clone multiply-claimed blocks? yes
 +</code>
unix/lvm_recovery.txt · Last modified: 2013/08/20 22:54 (external edit)

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International
CC Attribution-Noncommercial-Share Alike 4.0 International Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki