{"id":711,"date":"2018-05-31T12:37:57","date_gmt":"2018-05-31T12:37:57","guid":{"rendered":"https:\/\/www.nicktailor.com\/?p=711"},"modified":"2022-10-21T11:42:14","modified_gmt":"2022-10-21T11:42:14","slug":"how-to-recover-file-system-corruption-on-4t-lvm-on-ubuntu-using-a-vm","status":"publish","type":"post","link":"https:\/\/nicktailor.com\/tech-blog\/how-to-recover-file-system-corruption-on-4t-lvm-on-ubuntu-using-a-vm\/","title":{"rendered":"How to recover file system corruption on 4T LVM using DDrescue on a VM"},"content":{"rendered":"

How to recover file system corruption on 4T LVM on ubuntu using a VM<\/span><\/strong><\/p>\n

In this example we will be fixing a xfs filesystem that failed initial xfs_repair<\/p>\n

If this happens don\u2019t panic. We can fix most likely fix it.<\/p>\n

Steps to do<\/p>\n

\n
\n
\u2022<\/span> <\/span>Increase vm memory to 32Gig of ram<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>Add another disk that is the same size or even slightly larger as the lvm with the corrupted.<\/div>\n
\n
\n
\u25e6<\/span> <\/span>Make sure you use a complete new datastore and that\u2019s not being managed by DRS if your using vmware as this will be temporary.<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>reboot the VM<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Create new physical volume, volume group and logical volume<\/span><\/p>\n

\n
\n
\u2022<\/span> <\/span>Create a new physical volume so that it can be added to a new volume group<\/div>\n
\n
\n
\u25e6<\/span> <\/span>Fdisk -l (to find the disk)<\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>Pvcreate \/dev\/sdx  <- replace x with drive designation<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>Create volume group<\/div>\n
\n
\n
\u25e6<\/span> <\/span>Vgcreate recovery \/dev\/sdx < – use whatever drive designation was used in above step<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>Create logical volume<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>lvcreate<\/span> -l 100%FREE -n data1 $VGNAME<\/span><\/div>\n
\n
\n
\u25e6<\/span> <\/span>lvcreate -l 100%free -n data1 recovery <- example<\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>mkfs.xfs \/dev\/recovery\/data1<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\u2022<\/span> <\/span>Mount the new lvm<\/div>\n
\n
\n
\u25e6<\/span> <\/span>Mount \/dev\/recovery\/data1 \/mnt\/recovery<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Now install ddrescue and make image of the corrupted file system on the new logical volume<\/p>\n

\n
\n
\u25e6<\/span> <\/span>sudo<\/span> add-apt-repository universe<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>sudo<\/span> apt update<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>sudo<\/span> apt install <\/span>gddrescue<\/span><\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Make swap size 30gigs \u2013 this is needed so when we repair the filesystem it doesn\u2019t time out because it runs out of memory. Which tends to be the problem when trying to repair such large filesystems.<\/p>\n

\n
\n
\u25e6<\/span> <\/span>sudo<\/em><\/span> dd if=\/dev\/zero of=\/<\/span>swapfile<\/span> bs=30G count=2<\/span> <- move the \/<\/em><\/span>swapfile<\/em><\/span> to a location where there enough room<\/em><\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>chmod<\/span> 0600 <\/span>\/<\/span>swapfile<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>mkswap<\/span> \/<\/span>swapfile<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>swapon<\/span> \/<\/span>swapfile<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>swapon<\/span> -s (should show the new <\/span>swapfile<\/span>)<\/span>
\nSample outputs<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>Filename<\/span>\u2003\u2003<\/span>\u2003\u2003<\/span>\u2003\u2003<\/span>\u2003\u2003Type<\/span>\u2003\u2003<\/span>\u2003\u2003Size<\/span>\u2003\u2003Used<\/span>\u2003\u2003Priority<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>\/dev\/sda5 partition<\/span>\u2003\u20033998716<\/span>\u2003\u2003704<\/span>\u2003\u2003-1<\/span><\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>\/<\/span>swapfile<\/span> file<\/span>\u2003\u2003<\/span>\u2003\u20032097148<\/span>\u2003\u20030<\/span><\/strong><\/div>\n<\/div>\n<\/div>\n

Create rescue image on new logical volume<\/span><\/p>\n

\n
\n
\n

\u25e6<\/span> <\/span>ddrescue<\/span> -d -r3 $<\/span>oldfilesyetem<\/span> imagefile.img<\/span> loglocationpath.logfile<\/span><\/p>\n

ddrescue<\/span> -d -r3 <\/span>\/dev\/recovery\/data \/<\/span>mnt<\/span>\/recovery\/<\/span>recovery.img<\/span> \/<\/span>mnt<\/span>\/recovery\/<\/span>recoverylog.logfile<\/span><\/p>\n<\/div>\n

\n
\n
\u25e6<\/span> <\/span>this will take <\/span>awhile<\/span> to run, probably hours<\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

Once the file is <\/span>created<\/span> we want to repair it using <\/span>xfs_repair<\/span><\/p>\n

\n
\n
\u25e6<\/span> <\/span>xfs_repair -m 30000 \/<\/span>mnt<\/span>\/recovery\/<\/span>recovery.img<\/span><\/div>\n
\n
\n
\u25e6<\/span> <\/span>this will also take awhile to run:<\/div>\n

– agno = 29<\/p>\n

– agno = 9<\/p>\n

– agno = 10<\/p>\n

– agno = 11<\/p>\n

– agno = 12<\/p>\n

– agno = 13<\/p>\n

– agno = 14<\/p>\n

– 20:02:48: check for inodes claiming duplicate blocks – 88951488 of 88951488 inodes done<\/p>\n

Phase 5 – rebuild AG headers and trees…<\/p>\n

– 20:02:57: rebuild AG headers and trees – 41 of 41 allocation groups done<\/p>\n

– reset superblock…<\/p>\n

Phase 6 – check inode connectivity…<\/p>\n

– resetting contents of realtime bitmap and summary inodes<\/p>\n

– traversing filesystem …<\/p>\n

– traversal finished …<\/p>\n

– moving disconnected inodes to lost+found …<\/p>\n

Phase 7 – verify and correct link counts…<\/p>\n

Done<\/p>\n<\/div>\n<\/div>\n<\/div>\n

\n
\u25e6<\/span> <\/span>once complete you should be able to mount the image<\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>mount recovery.img \/mnt\/recovery\/data1<\/div>\n
\n
\n
\u25e6<\/span> <\/span>if successful it should mount under when you do df -h<\/div>\n<\/div>\n
\n
\u25e6<\/span> <\/span>\/dev\/loop0 3.0T 1.2T 1.9T 38% \/mnt\/recovery\/data1<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

.<\/span><\/span><\/p>\n

Written By Nick Tailor<\/p>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n

.<\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

How to recover file system corruption on 4T LVM on ubuntu using a VM In this example we will be fixing a xfs filesystem that failed initial xfs_repair If this happens don\u2019t panic. We can fix most likely fix it. Steps to do \u2022 Increase vm memory to 32Gig of ram \u2022 Add another disk that is the same size or evenRead More …<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[58,125,138,49],"tags":[],"class_list":["post-711","post","type-post","status-publish","format-standard","hentry","category-centos","category-diskstuff","category-linux","category-recovery"],"_links":{"self":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/711","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/comments?post=711"}],"version-history":[{"count":17,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/711\/revisions"}],"predecessor-version":[{"id":1605,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/posts\/711\/revisions\/1605"}],"wp:attachment":[{"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/media?parent=711"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/categories?post=711"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nicktailor.com\/tech-blog\/wp-json\/wp\/v2\/tags?post=711"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}