+44 20 3287 1023
support@nicktailor.com
My Git Repos

How to recover file system corruption on 4T LVM using DDrescue on a VM

| May 31, 2018

How to recover file system corruption on 4T LVM on ubuntu using a VM

In this example we will be fixing a xfs filesystem that failed initial xfs_repair

If this happens don’t panic. We can fix most likely fix it.

Steps to do

• Increase vm memory to 32Gig of ram
• Add another disk that is the same size or even slightly larger as the lvm with the corrupted.
◦ Make sure you use a complete new datastore and that’s not being managed by DRS if your using vmware as this will be temporary.
• reboot the VM

.

Create new physical volume, volume group and logical volume

• Create a new physical volume so that it can be added to a new volume group
◦ Fdisk -l (to find the disk)
◦ Pvcreate /dev/sdx  <- replace x with drive designation
• Create volume group
◦ Vgcreate recovery /dev/sdx < – use whatever drive designation was used in above step
• Create logical volume
• lvcreate -l 100%FREE -n data1 $VGNAME
◦ lvcreate -l 100%free -n data1 recovery <- example
◦ mkfs.xfs /dev/recovery/data1
• Mount the new lvm
◦ Mount /dev/recovery/data1 /mnt/recovery

Now install ddrescue and make image of the corrupted file system on the new logical volume

◦ sudo add-apt-repository universe
◦ sudo apt update
◦ sudo apt install gddrescue

.

Make swap size 30gigs – this is needed so when we repair the filesystem it doesn’t time out because it runs out of memory. Which tends to be the problem when trying to repair such large filesystems.

◦ sudo dd if=/dev/zero of=/swapfile bs=30G count=2 <- move the /swapfile to a location where there enough room
◦ chmod 0600 /swapfile
◦ mkswap /swapfile
◦ swapon /swapfile
◦ swapon -s (should show the new swapfile)
Sample outputs
◦ Filename        Type    Size  Used  Priority
◦ /dev/sda5 partition  3998716  704  -1
◦ /swapfile file    2097148  0

Create rescue image on new logical volume

◦ ddrescue -d -r3 $oldfilesyetem imagefile.img loglocationpath.logfile

ddrescue -d -r3 /dev/recovery/data /mnt/recovery/recovery.img /mnt/recovery/recoverylog.logfile

◦ this will take awhile to run, probably hours

Once the file is created we want to repair it using xfs_repair

◦ xfs_repair -m 30000 /mnt/recovery/recovery.img
◦ this will also take awhile to run:

– agno = 29

– agno = 9

– agno = 10

– agno = 11

– agno = 12

– agno = 13

– agno = 14

– 20:02:48: check for inodes claiming duplicate blocks – 88951488 of 88951488 inodes done

Phase 5 – rebuild AG headers and trees…

– 20:02:57: rebuild AG headers and trees – 41 of 41 allocation groups done

– reset superblock…

Phase 6 – check inode connectivity…

– resetting contents of realtime bitmap and summary inodes

– traversing filesystem …

– traversal finished …

– moving disconnected inodes to lost+found …

Phase 7 – verify and correct link counts…

Done

◦ once complete you should be able to mount the image
◦ mount recovery.img /mnt/recovery/data1
◦ if successful it should mount under when you do df -h
◦ /dev/loop0 3.0T 1.2T 1.9T 38% /mnt/recovery/data1

.

Written By Nick Tailor

.

.

.

May 31, 2018 Centos, Diskstuff, Linux, RecoveryNo Comments »

Leave a Reply

Your email address will not be published. Required fields are marked *

0