Recovering after a root-partition resize gone wrong

Recently I tried to resize a LVM logical volume with its volume group and the physical disk belonging to the vg. Because the logical volume contained all partitions except for /boot I knew it would be hard.

I also knew that I had no backup and no snapshot of the virtual machine. :)

Here’s the initial situation:

root@master:/home/sebastian# fdisk -l
Device     Boot  Start       End   Sectors  Size Id Type
/dev/sda1  *      2048    499711    497664  243M 83 Linux
/dev/sda2       501758 204797951 204296194 97,4G  5 Extended
/dev/sda5       501760 204797951 204296192 97,4G 8e Linux LVM

root@master:/home/sebastian# vgdisplay
  --- Volume group ---
  VG Name               master-vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               97,41 GiB
  PE Size               4,00 MiB
  Total PE              24938
  Alloc PE / Size       24938 / 97,41 GiB
  Free  PE / Size       0 / 0
  VG UUID               yka0ZI-k7Q3-aeYD-vxU7-xajX-OVOy-qyogee

root@master:/home/sebastian# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda5
  VG Name               master-vg
  PV Size               97,42 GiB / not usable 0
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              24938
  Free PE               0
  Allocated PE          24938
  PV UUID               TDHGY0-t1IO-qKuf-BQNf-fsWo-abut-PHZ8KJ

Note that sda5 is a logical partition on top of the extended partition sda2.

To resize the disk I used sfdisk because I read that fdisk alone couldn’t do the job.

Actually I followed this tutorial on Stack Exchange: https://unix.stackexchange.com/questions/379549/resize-extended-partition-lvm/379681#379681

I’ll paste the answer below for completeness:

To extend your sda5, you need to extend its container too, sda2. Using command-line tools, the simplest way to do this is to use sfdisk:

sfdisk /dev/sda

This will print the current partition table, which should match what you saw in fdisk:
Disk image: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe59ec859

Old situation:

Device     Boot  Start       End   Sectors  Size Id Type
image1     *      2048    499711    497664  243M 83 Linux
image2          501758 104855551 104353794 49.8G  5 Extended
image5          501760 104855551 104353792 49.8G 83 Linux

Type 'help' to get more information.
At the >>> prompt, start re-defining all your partitions:
2048,497664,83,*
(this is the first partition: start sector, length in sectors, type, and a * to make it bootable). sfdisk will respond with
Created a new DOS disklabel with disk identifier 0x03408377.
Created a new partition 1 of type 'Linux' and of size 243 MiB.
      image1 :         2048       499711 (243M) Linux
and prompts for image2 (sda2). Enter
501758,,5
which tells sfdisk to create an extended partition starting at sector 501758 and occupying all the available space after that; sfdisk will output
Created a new partition 2 of type 'Extended' and of size 99.8 GiB.
      image2 :       501758    209715199 (99.8G) Extended
For image3, enter simply
501760
sfdisk will then output
Created a new partition 5 of type 'Linux' and of size 99.8 GiB.
      image5 :       501760    209715199 (99.8G) Linux
and prompt for image6, which we don’t need, so enter
quit
which will cause sfdisk to print the new partition table and ask if you want to write it to disk:
New situation:

Device     Boot  Start       End   Sectors  Size Id Type
image1     *      2048    499711    497664  243M 83 Linux
image2          501758 209715199 209213442 99.8G  5 Extended
image5          501760 209715199 209213440 99.8G 83 Linux

Do you want to write this to disk? [Y]es/[N]o:
If you’re convinced the starting sectors all match, and sda1 is still OK, press Y to write the partition table and return to your shell.

Once that’s done, run
pvresize /dev/sda5
to resize your LVM PV; you should then be able to use the newly-allocated disk space (in new LVs, or to extend existing LVs).

After doing the above steps I rebooted and of course the server did not come back up. I connected with the vmware kvm tools to the server and saw an error message (among others) that told me volume group not found. It seemed I destroyed the partition layour or disk.

To fix this issue I mounted a live-CD into the server and booted into it. There I had access to the disk and could try repairs.

The LVM tools (pvscan, pvck) did not find any information regarding lvms on the disks:

> pvck /dev/sda5
Could not find LVM label on /dev/sda5

Also there was no backup in /etc/lvm/backup that I could use. But the internet told me that the configuration of the lvm is stored in the first bytes/sectors of the partition and that it is possible to restore this configuration with dd:

dd if=/dev/sdc3 bs=512 count=255 skip=1

And this did work! I spat out the configuration that contained the volume group, physical disk and logical volumes on this disk.

I then used pvcreate to recreate the physical disk and make it known to the lvm. I don’t really know if this command was necessary as I did use vgcfgbackup to restore the whole configuration.

> pvcreate --uuid "U3qzRV-jakK-0JFC-zVmq-F1aa-nxrg-9JYsXy" --restore /etc/lvm/backup/testvg /dev/sda5
> vgcfgrestore master-vg

vgcfgrestore restores the metadata of a VG from a text back up file produced by vgcfgbackup. This writes VG metadata onto the devices specifed in back up file.

After that I could see the logical volume again and everything looked just like before I tried to resize the partition. I did one final reboot and the system came back online! My disaster recovery worked! Now all I need to do is resize the disk…

Some useful links I found during my repair attempts:

Related posts: