Recovering DigitalOcean droplet landing on grub-rescue prompt

 One night we rebooted one of our DigitalOcean Ubuntu 18.04 droplet (VM) and after starting, the VM was giving an error and directly going to the grub rescue prompt.

The error was displayed as 

error: file /boot/grub/i386-pc/normal.mod not found

grub rescue> 

We used the ls command and it will show all the disk devices or partitions which were connected to our VM.

grub rescue> ls

(hd0) (hd0,gpt15) (hd0,gpt14) (hd0,gpt1) (hd1) (hd2)

grub rescue> ls (hd0,gpt1)/

We can see from the above output that our /boot directory is missing. 

As the /boot folder is missing or probably got deleted by mistake, we tried to restore the droplet backup also. But unluckily in available droplet backups also the /boot folder was missing and the droplet won't start.

So we decided to go with DigitalOcean recovery option. We stopped our VM and took a snapshot and proceeded to boot from Recovery ISO.

1) Go to the Recovery link in DigitalOcean console, after the VM is shut down. Select the Boot from Recovery ISO option. 

Turn the VM on and go to the recovery console.

Click on Launch Recovery Console.

2) Once you are in the recovery console, choose option 1. Mount Your Disk Image. This will mount our droplet's root volume.

3) Then choose option 6 to go to the Interactive Shell.

4) In the interactive shell, execute the below commands:

  a.   mount -o bind /dev /mnt/dev

b. mount -o bind /dev/pts /mnt/dev/pts

 c. mount -o bind /proc /mnt/proc

d. mount -o bind /sys /mnt/sys

e. mount -o bind /run /mnt/run

5) Change root for your mounted disk and go to droplet’s root directory.

chroot /mnt

6) Create the GRUB config file using the command: /usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg

7) Our droplet’s disk is /dev/vda and we will install GRUB on this disk /usr/sbin/grub-install /dev/vda

8) At this point, we can exit from the chrooted environment.

exit

9) Shutdown the VM and turn the VM on from the VM's hard drive. But in our case VM didn't boot and went to the grub> console.

10) To resolve this I rebooted the VM and again performed the above 1-7 steps.

After that performed an upgrade of the installed packages.

a) apt update

b) apt upgrade

But the apt upgrade command failed, with the below error:

Could not find /boot/grub/menu.lst file.

Would you like /boot/grub/menu.lst generated for you? (y/N)

/usr/sbin/update-grub-legacy-ec2: line 1101: read: read error: 0: Bad file descriptor

11) To resolve the error, I created the /boot/grub/menu.lst file manually.

touch /boot/grub/menu.lst

12) After that again I run the apt upgrade command.

Now the apt command showed me the below question for the/boot/grub/menu.lst file.

From the available option, select the first one, "install the package maintainer's version"

13) This time apt upgrade command was successful. 

After that, we can exit from the chrooted environment.

Use the exit command to exit from the chrooted environment.

14) Shutdown the recovery environment

shutdown -h now 

15) Start the VM after selecting the Boot from Hard Drive option from DigitalOcean's Recovery Link. 

This time our recovery was successful and the VM started without any issue.