Linux Package and Boot Recovery

Package and boot failures are high-pressure because the normal management path may be broken. Recovery work needs conservative sequencing: preserve evidence, identify the last change, boot a known-good path, restore package consistency, and only then clean up.

Examples here assume Debian-family systems with Ubuntu Server as the default operational target.

Command Examples

cat /etc/os-release
uname -a
systemctl --failed
journalctl -xb
dpkg --audit
apt-mark showhold

Example output and meaning:

When the host cannot boot normally, collect the same facts from emergency mode, rescue media, serial console, or a mounted root filesystem.

Broken Packages

Package transactions can fail because maintainer scripts fail, dependencies conflict, disk fills, dpkg locks remain, or repositories changed.

Recovery sequence:

sudo dpkg --audit
sudo apt -f install
sudo dpkg --configure -a
sudo apt update
sudo apt install --reinstall <package>

Check logs:

less /var/log/apt/history.log
less /var/log/apt/term.log
less /var/log/dpkg.log

Do not remove packages blindly. A forced remove of core packages can turn a package issue into a boot issue.

Bad Kernel or Initramfs

Kernel updates can fail because the new kernel, module, DKMS build, firmware, storage driver, root UUID, or initramfs is wrong.

Useful commands from a working boot or chroot:

ls -lh /boot
dpkg -l 'linux-image*' 'linux-modules*'
lsinitramfs /boot/initrd.img-$(uname -r) | head
sudo update-initramfs -u -k all
sudo update-grub

If the latest kernel fails, boot the previous kernel from GRUB advanced options, then inspect /var/log/apt/history.log, DKMS logs, and kernel logs.

GRUB Rescue and Emergency Mode

Common entry points:

Useful emergency commands:

mount -o remount,rw /
findmnt /
blkid
lsblk -f
journalctl -xb
systemctl default

For filesystem or /etc/fstab problems, findmnt --verify and commenting a bad mount can restore boot while preserving a follow-up fix.

Chroot Repair

From rescue media:

sudo mount /dev/<root-volume> /mnt
sudo mount --bind /dev /mnt/dev
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys
sudo chroot /mnt
dpkg --configure -a
update-initramfs -u -k all
update-grub

If /boot or EFI System Partition is separate, mount it before chrooting.

Rollback Strategy

• Keep at least one known-good kernel installed.
• Snapshot before risky package, kernel, driver, or bootloader changes.
• Avoid unattended upgrades for kernel-sensitive fleets without staged rollout.
• Keep console access tested.
• Record package holds intentionally.
• Test restore and chroot procedures before incidents.

Runbook

1. Preserve console output and package logs.
2. Identify last package, kernel, driver, or bootloader change.
3. Boot previous kernel or rescue mode if possible.
4. Repair package state with dpkg --audit, apt -f install, and dpkg --configure -a.
5. Rebuild initramfs and GRUB only after confirming /boot, root UUIDs, and drivers.
6. Reboot once into the intended kernel and verify services, mounts, and networking.
7. Add a postmortem note with rollback and prevention changes.

Study Cards

References

• Debian package management reference
• Ubuntu recovery mode documentation
• update-initramfs(8)
• GRUB manual