This morning I woke up, rebooted a living room pc and got thrown into a 2-hour session of troubleshooting for a problem I do not understand why it exists. I’m writing this in hopes of understanding the whys, and how to avoid similar pitfalls.
I just recently installed a living room pc running Fedora 44, it’s running Plasma Big Screen, and it’s purpose is to be a steam link machine, jellyfin server and maybe a game server down the line for some coop games (zomboid, valheim…). For about a week, everything was perfect.
Until this morning.
After turning on my tv, my system was showing some errors on qbittorrent, and I decided to reboot just in case. And that was when my system just completely locked me out, it threw me into emergency mode and I had no access to root, so nothing could be done, just watch an endless loop of my system trying to do something that was impossible and occasionally pressing enter to restart the loop. That is my first gripe: why throw someone into emergency mode if it’s just going to lock them out?
I tried restarting a few times, unplugging things, reseating ram and the likes just in case. When nothing worked, it seemed I’d have to do research, in the hopes of not having to wipe it clean and start anew.
So, here I went, searching the web with my problem and trying to find a solution. After reading some very long forum posts, I apparently needed more information about what had actually caused this, but it was likely something about fstab. And here is my second gripe: why did the system not immediately inform me of the error first before starting emergency mode? I got 0 error messages because the default setting is Rghb quiet… Is this a thing about just fedora or is every linux distro the same?
(I’m going to add in here that I’m in the process of switching all my pcs to Linux, and this was a first test. But I also am going to switch my family’s pcs, and I need to shine my Linux shoes and put some big boy IT pants for the future, so that’s why I’m writing a post: to learn from your experiences)
So here I go, to do some stuff with GRUB to find the error. I decided to test chatgpt and see if it could guide me (I’m a noob if it wasn’t obvious yet), and took more than an hour of troubleshooting with grub and bash to finally see that the problem was about a drive with an UUID that did not match my system drive (a silver lining I guess). But, here’s the thing, as soon as the reboot loop started, I had an inkling of a suspicion that it might have been one of the old spinning hdds causing it (I need to replace those, but they’re fine and working for now, and in this economy…). So I had unplugged all of them when I did my hardware troubleshooting step, and kept only my nvme disk (which is brand new) on the system up to here. So I had been completely blindsided that even if the drives are disconnected, my system still won’t boot, because it expects the drives to be there, and if they aren’t, even though everything else is working fine, it won’t boot! This is my third gripe. Is this a default setting? Something about Fedora? Why is this the way it is done? It just doesn’t seem logical to me to lock me out of the whole system because a non-essential part is not working/present.
Anyways, after unplugging and re-plugging the drives, I finally discovered it was not my drive, but a pcie sata expansion card that had timed out, and it was this one smaller drive I had been using with the card that was the problem, but after plugging it straight to the mb (the slots are precious, okay? I was saving them for bigger drives in the future), it worked just fine. My system booted normally.
That was 2 hours-ish that could have been just 5 minutes if the system had actually told me it was having problems with connecting to a drive. Also, chatgpt did help, but boy, it didn’t have a good troubleshooting order at all. It was just shooting in every direction and hoping something would stick. But I don’t think trying to find my fix in forum posts would have been any better.
This is very frustrating, and I’ve been there myself by accident but the system does this for good reason. When you have a drive listed to automount in fstab without the “nofail” flag set, if the drive is unavailable the system will boot into emergency mode. This is because without “nofail” it is deemed the drive is an essential part of the system, and Linux won’t boot to prevent unintended damage. A drive might be unplugged or it might be failing and corrupted; but the OS has no way of knowing why the drive isn’t there; it juts knows it’s essential and locks itself down to allow you to fix the problem and prevent damage.
Emergency mode is a bit daunting, but booting into it when a essential drive is missing is actually a safety feature to prevent you experiencing data loss or unintended consequences.
The real issue here is not how fstab works, but rather wherever you got guidance on auto-mounting a drive in Linux. You have to actively set up a drive to automount (linux by default only automounts the root system) - so either you yourself edited the fstab or you used a GUI tool to do it. Fedora does not do this automatically. Whatever guide you followed is the problem here: A good guide should have explained how fstab works, and particularly what “nofail” is and how important it is. If it was a GUI method (like Gpart of KDE Partition Manager), then they also include methods to mark a drive as non-essential (in KDE Partition Manager for example, when setting up a mount point you’d need to ensure “Don’t prevent the system from booting if not mountable”). The guide you followed should have flagged this things up. I know it seems harsh but brutally, you did this. It can be a tough pill to take with linux but the more you tinker, the more you are responsible for when things go wrong; so it’s honestly really useful to actually learn how the things you’re changing work.
If you’d like to understand fstab, the Arch linux wiki has a decent overview: https://wiki.archlinux.org/title/Fstab
It links through to a page on mount options in fstab for ext4 file system, which includes nofail and many other options: https://man.archlinux.org/man/ext4.5#Mount_options_for_ext4I would go through your fstab and ensure that any other auto-mounted drives not essential for boot (e.g. media drives or external drives or USB drives) are marked as nofail. Last thing you want is being locked out of yourself if an unessential drive does fail, as emergency mode can be tedious way to problem solve that issue. But before you make any changes, back up your fstab file, and also briefly learn how to use emergency mode (you probably already have learnt how to manually mount the root file system in emergency mode from Chatgpt, but this is an essential first step to fix most problems in emergency mode)
It just doesn’t seem logical to me to lock me out of the whole system because a non-essential part is not working/present.
It’s a fair criticism - but the system doesn’t know they are non-critical. You’ve, effectively, told it they are critical in fact.
You can add a nofail option to the drives that aren’t needed to boot in /etc/fstab.
e.g.
UUID=9d1f7f1b-2a78-4b92-a8d5-0b05e7273920 /mnt/backup btrfs defaults,nofail,subvol=@backups 0 0I use this for a disk that’s external and sometimes not connected.
I find that behavior weird too, and had a few problems with it with failed sd cards in raspberry-like devices. By the way. did you put the nofail boot option in fstab to avoid this in the future?
No, but I will now lol
If you enjoy the experience of learning how to fix stuff and how Linux works, stay with classic distros like the Fedora you are rocking. If you don’t give a shit and just want it to work, I advice you to switch to an immutable/atomic distro. If Gaming is important, I suggest Bazzite.
Otherwise, provide logs and we’ll help you out.
just want it to work,
Unless you’re doing software development and using lots of tools that need dbus or ever plan to do anything other than “use a browser and flatpaks”. Then you get to figure out how to fight with distrobox to make things interoperate with your desktop environment and other distroboxes.
And I doubt bazzite would have dealt as well with a missing disk.
Here’s how you could have figured it out yourself:
Boot something else, read the logs.
Use a usb to boot some other operating system (or the same one) and access the logs of your system which will show where it’s failing and how.
Another person put that you could add nofail to fstab, you could also do this by moving the devices or services to nonessential branches of systemd.
Only 2 hours? /s
Next time use Gentoo. Your life will be ruined forever but I assure you none of those “defaults” are imposed… And you get plenty of error messages oh yeah
I need to replace those, but they’re fine and working for now, and in this economy…
that’s why i haven’t bought any parts to rebuilt my everything home server. lol
and the more “exotic” your setup is – and use the word exotic very loosely here – the more time you have to spend thinking of how exotic it is compared to the plain-jane systems exist out there and compensate for it.
for example: most computers have a permanent drives, so if any of your drives are not permanent; your system automatically becomes exotic.
not using hdmi, vga, display port or hdmi; youi’re also exotic.
is your keyboard not the normal 104 key; you’re also exotic.
using pata & sata together; you’re also exotic.
…
That’s also a trip. My keyboard is not the standard 104 american layout, because I’m not American, and while using GRUB I had to discover what keys actually stood in for symbols like / and | because it had reverted back to standard layout.
Idk man, seems like you installed at least the system files on the wrong drive, but that would be obvious and idk how you would even do that with linux installing fedora, pretty sure there is no option to have home/data on a seperate drive
You wouldn’t have this problem with a proper NixOS config.
fstab boot issues were the reason I made the switch.
NixOS config just sets Fstab. The issue was “nofail” was not set. This config calls that out which is good, but the issue isn’t fstab, it’s wherever OP got guidance on automounting drives in fstab as that guidance clearly failed to highlight the importance of the nofail mount option.
Don’t run before you can walk.
Fstab is how it works in the background, best to learn that first.
How would using NixOS have made OP’s system immune to the critical hardware issues that they discovered were the root cause making their computer fail to boot?
I’ll look into it! Fstab seems like… something.
The issue isn’t fstab. The nixos thing posted there would have the exact same issue: if nofail wasn’t included in the configuration, the system would go to emergency mode when it can’t find a drive.
Fstab is the “File System Table”, and NixOS still uses it. In your case, a non-essential auto-mounting drive was not correctly flagged as “nofail” in your Fstab; when the system couldn’t find that drive it treated it as essential and locked the system down.
Don’t, they’re a religious follower. You did well.
