So…long story short, I hosed my main system. I think it happened when I was trying to troubleshoot a problem with the newer version of the kernel. But I learned something about the home directory that may seem obvious to many.
If you look at one of my last posts (it’s been a little while…sorry…) you’ll see where I used the FedUp utility to upgrade my main machine from Fedora 21 to Fedora 22. Everything worked really smoothly. I was worried since I had read a few things about the kernel mods used by the Nvidia driver (I have a GTX 780TI) had trouble with kernel upgrades. But, it hadn’t given me trouble between 3.17-4.0.4, so I thought everything was fine.
However, over the past several weeks, I had done my regular updates and had trouble booting…it happened when I installed 4.0.6, 4.0.7, and finally earlier today I updated to 4.0.8. I couldn’t finish the boot process…it would get to the point of starting up GDM, then the “oops, something bad happened” screen would show up (I’m paraphrasing, I don’t remember exactly what it said) and it would go back to the console. I couldn’t even get far enough to Alt-Ctrl-F2 to another virtual console…it would just hang. So, I ended up booting back into 4.0.5. I went through this with each update between 4.0.6 and 4.0.8, hoping it was a kernel bug affecting my hardware and that the next update would fix it. However, when it came time to install 4.0.8 I really wanted to make sure I was figuring out what was going on.
I noticed with Fedora that it only keeps (by default) three kernel entries in storage…so if I installed 4.0.8, I would lose 4.0.5, which is the one that still worked for me. First thing I had to do was figure out how to keep old kernels. I found a few articles talking about how to keep a specific kernel, but I wanted to increase the number of kernels it kept in general. Here’s how I did it:
Edit the /etc/dnf/dnf.conf file (note that in F22 dnf has replaced yum as the CLI package manager…this was a tough change at first, but I like dnf a hell of a lot better now that I have gotten used to it.
sudo nano /etc/dnf/dnf.conf
Change the below parameter from 3 to 0 (zero keeps all until you delete them manually, or you could pick a different number, like 5 or 347):
installonly_limit=0
This will keep all of the old kernels. Okay, now it’s time to cross my fingers and hope that 4.0.8 will fix all of my troubles.
It didn’t.
Now I’m pissed. Time to google (verb)!
So it seems it had something to do with my Nvidia drivers…okay, tricky…let’s see what we can do. Uninstalled Nvidia drivers, rebooted.
Now I’m defaulting to the “crapeau” nouveau drivers. They work! But I can only get 800×600 on my 1440p monitor. (The other monitor went into sleep mode). Well, xorg started, at least. Okay, let’s reinstall the drivers from rpmfusion.
Now it doesn’t work again. I tried several different versions, installed and uninstalled, and even made it so that booting back into 4.0.5 didn’t work either. Finally, I uninstalled them and resigned to booting up with Nouveau, getting to the command line via Alt-Ctrl-F2, and backing up my entire home directory to the secondary SATA drive in the machine. Screw it – I’m nuking the whole install and starting over. When I upgraded from F21 to F22, I was thinking about doing a fresh install eventually…here’s my chance.
While the ~50GB of my home directory was copying, I started doing some thinking…What do I have worth saving? The only thing I can think of that wasn’t in my home directory was my Steam library…which was also located on the spare SATA drive in the machine, but that’s easy. Steam lets you pick up where you left off by scanning a library folder. Once I reformatted (as long as I made sure the installer didn’t touch that drive) I should be able to point to the library folder and be good to go. What else? SAVED GAMES. Wait – they’re all in my Steam folder. But they’re not all in one place – some are hidden in the home directory in ~/.local/share/steam or some such…whatever, I’ll sort all that out later. In the meantime, I’ll just copy the whole damn folder.
While ~/ was copying, I went ahead and downloaded F22 via bittorrent and dd’d it to a USB drive. Which reminds me – dd is a scary command…but to me, what’s the scariest about it is not knowing the progress…(Is it still working? Or did it lock my system up?) I found a neat little workaround…have two terminal tabs or windows open. In one, issue the dd command, i.e.:
sudo dd bs=4M if=~/fedora.iso of=/dev/sdb
Now, in another terminal, type the following:
watch -n5 'sudo kill -USR1 $(pgrep ^dd)'This looks like it kills the process – but it doesn’t. Instead, it shows the amount copied (in the original terminal, the one you used dd in) every 5 seconds. Handy!
SO, now I am armed with the Fedora 22 bootable USB, and my home directory having been backed up to another drive. Let’s do this.
Boot off the live USB, everything looks great. Both monitors working at native resolutions, everything seems good to go. Run through the install…easy as always. Made sure to only have the installer wipe and re-partition the SSD, not the “spare” drive with backup data, and it copies files to disk, etc, etc. Time to reboot.
Now it hangs again. No weird error message…but what happens is after I log in using the GUI, it just hangs there with a mouse pointer (which doesn’t move by the way) on the grey background. What gives? I thought maybe I just needed to shut down completely and boot up again. (Don’t ask me why, but doing things like this has helped and worked for me in the past…it makes no sense whatsoever.)
Same thing. WTF?!?!
I try to calm myself down, and I decide to go to the console and see what I can do. Ctrl-Alt-F2. Nothing. Reboot. Maybe if I go to console prior to logging in it will be different. So, before clicking my name and typing in my password, I go to console. I get a login prompt. Okay, this looks good. So I log in, and I’m at bash.
First thing I do is run all of the updates. Maybe the newest kernel will fix this (since it worked out for me so well the last time I used this logic). F22 ships with 4.0.4 (I think) so it had a LOT of updates to run, installing 4.0.8. Took a little while…but same issue. I think it might be the Nvidia drivers (again). So…what’s the easiest way to install them?
You can download and install from Nvidia, but I decided to use RPMFusion. My logic was that since that was part of a repository, they would help me keep the package up to date…maybe this would prevent problems in the future. So, after a little googling, I installed the RPMFusion repository via CLI:
su -c 'yum install --nogpgcheck http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm'
Okay, this is a good sign. Now let’s try installing the Nvidia drivers:
yum install akmod-nvidia "kernel-devel-uname-r == $(uname -r)" yum update -y
Reboot. HOLY SHITSNACKS IT WORKS. Okay, time to get down to business. Following my normal new-install routine, and install Fedy. (This guy has saved me so much time. Fedy is really an amazing application…simplifies so much. So then I go through and install all my normal applications, like Chrome, Sublime Text, nano (because I just can’t get into vim or emacs, sorry) and a few others. Up and running in no time.
So this leads me to the main point of the post:
The Home Folder Is Amazing.
Every time I would install an application, I would just copy the dotfiles from the backed up home directory into the new one. Suddenly all my preferences and links where there.
I thought it would be a good idea to ssh into turngren.net to run updates. Oh no! I can’t log in because I set it to use an RSA key instead of a password! Copied my .ssh folder…done. Fixed. Next!
Steam library pulls up perfectly from the spare drive, preferences are all there, everything works like a charm, and aside from the fact that I haven’t yet changed my desktop wallpaper to the same one I was using before, you almost can’t even tell that my system had been formatted.
Hopefully someone will learn something from this, if only how to save time and not troubleshoot with a heavy hand like I did…although it took that amount of frustration to really find appreciation for how the home directory is used.