I’m starting to get good at this. Fired up an Ubuntu 18.04 LTS VPS and moved everything. More details (maybe?) to follow.
Update: I’ve found a few more hits on this muhsti thing, like this one. And the process came back at least once, I deleted a few more locations – more research needs to be done on this. When Ubuntu 18.04 is released, I’ll probably migrate to a new server anyway.
Second update: I’ve restored a week-old backup onto a new Linode, and pointed my domain at the new IP. Going to harden what I have here, and hope that this won’t happen again. Also going to back up now, after re-adding this post.
Waking up to emails from Linode doesn’t always mean something bad is happening, but sometimes it does.
Here’s a shot of my WordPress directory (which is the root of my apache site). Anything look odd? Does any of this scream “malware” to you?
How My Day Started
Every morning when I wake up, I have a habit of checking my email first thing. This morning, there were several automatically generated emails from Linode (my VPS host) indicating that my CPU usage was at 100% for an extended period of time. All I run on this server is my LAMP stack and this wordpress site, so this is unusual. My first thought was some kind of runaway process, so I logged into the Linode manager and rebooted the server. I figured there was a good chance that whatever happened, this would fix it. Certainly wasn’t expecting malware. It also takes <30 seconds, and just a click of one button. An hour or two later, I get another email.
So, I walk downstairs, make a cup of coffee, and sit down at my computer. I ssh into my site, and run htop. First thing that I see is two instances of a command “muhsti” which is at the top of the list and using up all of my CPU. I knew right away that this wasn’t a command that I normally have running, and it was being run under the www-data user, so it had something to do with the web server. Normally the only processes running under that user are the LAMP processes, like apache and mysql. Big red flag.
(Note: this is a picture from a user on serverfault, I didn’t take a screenshot when I first discovered this process. Luckily, the process on my server had only been running for a few hours, not a very long time like on this user’s server.)
After some searching, I found this. Turns out it’s a crypto mining malware, and at least one other person has written about it. This is the only source I can find about this process. Turns out, this process mines monero and sends it back to a third party. After reading through the info, it seemed fairly straightforward. But how did I get it?
Chasing Down the Cause
The only other mention I was able to find of “muhsti” was a serverfault page which linked to the above page. This user mentioned that it came in through a WordPress plugin called muhstikx86. I logged into the dashboard, looked at my installed plugins, and didn’t see it. Okay, so it’s not an overtly obvious plugin. I thought, “I’ll take a look in the plugins directory in case there is something there that seems out of place.”
Looks normal. Went to the root directory:
Wait – what’s plugins? I try to go into the directory. Not a directory. Go to edit the file – bunch of gibberish. That’s my first (although crude) way of telling something is an executable, not a script or a text file. Found it. Don’t know how it got there, but I found it. So, of course, I removed it.
I also re-read the article I posted above, and looked to the other sources of the problem – the way that the miner replicates itself, and the method by which (cron) it triggers itself. Here’s the cron process it entered:
I went through and deleted everything related to it. I still don’t know exactly where it came from, but as I’ve had htop open for a while, and haven’t seen the process resurface, I think – for now at least – I’ve gotten rid of it. I also changed the permissions on the www-user crontab, so that only root can write to it. An inelegant solution, but as of now, I have no cron jobs that the www-user user needs to be running. This will prevent replications like this in the future.
Where do I go from here?
I think I need to revisit the permissions of not only the files/folders in my webroot directory, but also the permissions of the www-data user. I don’t want them to be able to write to /dev/shm, I don’t think at least, (I’m going to keep reading up on this) and I don’t want it to generate cron jobs.
Hit me with some comments if you have any input, advice or ideas. Curious to see how widespread this is, and maybe where it came from originally. Perhaps I’ll never know.
So, it’s been a long time without any news – time for an update. A new job, and all kinds of other things have gotten in the way. I’ve made some changes to try to be as open source, Linux, and privacy-minded as I can.
Due to a whole range of problems and issues surrounding updating from Ubuntu 14.04 LTS to 16.04 LTS, I’m in the process of migrating to a new Linode that I’ve set up. This site will be going up and down over the next few hours, and will also probably be throwing out some SSL certificate errors as I get everything moved over and troubleshoot.
So, on 10/21/2016 I was in a car accident. I was the passenger, and the driver I was with wasn’t at fault. I completely borked my knee.
That, in addition to my new job, is why I haven’t been able to update the site much in the past little while. Continue reading The Accident and My Knee
I know I’ve been laid up for a little while, and I know that Fedora 25 has been out for a few weeks, but here’s my updated guide for installing and configuring Fedora 25 on the Thinkpad X1 Carbon (2015). Not much has changed, in fact it becomes a little easier every release…more and more of the hardware is supported by the kernel by default, and Gnome is getting better with every release at dealing with HIDPI displays. (Of course, you can ignore the HIDPI stuff if you got the model with the 1080p display.) I still have the same issue with the fingerprint sensor – but it’s a minor issue. It lights up any time the computer asks for a password (including in the terminal) and doesn’t turn off – even after you type in your password – until you run your finger across it. Again, minor issue.
Not much has changed since I wrote the last one, so a lot of this is largely copied and pasted from my last guide. Continue reading Fedora 25 Post-Install Guide: Thinkpad X1 Carbon (2015)
Well, it’s been a long time…almost 6 months…since I’ve posted. That’s quite a hiatus. A lot has gone on. Fedora 25 is out, I have a new job, I’ve cooked a lot, and I’ve been in a car accident.
My old job involved my having a ton of spare time – plus my website wasn’t blocked at work. That means I could cook something or mess with a computer at home, and I could just go to work the next day and write up an article. Not so anymore.
But, now I’m working from home for a while – since I got in the accident – so I have some time where I can update. I plan to add more later, I have some frozen meal reviews to write, for instance.
But for now, I’ll just leave you with a picture of my knee.
I’ve been searching for a while for a good docking solution for my X1 Carbon…and I’ve settled on Lenovo’s Thinkpad OneLink Pro dock. I did a lot of searching since I run and wanted it to work with Linux, and I found some outdated and/or inaccurate data from over the past several years. I wanted to add a new data point, and hopefully it will help someone else in a similar situation.
Here’s the dock:
As promised, here is my post-install list for the Thinkpad X1 Carbon (2015 model, 3rd gen). I spent most of yesterday evening and this afternoon getting things set the way I want, and I love it so far. I have one nagging issue – the fingerprint sensor – but everything else is working well. I’ll address my issue at the bottom.
A few things have changed since I wrote the last one, so I’m going to basically start from scratch. Continue reading Fedora 24 Post-Install Guide: Thinkpad X1 Carbon (2015)
Fedora 24 is finally officially released. I’ll outline a little about how to upgrade, but I won’t be able to upgrade my own machines until I get home from work.
Stupid work, getting in the way of my Linux upgrades.
Anywho, let’s take a look at the upgrade process to get from 23 to 24 – I’ll just add this for informational purpose, as I’m a stickler for clean installs. Plus, my 23 installs are fairly recent, so I won’t be losing a ton of stuff there…so I don’t have much of a reason to not to a fresh install.