Update: I’ve found a few more hits on this muhsti thing, like this one. And the process came back at least once, I deleted a few more locations – more research needs to be done on this. When Ubuntu 18.04 is released, I’ll probably migrate to a new server anyway.
Second update: I’ve restored a week-old backup onto a new Linode, and pointed my domain at the new IP. Going to harden what I have here, and hope that this won’t happen again. Also going to back up now, after re-adding this post.
Waking up to emails from Linode doesn’t always mean something bad is happening, but sometimes it does.
Here’s a shot of my WordPress directory (which is the root of my apache site). Anything look odd? Does any of this scream “malware” to you?
How My Day Started
Every morning when I wake up, I have a habit of checking my email first thing. This morning, there were several automatically generated emails from Linode (my VPS host) indicating that my CPU usage was at 100% for an extended period of time. All I run on this server is my LAMP stack and this wordpress site, so this is unusual. My first thought was some kind of runaway process, so I logged into the Linode manager and rebooted the server. I figured there was a good chance that whatever happened, this would fix it. Certainly wasn’t expecting malware. It also takes <30 seconds, and just a click of one button. An hour or two later, I get another email.
So, I walk downstairs, make a cup of coffee, and sit down at my computer. I ssh into my site, and run htop. First thing that I see is two instances of a command “muhsti” which is at the top of the list and using up all of my CPU. I knew right away that this wasn’t a command that I normally have running, and it was being run under the www-data user, so it had something to do with the web server. Normally the only processes running under that user are the LAMP processes, like apache and mysql. Big red flag.
(Note: this is a picture from a user on serverfault, I didn’t take a screenshot when I first discovered this process. Luckily, the process on my server had only been running for a few hours, not a very long time like on this user’s server.)
After some searching, I found this. Turns out it’s a crypto mining malware, and at least one other person has written about it. This is the only source I can find about this process. Turns out, this process mines monero and sends it back to a third party. After reading through the info, it seemed fairly straightforward. But how did I get it?
Chasing Down the Cause
The only other mention I was able to find of “muhsti” was a serverfault page which linked to the above page. This user mentioned that it came in through a WordPress plugin called muhstikx86. I logged into the dashboard, looked at my installed plugins, and didn’t see it. Okay, so it’s not an overtly obvious plugin. I thought, “I’ll take a look in the plugins directory in case there is something there that seems out of place.”
Looks normal. Went to the root directory:
Wait – what’s plugins? I try to go into the directory. Not a directory. Go to edit the file – bunch of gibberish. That’s my first (although crude) way of telling something is an executable, not a script or a text file. Found it. Don’t know how it got there, but I found it. So, of course, I removed it.
I also re-read the article I posted above, and looked to the other sources of the problem – the way that the miner replicates itself, and the method by which (cron) it triggers itself. Here’s the cron process it entered:
I went through and deleted everything related to it. I still don’t know exactly where it came from, but as I’ve had htop open for a while, and haven’t seen the process resurface, I think – for now at least – I’ve gotten rid of it. I also changed the permissions on the www-user crontab, so that only root can write to it. An inelegant solution, but as of now, I have no cron jobs that the www-user user needs to be running. This will prevent replications like this in the future.
Where do I go from here?
I think I need to revisit the permissions of not only the files/folders in my webroot directory, but also the permissions of the www-data user. I don’t want them to be able to write to /dev/shm, I don’t think at least, (I’m going to keep reading up on this) and I don’t want it to generate cron jobs.
Hit me with some comments if you have any input, advice or ideas. Curious to see how widespread this is, and maybe where it came from originally. Perhaps I’ll never know.