Cloud WAF Comparison Using Real-World Attacks

Many teams consider use of Web Application Firewalls (WAFs) as a best practice or a compliance requirement when implementing web applications. All firewalls are used to control and monitor traffic. A WAF is a specific firewall that is designed to identify and prevent attacks from web application traffic. The expectation is that the firewall will prevent command injection attacks, cross-site scripting attacks, protocol violations, and other common attacks against web applications.

Are the cloud WAFs any good in blocking common web application attacks? We decided to find out and the results were surprising.

Back in the day WAFs were physical appliances protecting on-premises applications. Today, more and more applications are in the cloud and they are protected using WAFs provisioned in the cloud. Major cloud vendors have a range of WAF offerings available. They also provide their own in-house WAFs that integrate well with their respective load balancers. Is their security performance at the level of top tier offerings?

According to customer reviews published by Gartner, one software engineer believes that AWS Web Application Firewall is “thе bеst sоlutіоn tо stор аttасks оn оur wеb аррlісаtіоns”. Another specialist calls it “a реrfесt sоlutіоn fоr оur nееds.” As for Azure, reviews appear more mixed. “Still behind dedicated appliances” believes one architect, although many find it “easy to implement and use.”

For a security solution, the primary measure of usefulness should be the ability to protect your applications. Are the cloud WAFs any good in blocking common web application attacks? We decided to find out and the results were surprising.

Test setup

We build a test environments in AWS and Azure to put the cloud WAFs into a test. Our setup comprises of the following:

An attacker’s host executing thousands of scripted test cases to a specific target web host. We injected our payloads to HTTP GET and POST requests going towards the target.
A target web host running a containerized custom web server in a cloud container platform – AWS Elastic Container Service ECS or Azure Container Instances ACI. This web server answered with the HTTP response code ‘200 OK’ to all incoming HTTP requests.
A WAF-capable cloud load balancer – AWS Elastic Load Balancer ELB or Azure Web Application Gateway – in front of the target web host.
Barracuda WAF-as-a-Service (WaaS), provisioned from the Azure Marketplace, directing traffic to a target web host on Azure ACI.

In selection of the products we wanted to put the AWS and Azure in-house solutions against commercial products that are also available as managed services in the cloud. The services selected for the testing were the following:

Azure Application Gateway WAF, using CRS 3.1 rules from the Open Web Application Security Project (OWASP)
Barracuda WAF-as-a-Service (WaaS), provisioned from the Azure Marketplace, using Barracuda managed rules
AWS WAF using Amazon managed rules
AWS WAF using Fortinet managed rules

The overall test architecture looked like the following when using Azure cloud. An identical setup was built on AWS cloud.

High-level architecture for testing WAFs connected to cloud load balancers in Azure and AWS.

An alternate setup was required for testing the Barracuda WAF-as-a-service. We provisioned the service on Azure cloud and implemented minimal changes to the setup otherwise.

High-level architecture for testing Barracuda WAF-as-a-service in Azure.

We conducted the test on February 19th using the managed defaults available on those services during that day. To accurately reflect the most common way of using these services, we did not remove or add any rules or definitions that might affect the capability to detect or block attacks. In case of AWS Managed Rules, a selection of rule groups had to be made, as there is a limit to the number of rule groups that can be active simultaneously. We selected a combination that we though would best cover OWASP Top 10 threats and provide best protection against our test cases. The selected rule groups were:

AWS-AWSManagedRulesSQLiRuleSet
AWS-AWSManagedRulesLinuxRuleSet
AWS-AWSManagedRulesKnownBadInputsRuleSet
AWS-AWSManagedRulesCommonRuleSet
AWS-AWSManagedRulesPHPRuleSet
AWS-AWSManagedRulesAdminProtectionRuleSet

Test cases

For testing the WAFs, we have collected several of real-world attacks and evasions. We are launching these attacks over HTTP against a custom web server that keeps track of requests that have passed the WAF.

The test case collection contains the following groups used for reporting the results (explanations adapted from owasp.org):

Command execution. Commands injected into an application through input can be used to compromise a system.
Server-Side Includes (SSI) Injection. SSIs are directives present on Web applications used to feed an HTML page with dynamic contents. The Server-Side Includes attack allows the exploitation of a web application by injecting scripts in HTML pages or executing arbitrary codes remotely.
SQL Injection. The attack consists of insertion or “injection” of a SQL query via the input data from the client to the application.
Path Traversal. A path traversal attack (also known as directory traversal) aims to access files and directories that are stored outside the web root folder.
Malformed XML Documents. Malformed documents can be used to consume resources or inject malicious commands.
Cross Site Scripting (XSS). Cross-Site Scripting (XSS) attacks are a type of injection, in which malicious scripts are injected from a malicious website into otherwise benign and trusted websites via a web browser.

This collection of tests represents typical attacks that are targeted against websites. The intention of the test cases is not to represent business logic weaknesses or any other vulnerabilities where specific application logic would be exploited to a malicious end.

Test results

The test results are presented on the following table.

Results of our WAF test. The percentage designates attacks blocked. Higher number is better.

The results shows a massive spread in security performance. Azure WAF using CRS 3.1 rules outperforms the rest of the services with a good margin. It is also the only of the tested services that performed solidly across our test case set.

What was interesting to discover was that the security performance of the WAFs varied whether URL character encoding was used or not. The difference was dramatic in some of the test cases. For instance, Barracuda blocked all of our unencoded SSI test cases and only half of them when they were encoded. From an attacker’s point of view, trying different encodings could be an effective way to evade WAF protection. We will look into this more closely in future posts.

These results are not intended to be interpreted in any other context other than comparing relative performance of the services when using our test set.

Examples

The AWS managed WAF was the most permissive one of the selected WAF services, allowing through such attack payloads as:

Command execution in a GET request: & echo “” > cmd.php
Simple SQLi payload: UNION ALL SELECT @@VERSION,USER(),SLEEP(5) --

Both examples were also allowed by the Fortinet Managed WAF in AWS and the Barracuda WAF-as-a-Service in Azure.

Looking at the payloads and rejected responses, it is difficult to draw many conclusions on the internal workings of the WAFs. Sometimes the behavior is downright confusing and hilarious, for example:

The AWS Managed WAF seems to handle POST and GET requests completely differently. The following request was blocked in GET request and allowed in the body of POST requests:%2e%2e//etc/passwd
The AWS managed WAF blocks the payload ls -l /var/www/*, but allows the payloads && ls -l /var/www/*, | ls -l /var/www/* and so on.
The Barracuda WAF-as-a-Service insists on blocking the character ‘ when it is the only character of the payload. Still, it allows payloads such as eval(‘sleep 5’); and ‘ whoami.

Conclusions

The results show that not all cloud WAF services are alike. When testing AWS, Azure, and Barracuda with our test cases, Azure WAF was the clear winner and the only service that performed well. What comes to the AWS and Barracuda offerings, the security benefit for protecting against some of the most common attack types is flaky at best.

If you are building an application in the cloud, you should be mindful about which security services to provision. While having a cloud WAF might tick your compliance check box, the security benefit might be different from what you expect.

Azure WAF was the clear winner and the only service that performed well in blocking real-world attacks in our test.

We undertook this test expecting we would need to develop custom evasions to witness difference in security performance of these services. This was not the case. However, we will continue looking at the topic in future posts.

About the authors

This cloud WAF test was conducted by Tuomo, Tommi, and Marko of Fraktal cyber security team. Found this post useful? Let us know what you think on @FraktalCyber Twitter.

Edited March 11th 2020: The original version of this article did not include list of AWS rule groups used. The list of AWS rule groups was added, as this is relevant information for interpreting the results.

Edited March 12th 2020: The original version of this article suggested that the CRS ruleset is managed by Microsoft. Clarified that the rules are from OWASP (coreruleset.org).

https://medium.com/fraktal/cloud-waf-comparison-using-real-world-attacks-acb21d37805e