Context

Scanning using AppCheck is designed to be non-disruptive and is used by thousands of users to test their production systems every day. 

There are cases however where scanning in production may not be permitted. The common workaround is to scan an identical, or close to an identical system in a non-production environment. This is a widely accepted approach and in the main is recommended. 

However, this approach can result in crucial vulnerabilities being overlooked, especially those which only occur due to interoperability between systems in production and do not materialise in other environments.  

To help solve this problem, AppCheck includes a scan template which disables Dynamic Application Testing checks and focuses on environmental vulnerabilities that occur due to server configuration, patch level and system interoperability. In other words, the vulnerabilities that may be introduced in your production environment that were not present in pre-production. 

By combining the results of this scan with a full scan of the pre-production build, optimal coverage is achieved without risking unwanted side effects associated with DAST scanning of sensitive systems. 

We also have the following FAQ that contains further information on Minimising Risk of Web Application Scanning.

Scanning sensitive production systems 

AppCheck is designed to be used to scan production systems and avoids disruptive testing by design. However, there are cases where the nature of the target application means that any testing, automated or otherwise, could have an undesirable effect. For example, an application that allows users to authenticate and place an order, or request a service from the organisation, may create an entry in the database that then requires further action by a human (e.g. an employee needs to act on the request or order). Another case may involve authenticated testing of a management system such as a Content Management or HR system. A core part of scanning involves recording legitimate requests then replaying them to include test cases, if those requests were designed to edit a page or amend a salary, scanning with a powerful user account would naturally risk causing disruption. 

The widely accepted solution to this problem is to scan a non-production build of the application. Providing that the code used in production is the same as the non-production build, the scan should be comprehensive in identifying code vulnerabilities and be free from unwanted side effects. 

However, although the code may be identical, there are often differences between production and pre-production environments that can introduce serious vulnerabilities. For example, the web servers and application servers may be a different versions or differ in configuration. The production environment can include additional systems such as reverse proxies, caching servers, web application firewalls and cloud services such as Cloudflare or Akamai.  

These differences, no matter how subtle, can introduce security flaws that can only be identified when considering the production system as a whole. 

System Interoperability Vulnerabilities 

One of the most common areas for vulnerabilities to be introduced lies in the interoperability of systems in production environments. These flaws are often overlooked because each system is tested in isolation. 

For example, the common Cross-Site Scripting (XSS) vulnerability occurs when data submitted via a web browser is embedded in the application's response without proper filtering. The attacker could exploit this by tricking the user into following a malicious link that embeds JavaScript in the page designed to hijack their session with the application. This is one of the most common findings in application scanning but only applies to parts of the request that the attacker can control through a malicious link or an evil web page (i.e URL components, Form data and the referrer header). If this behaviour is observed in an arbitrary HTTP header, it is often either reported as Informational or ignored completely. This is because there is no way for the attacker to control arbitrary headers in an XSS attack (with the exception of the referer header) and therefore at face value poses no risk. 

Whilst this finding is likely to be overlooked as a non-issue (which it often is), consider how this may change if a web cache is introduced when the app is deployed in production.  

Web caching servers store web server responses to speed up access to commonly requested components. The cache identifies requests by building a “cache key” that is then used to store are retrieve items from the cache. Typically, cache keys are based on the URL (hostname, path and query string) but don’t take into account other headers. Therefore, in our example, the attacker may be able to poison the cache with a malicious response by embedding JavaScript code within the affected header. Users who access the same URL will now retrieve the malicious version of the page, turning this previously considered non-issue into a devastating critical impact attack. 

You can read more about XSS and Cache Poisoning here: https://appcheck-ng.com/cross-site-scripting/ and https://appcheck-ng.com/web-cache-poisoning-explained/ 

Another popular and powerful attack is HTTP Request Smuggling, a vulnerability that occurs due to two or more systems in a chain calculating the length of a malformed HTTP request differently. In this attack, its possible for the attacker to remotely manipulate requests submitted by other users. Crucially, this attack occurs due to the Interoperability of two or more systems and may only affect your production environment. Additionally, an update to this attack technique, focusing on HTTP/2 was presented at Blackhat 2021 and had breathed new life into this powerful attack. 

Hosting environment vulnerabilities 

Hosting components such as Application Frameworks, Web Servers, Application Servers, Content Management Systems and intermediary devices are a crucial part of your attack surface. Threat actors have redoubled their efforts on these systems recently due in part to the global pandemic, where remote access systems have become more widespread and business-critical. In 2021 the UK’s National Cyber Security Centre (NCSC) and the NSA released a joint advisory stating that Nation State-backed threat actors were actively targeting security flaws in; Cisco, Citrix, F5 BigIP, Fortigate, Pulse Secure, and VMware, systems commonly found at, or close to the internet gateway. 

In the same year, critical impact vulnerabilities have also been discovered and subsequently widely exploited in the Apache Webserver and countless popular Content Management Systems (CMS) and plugins. 

This area of your exposed Attack Surface is perhaps one of the most volatile. New security flaws are disclosed daily, a system that was believed to be secure yesterday may now be known to contain a critical security flaw. It is not uncommon for server build vulnerabilities to be dismissed as an acceptable risk or overlooked in development and staging environments, especially when they are restricted to trusted IP addresses/users.  

It is essential that vulnerabilities within the production environment are identified on an ongoing basis and quickly addressed. Attackers are quick to jump on the latest and greatest attacks so it’s essential we detect these vulnerabilities quickly without waiting for the next full scan. 

Configuration Differences 

It’s often not possible to have the exact same configuration across production and pre-production, practical differences such as location and resourcing among many other factors can play a part and affect security. 

For example, differences in proxy and firewall configuration or configuration to support the use of an additional system such as a CDN service, load balancer, remote management system or web application firewall can impact security. Even in cases where the additional system is designed to improve security, there are cases where these systems introduce interoperability flaws or can even introduce vulnerabilities themselves.  

Gaining complete scan coverage in production 

To help address the requirement for scanning sensitive systems in production, AppCheck includes a scanning template named “Sensitive Production Infrastructure”. This template disables Dynamic Application Security Testing (DAST) modules that operate by recording and replaying application requests. Instead, this template focuses on vulnerabilities that are most likely to occur between different environments such as; HTTP Request Smuggling, Cache Poisoning, CVE scanning, Web server configuration and Build scanning and other infrastructure level vulnerabilities. 

The recommended way to use this template is to perform a full scan of the pre-production environment and supplement that with a Sensitive Production Infrastructure scan of the production environment. The latter would typically be an unauthenticated scan and is designed to be run regularly.