Automated penetration testing (pen-testing) is an up-and-coming approach that aims to address the issues businesses face with more traditional approaches to pen-testing, which relies almost completely on the skill and experience of pen-testers and the budget and timeframes applied to the pen-testing engagement.

Caveat

The author of this article has not evaluated or reviewed an automated pen-testing solution as part of research or during a PCI DSS assessment. All information regarding automated pen-testing solutions has been sourced from vendor-provided documentation publicly accessible on the Internet. The article does not endorse nor provide a formal opinion as to whether an automated pen-testing solution meets applicable PCI DSS requirements but merely provides discussion points regarding compliance based on the documentation available to the author. The author is not a qualified pen-tester but is an active QSA.

What is Automated Penetration Testing

Automated pen-testing is an approach where systems and devices are continuously scanned to detect vulnerabilities within the business. The service automatically attempts to exploit the vulnerabilities using pen-testing tools and techniques without the need for a human pen-tester. Automated pen-testing tools appear to have been built on years of automated vulnerability scanning, making the leap to attempting to exploit the vulnerabilities that have been found. The benefits of automated pen-testing systems are readily apparent and attractive to any business requiring pen-testing as part of their compliance obligations such as PCI DSS.

Benefits of Automated Penetration Testing

The benefits include:

  1. Continuous pen-testing of systems. Instead of the traditional point-in-time pen-tests undertaken annually or as part of a significant change to the information technology environment, automated pen-testing can be ongoing. A significant and exploitable vulnerability could be introduced into the environment at any time via patching, upgrading systems, or updates to custom-code. If a pen-test is only undertaken annually then there is risk that a vulnerability could be available for a significant period of time before its detected and exploited.
  2. Improved speed and coverage. Using computer-based automation dramatically increases the speed, coverage and type of pen-testing techniques that can be used. Where human pen-testers may only be able to focus on a small sample-set of systems and devices per engagement, automated tools can run multiple streams of tasks on the entire environment and do not rely on a single person’s skill set.
  3. Quick updates. Depending on the automated pen-testing vendor, the automated pen-testing system could be updated with the latest vulnerability data, attack vectors and exploit techniques hours/days after the detection and identification of the vulnerability is known the security community. However, with the traditional pen-testing engagement, it may take longer for pen-testers to develop the tools to exploit new vulnerabilities.

Why Automated Penetration Testing Might Not Be a Silver Bullet

Although automated pen-testing systems would be a very attractive proposition for a business over engaging human pen-testers, consideration of compliance requirements must be factored into the procurement process.

For PCI DSS, Requirement 11.3 and its sub-requirements address pen-testing. An extract of Requirement 11.3 is provided below.

11.3 Implement a methodology for penetration testing that includes the following:

    • Is based on industry-accepted penetration testing approaches (for example, NIST SP800-115)
    • Includes coverage for the entire CDE perimeter and critical systems
    • Includes testing from both inside and outside the network
    • Includes testing to validate any segmentation and scope-reduction controls
    • Defines application-layer penetration tests to include, at a minimum, the vulnerabilities listed in Requirement 6.5
    • Defines network-layer penetration tests to include components that support network functions as well as operating systems
    • Includes review and consideration of threats and vulnerabilities experienced in the last 12 months
    • Specifies retention of penetration testing results and remediation activities results

Based on the publicly accessible information available for automated pen-testing tools reviewed at the time of this article’s published date, further research is required in the automated pen-testing tool’s ability to meet the requirement’s objective for application layer pen-testing (PCI DSS requirement 6.5).

The objective of requirement 6.5 is to provide a minimum level of testing custom applications for security vulnerabilities. Requirement 6.5.1 through to requirement 6.5.10 cover the following vulnerabilities:

  1. 6.5.1 – Injection flaws such as SQL injection and command injection.
  2. 6.5.2 – Buffer overflows.
  3. 6.5.3 – Insecure cryptographic storage which addresses the use of insecure cryptographic ciphers, keys and algorithms.
  4. 6.5.4 – Insecure communications
  5. 6.5.5 – Improper error handling such as dumping out a stack trace to the end-user or providing too much information about the error to the end-user.
  6. 6.5.6 – Ensures that an organisation’s policy and procedures are updated and robust enough to identify all “high-risk” vulnerabilities.
  7. 6.5.7 – Cross-site scripting (XSS).
  8. 6.5.8 – Improper access control such as directory traversal and restricting access to functions.
  9. 6.5.9 – Cross-site request forgery (CSRF)
  10. 6.5.10 – Broken authentication and session management such as incorporating sensitive session variables in the session cookie.

In regards to off-the-shelf software, the requirements above all sound doable for the automated pen-testing tool since there is an abundance of vulnerability sources publicly available for off-the-shelf applications such as web servers, blogs, office applications, CMS and CRM systems etc.

However, consider an application that has been developed solely for an organisation. The automated pen-testing tool may identify a SQL injection vulnerability in the login page, but could the tool be smart enough to exploit this vulnerability to attempt further exploration within the application in order to identify and exploit more vulnerabilities? The SQL injection vulnerability on the login page may be identified and patched however, what other vulnerabilities still exist that were not identified via automated pen-testing?

For example, if a skilled human pen-tester identified the SQL injection vulnerability within the login page the pen-tester could use a known users account name and bypass the login process using SQL injection technique, then locate the user’s current password and log into the server via SSH. Upon finding that the MySQL server root account has no password, the pen-tester logs into mySQL server as root and searches the database for valuable information. Further, based on the mySQL version the pen-tester uses a well-known function within the mySQL sever to gain root privileges to the server. As this example shows, the human pen-tester has gone from identifying a SQL injection vulnerability in a web-based login page to “owning” the entire server.

It could be suggested that the whole example scenario above would be removed once the SQL injection vulnerability was located but there are many more ways to get a user’s password. The lax security for SSH access and mySQL running as root with no password on the root account stills remains to be exploited another way.

Automated Pen-Testing & Application Layer Testing

NIST Special Publication 800-115 “Technical Guide to Information Security Testing and Assessment” provides guidance on “application security testing”.

Application security testing and examination help an organization determine whether its custom application software—for example, Web applications—contains vulnerabilities that can be exploited, and whether the software behaves and interacts securely with its users, other applications (such as databases), and its execution environment. Application security can be assessed in a number of ways, ranging from source code review to penetration testing of the implemented application.

Custom application software provides challenges to pen-testers in that the application or components of the application are not known outside of the business and any third-party developers and testers engaged to develop and test the application. Whereas vulnerabilities detected within operating systems and off-the-shelf applications are shared publicly, and in doing so allow effective tools to exploit the vulnerabilities to be crafted and released, custom code applications almost always require a reasonable amount of time researching the functions provided by the application, testing possible attack vectors and applying a unique path of attack that may result in further vulnerabilities being detected.

The cost of black-box pen-testing of custom applications is a constant discussion point with a business due to the pen-testers effectively testing “blind” which consumes more time and resources with a reasonable risk that not all vulnerabilities will be detected. In many cases white-box testing is more effective at detecting vulnerabilities in custom code since the source code is often provided to the pen-testers to locate vulnerabilities not easily detected at the applications interfaces.

A good example of the lengths pen-testers have to go to when pen-testing custom applications is the following video of a pen-tester attempting to detect and exploit a vulnerability in a war-game application “Learning object serialization vulnerabilities – Natas 26 Walkthrough – Overthewire.org”

[fusion_youtube id=”https://youtu.be/ILfPqpdTm0k” alignment=”center” autoplay=”false” api_params=”” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” class=”” css_id=”” /]

When reflecting on the uniqueness of custom code applications where consideration such as software methodologies used, coding languages used, information disclosure within comments, logic path flows and manipulation of application-level data are researched and tested resulting in a custom attack vector that has been hand-crafted, it is difficult for the author to see how an automated pen-testing tool could obtain the same or better result than a human pen-tester. The automated pen-testing tool may allow for such attack vectors to be scripted and therefore automated. However, there is always the discussion point around who would be qualified and experienced enough in pen-testing to script the attack? The developers of the application or the in-house testers could be a possibility but it would be rare that a developer or application tester would also be skilled in pen-testing.

Another possibility that the use of automated pen-testing tools could meet PCI DSS version 3.2.1 pen-testing requirements is a combination of automated pen-testing and human pen-testing. The automated pen-testing tool could be used frequently to detect and exploit attack vectors that can be scripted and repeated without requiring changes such as detecting unencrypted user account credentials, weak password policy, insecure protocols, daemons and services and exploiting known vulnerabilities in off-the-shelf applications. The benefit to the business would be reduced costs for pen-testing as less engagement of human pen-testers would be needed, faster detection of vulnerabilities and consistent testing and reporting. However, as shown in above, the automated pen-testing tool could identify a SQL injection vulnerability in a login page for example, but a skilled human pen-tester would use that exploit to explore the environment and more than likely identify more vulnerabilities. So, would the QSA consider the automated pen-testing tool a very sophisticated vulnerability scanner only?

Pen-testing or Vulnerability Scanning

For the QSA further guidance is provided in The Information Supplement: Penetration Testing Guidance version 1.1 which provides the following guidance in section 4.2.2 Network Layer. The author has highlighted in bold the most relevant part of the extract.

Since most protocols are well defined and have standard modes of interaction, network-layer testing is more suitable for automated testing. This makes automation the first logical step in a network-layer test. Because of such standardization, tools may be used to quickly identify a service, the version of the software, test for common mis-configurations, and even identify vulnerabilities. Automated tests can be performed much faster than could be expected of a human. However, simply running an automated tool does not satisfy the penetration testing requirement. Automated tools cannot interpret vulnerabilities, misconfigurations, or even the services exposed to assess the true risk to the environment. The automated tool only serves as a baseline indication of the potential attack surface of the environment. The penetration tester must interpret the results of any automated tools and determine whether additional testing is needed.

This version of the supplement was written in 2017 so consideration of automated pen-testing tools may not have been incorporated. However, section 2.1 “How does a penetration test differ from a vulnerability scan?” provides further guidance on what PCI DSS considers as implementing pen-testing. You can also read more about this in Confide’s PCI Basics article: Penetration Testing and Vulnerability Scanning, What’s the Difference? In fact, the guidance infers frequently through the document that pen-testing is a human executed process.

Until further understanding of how automated pen-testing tools support the ability to apply complex and intelligent pen-testing techniques, and the PCI Security Standards Council provides guidance on the use of automated pen-testing tools, further discussion between your QSA or acquiring bank may be needed.