Before discussing techniques and tools, it is essential to define the ‘secrets’ sought during penetration tests.
These secrets are generally private character strings which, if compromised, can be used to access a system, break encryption or forge data useful for authentication. Examples include a username and password pair, API keys, private keys or a session token that is still valid.
Obtaining a secret is often a major problem. This can enable an attacker to compromise the integrity, confidentiality and availability of system data.
Depending on the phase of the audit, its type (web, mobile, internal, etc.) and its technical conditions (black, grey or white box), different techniques and tools can be used. This article looks at some of the methods we use, without claiming to be an exhaustive list of possible techniques.
Comprehensive Guide to Techniques and Tools for Detecting Secrets
In the Reconnaissance Phase Prior to Technical Testing
During the reconnaissance phase of an audit, the aim is to gather as much information as possible about the target system or client organisation.
Part of this phase involves searching the Internet for secrets. Uncovering secrets at this stage can provide access to the target application and reveal important data leaks.
This information can then be exploited by an attacker to target the organisation more precisely.
Consulting data leak aggregators
There are online services that offer APIs for monitoring data leaks linked to a domain or subdomain.
Although these services are often not free, they can be very effective. They typically retrieve and analyse the following information:
- Data leaks from third-party services (such as LinkedIn, Adobe, etc.).
- Any data sold or traded on the dark web.
- Data leaks from ‘stealer’ malware, which steals logins and passwords.
Paid aggregators include services such as BreachLeaks, CybelAngel, Flare.io, leak-lookup.com, etc.
Once a list of identifiers, secrets or passwords has been detected for a domain or email, it is possible to test this data on the systems associated with the penetration test. This has often resulted in valid access.
Using search engines
The use of advanced operators in search engines can be very effective in quickly detecting secret leaks linked to indexed content. Here are two examples:
Example 1: searching directly for secrets linked to a domain
In this example, the aim is to discover whether public pages or documents indexed by Google contain identifiers in clear text. For example, the following search can be used: site:.example.com intext:password OR intext:secret.
This search lists all pages in .example.com subdomains containing the keywords ‘password’ or ‘secret’.
This type of search often returns false positives, such as authentication pages. However, by refining the search and analysing the results, advanced search operators can reveal identifiers, API keys or other secrets overlooked in an indexed page. However, this happens relatively infrequently.
Example 2: Searching known sites
Some sites, such as pastebin.com, are known to be used by malicious people to leak data. It can therefore be useful to carry out specific searches on this type of site. For example :
site:pastebin.com and intext:secret and intext:vaadata.com
This search will list pastebin.com pages containing the words ‘secret’ and ‘vaadata.com’.
For more details on exploiting Google dorks, see this article: Exploiting Google dorks to strengthen your security.
Github
Many companies or their employees have GitHub accounts with public repositories. It can be useful to examine these repositories as well as any commits made in search of secrets.
To identify a company’s employees, OSINT techniques can be used.
To search for and list identifiers in a repository, several tools exist, including the following:
Secrets are sometimes leaked in this way.
During Technical Tests: the Concrete Case of Web Pentests
During the test phase, also known as the active phase, attacks are carried out directly on the target platform.
It is crucial during these tests to detect whether secrets are being leaked. Depending on the conditions (black, grey or white box) and the type of audit, different tests can be carried out.
In white box conditions
During a white box test, the auditors have access to the source code of the solution as well as the configuration of the servers or other systems.
In this case, this information constitutes valuable research areas. Although it is relatively unlikely that an external attacker would have direct access to the source code, it often happens that access to sensitive services is forgotten in the code or configuration. If this data is ever accessed through a vulnerability, the consequences can be disastrous.
If you have access to the source code, it is essential to analyse it for hard-coded secrets.
To do this, you can re-use the tools in the ‘GitHub’ section on the repositories to which you have access. Other static analysis tools, such as Semgrep, SonarQube, etc., can also be used to identify secrets in addition to potential vulnerabilities detected in the code.
Another approach is to search for keywords in the source code.
Here is a list of keyword ideas that might be worth looking at:
- password
- secret
- private_key
- access_key
- access_token
- app_secret
- admin_key
- token
- API key
- apiKey
- api_key
- authsecret
- appkey
A simple ‘grep’ may be enough:
grep -Rna ‘access_key’This command searches for the keyword ‘access_key’ in the current directory recursively (-R), displaying the line on which the word is detected (-n).
The -a option is used to treat binary files as if they were text files and therefore to search their contents for readable text.
Code editors such as Visual Studio Code can also be used to carry out rapid searches within source code, thanks to advanced search functions and plugins adapted for code analysis.
Using a black or grey box approach
Server responses
Although it remains relatively rare, it can happen that secrets, such as identifiers or private API keys, are forgotten in the code of an HTML, JavaScript or CSS page, particularly in HTML or JavaScript comments.
Extracting data
Tools such as Burp Suite can be used to extract this data. In Burp Suite, go to the ‘Target’ tab, right-click on the target site and select ‘Engagement tools’.
The ‘Find comments’ option allows you to list and export all the comments on the pages visited, while the ‘Find scripts’ option allows you to list and export all the scripts. These files can then be analysed to detect any secrets or sensitive information that may have been forgotten.
Searching the data
Once the files have been exported, keyword searches can be carried out to identify secrets. More frequently, however, secrets are found in the server responses returned by the APIs.
A recurring example concerns the password hashes of a platform’s users.
Imagine an API request to an endpoint such as ‘/api/users’ to list the users of a web application. It is not uncommon to see applications where the entire ‘user’ object in the database is returned in the server response, including the password hash.
Fortunately, during an audit, tools can facilitate and automate the detection of this type of secret. One of the most widely used tools is Burp Suite, which provides a feature called BChecks.
BChecks are customised scans that users can create using a specific language. The official documentation for creating BChecks is available here: BChecks Documentation.
For secret detection, the idea is to create BChecks that passively detect the presence of different types of hash (such as md5, bcrypt, etc.) in server responses.
The hashes have a standard format, making them easy to identify. BChecks can also be used to detect specific keywords or certain API keys that follow a standard format, such as AWS Access Keys.
When a BCheck is triggered, an alert is displayed on the Burp Suite dashboard according to the message configured, making it possible to report the leak of secrets.
Configuration files
It is not uncommon for secrets to be directly visible in web application configuration files or test files.
Generally, these files are not directly accessible from the Internet. However, improper configuration of the web server can expose these extremely sensitive files.
As an attacker, the technique is simply to perform a brute force attack using existing file dictionaries to find out whether a configuration file is accessible. These files may contain secrets such as identifiers, API keys or sensitive parameters.
Various lists of known files are available on the Internet. The SecLists repository is a perfect example, offering a large number of dictionaries for searching files or directories: GitHub SecLists.
Tools such as feroxbuster, dirsearch and ffuf use these lists to detect the presence of files. It is essential to configure these tools correctly when using them to avoid overloading web servers or being detected by any anti brute force protection.
These tools make it easier to automate the search for sensitive files in publicly accessible directories, which could reveal poorly protected secrets.
Brute force
When authentication pages are present in an application and are not protected by rate limiting mechanisms, it becomes possible to launch brute force attacks using dictionaries to try to discover valid passwords.
Default credentials
Some applications or systems use known solutions or frameworks.
If default authentication pages are present or a known service uses API keys, it may be worth consulting the associated documentation.
Default credentials could potentially exist and be used. If the configuration has not been changed following deployment of the service, this can make it easy to gain access to the service concerned. This still happens regularly during our audits.
Other Techniques for Finding Secrets Depending on the Test Target
Mobile applications
The local storage of a mobile device sometimes holds surprises, as does the decompiled code of an application.
It is useful to analyse local databases, shared preferences or strings extracted from decompiled code.
These analyses can be used to identify secrets, such as API or encryption keys.
Here is an article that illustrates why it is essential to check the source code of a mobile application: Insecure Authentication Tokens leading to Account Takeover.
Internal networks
During an internal penetration test, listening in on network communications (man in the middle) is a key practice.
It enables the possible transmission of identifiers or hashes to be identified and analysed. Analysing file shares can also reveal leaks of secrets.
In addition, if you have access to the server or system configuration, examine the configuration files and environment variables. These items may contain sensitive information.
IoT systems
During audits of connected objects, secrets such as encryption keys are sometimes integrated directly into the firmware or memory.
Ideally, these secrets should be stored in a dedicated chip and encrypted at the time of manufacture, but this is not always the case.
It is therefore essential to extract the RAM memory and firmware and analyse them as source code, using regular expressions or secret detection tools.
Analysis of network traffic (Bluetooth, TCP, UDP, etc.) can also reveal secrets in transit.
Phishing
Phishing is another method of recovering secrets, particularly logins and passwords.
However, this attack mainly targets humans and is often aimed at stealing accounts. It is therefore not a genuine technique for identifying secrets.
How to Prevent the Risk of Secrets Compromise?
As we have seen, there are different ways of identifying secrets. These searches can be simplified using specialised tools.
However, it remains difficult to control all the secrets in an information system and completely prevent leaks.
To limit the risks and impact of such leaks, it is essential to adopt good security practices. Here are a few recommendations:
- Use strong secrets: Opt for complex passwords and keys. Avoid default secrets or secrets that are too weak, even in test environments.
- Implement strong authentication: Activate two-factor authentication (2FA) for all users. Use certificate authentication rather than password authentication for services such as SSH.
- Monitor data leaks: Regularly check whether your organisation is affected by known leaks, for example via Have I Been Pwned.
- Avoid hardcoding secrets: Don’t store secrets directly in source code or Git repositories. Use environment variables instead.
- Encrypt communications: Always use secure protocols such as HTTPS, SFTP, FTPS or SSH.
- Set up monitoring: Activate logging and monitoring of network activities to quickly detect abnormal traffic or a compromise. A rapid response can limit the damage.
- Train your teams: Make employees aware of security issues, phishing and the use of password managers.
- React in the event of a compromise: If a vulnerability is identified, correct it immediately and change all potentially exposed secrets.
Author: Yoan MONTOYA – Pentester @Vaadata