‘Bug bounty signs the end of pentests’ ‘Bug bounty, the death of the traditional penetration test’… Do you remember these article headlines? Today we see that this is not the case and that both approaches continue to exist.
Both services address the same initial need: to test a company’s web applications and infrastructure with realistic attacks. What are the differences between the two approaches? How do you choose between a bug bounty and a penetration test?
We present you nine main criteria to consider.
Pentest or Bug Bounty: What is it?
Pentests and bug bounty programs allow testing web platforms by simulating attacks to detect and fix vulnerabilities. A pentest is a service performed by a team of consultants working for a specialised company, while a bug bounty program relies on independent hackers paid per vulnerability.
These two approaches exist in a range of variations and nuances depending on the offers and providers. In this article, we present the most common characteristics of each approach.
Pentest or Bug Bounty: Length, cost and scope of the security tests
The cost of time is more complex that it seems, or: elements to take into account
The question of the duration of the tests
A pentest is carried out over a defined period of time (usually a few weeks per semester or per year). The length of the penetration test is determined according to the functional complexity of the target and the degree of depth wanted for the security tests. It allows you to obtain security feedback for a specific date. You can therefore plan the pentests according to your rate of release, to evaluate your security and rank the security priorities at regular intervals.
Bug bounty is often conducted throughout the year, for an indefinite period of time. Continuous testing ensures that the perimeter is constantly monitored. Independent hackers (or bug hunters) check new implementations as they are released.
Some security researchers may dedicate a lot of time to the project, if they are particularly interested in it (for reasons of technical challenge, attractive reward or personal interest in the subject).
The argument of the cost of testing
Money is the key of security. At first sight, bug bounty seems more interesting financially, because there is no expense until there are no reported flaws. Theoretically, a wide scope should be covered without increasing costs. However, in order to attract good security researchers, plan attractive rewards, because there is competition between bug bounty programs to attract the best researchers. In addition, you need to add to this the time your security team will devote to managing the program (management of flaws, exchanges with researchers, etc.).
The advantage of penetration testing is that the cost is controlled and defined in advance, as the type and level of criticality of the vulnerabilities discovered do not influence the cost of the service. Similarly, the number of vulnerabilities reported does not multiply the price. For example, injection flaws in four different locations within the tested scope will not result in four separate reports and four separate payments as in a bug bounty program. That’s why bug bounty can be risky from a financial point of view.
However, the cost of a penetration test can be high when it comes to covering a large area and carrying out exhaustive tests, as an appropriate amount of time needs to be budgeted.
The challenge of methodology, scope and target type
The methodology of a penetration test ensures that everything in the perimeter is examined, and with the same rigour. It is a security audit, based on a structured approach, and enables all types of vulnerabilities to be investigated, without neglecting any angle of attack. ’Weak’ or ‘information’ vulnerabilities are also reported to improve overall security.
The approach of a bug bounty programme is different since each researcher is free to determine their research as they wish, without constraint of methodology or coverage rate of the perimeter.
The scope of a pentest is fixed for each session. It can evolve as the testing sessions progress to integrate new functionalities or to focus on sensitive elements. Variations in the scope are specifically briefed to the team in charge of testing.
A bug bounty program offers great flexibility regarding tests scoping. The scope can easily evolve, to grow with each release and as the company matures on more elements.
As for the testing approach, bug bounty programmes are perfectly suited to testing web targets, in a black box approach. Some private programmes (more restricted) also allow grey box testing of web applications. Penetration tests are perfectly suited to black box and grey box testing of web targets. Pentests can also be used to test a wider range of targets: internal company network, social engineering, hardware, etc.
Pentest or Bug Bounty: The skills of security experts
The expertise of ethical hackers
During a bug bounty, many people work on the same scope. There is therefore a wide variety of approaches and experiences represented. Researchers can think of different attacks and detect many flaws. There is a form of competition between the participants, who are independent and work alone. The level of experience and expertise of the participants vary, but it is possible to filter by selecting only security researchers whose level has been validated by the bug bounty platform.
During a pentest, the consultants work alone or in teams, depending on the internal organisation of the company and the specificities of the project. Many penetration tests are carried out by teams of two or three people, which favours an effervescence of ideas, in addition to the complementarity of skills. In addition, a pentest team capitalises on the experience of other pentests conducted by the company and brings together people whose job is to search for vulnerabilities, which guarantees advanced expertise.
The company’s responsibility
A pentesting company is responsible for the quality of the penetration tests performed, knowing that its reputation is its best business card. This goes through the work methodology, the means allocated to technical monitoring and training, as much as the profiles of the security auditors. A bug bounty platform is accountable for the researchers present on its platform, which is based on a process of validation of the identity and skills of ethical hackers.
Both penetration testing companies and bug bounty platforms vouch for the confidentiality of their clients’ data and the legal responsibility of professional hacking.
Penetration testing and bug bounty are two popular approaches for companies wishing to prove the security level of a web platform to third parties (prospects, partners, investors). Historically, pentesting has an advantage, as it has been recognised for longer and has reassuring aspects. The methodological approach of the pentest (which is a security audit) makes it possible to value the type of tests that have been carried out, beyond the flaws that have been reported.
An audit report explains the portions tested and the tests performed. It provides a perspective on the security level of the audit target and is a deliverable that can be communicated to third parties and be taken into account in obtaining certification such as ISO27001 or SOC2.
Pentest or bug bounty: Management of security tests
Management of detected vulnerabilities
Managing discovered vulnerabilities in real time is one of the main challenges of bug bounty. There is a risk of duplicate reports, when several researchers submit the same vulnerability before it is resolved. If the process of recording reported vulnerabilities is not transparent, it can lead to frustration for ethical hackers that they have spent time for nothing.
Similarly, it is important to provide visibility into the progress of vulnerability handling, from report acknowledgement to payment of remuneration, to avoid a drop in commitment from bug hunters. Independent researchers appreciate knowing what stage you are at.
For the team in charge of managing the identified flaws, this leads to an expectation that the reports will be investigated quickly, both by researchers and by the internal team in charge of fixing them. This can lead to ‘alert fatigue’, which can lead to misjudging the importance of a vulnerability. If disagreements arise about the level of severity of the flaw and therefore the amount of compensation, researchers may lose interest in the programme.
In penetration testing, contrary to what is sometimes thought, it is possible to report important vulnerabilities in real time, without waiting for the final report. Depending on your provider, you can ask to be alerted immediately when a critical vulnerability is discovered.
Monitoring the tests performed
A penetration test facilitates communication. In the event of a problem (page unavailable, functionality not working), the pentesters contact you and exchange information immediately. If a test has been a little too aggressive, this also helps to quickly limit the undesirable effects.
Another advantage of pentesting is that you can agree with your service provider on the timing of certain tests that could disrupt your business (denial of service tests, for example).
Bug bounty programs benefit from the adaptability of the launch. Unlike penetration testing, where the dates of intervention are fixed in advance, the start of bug bounty tests can easily be shifted if your development team has had a delay.
Bug bounty platforms now often offer a VPN to monitor the activities of researchers. This way, as with a pentest, the IP addresses are communicated upstream and known, which makes it easy to follow the tests and differentiate them from real attacks.
The remediation phase of identified vulnerabilities
Once the flaws have been identified, the remediation phase begins. One advantage of penetration testing is that it facilitates the prioritisation of the vulnerabilities discovered, as they are all reported at the same time and prioritised in the security audit report. The criticality level of each vulnerability is detailed in the report, as well as precise recommendations for correction.
The oral presentation of the results allows an efficient and direct exchange between the technical teams and the pentesters, which is a source of knowledge transfer and increases the skills of your teams.
During a pentest, the validation of corrections is a second audit phase. It allows you to confirm that the remediation works and to ensure that it has not created any side effects (i.e. caused flaws in other places).
We have discussed vulnerability management in bug bounty programs. As for remediation, researchers are usually available to re-test corrected vulnerabilities, as their remuneration often depends on the report being closed.
Support and advice
The pentest provider advises upstream on the scope and approach, but also downstream on the next steps to be taken. A contact person discusses with you the specific risks related to your activity, in order to establish the most relevant penetration test strategy for your context.
If the bug bounty program is managed, your partner can also advise you on the scope, the amount of payments and the management of the flaws detected, in order to facilitate the processing and the exchanges with the researchers.
The level of support depends on the company itself (penetration test or bug bounty) and the person who will be working with you. It should be noted, however, that as penetration testing is applied to a wider range of targets (web, infrastructure, hardware, social engineering…), the support can take into account broader security issues.
Pentest or bug bounty: Complementary advantages
As you have just read, there are different points to consider when choosing between a pentest and a bug bounty program. Beyond their specificities, performing a penetration test or launching a bug bounty campaign have the same objective: to identify vulnerabilities in a precise target by carrying out security tests. Each approach has its advantages and disadvantages, and is adapted to specific needs and issues.
If the goal is to obtain a deliverable (pentest report or executive summary) or a certificate, to close a sale or reassure partners or investors, a penetration test will be more suitable. Moreover, performing a pentest is a necessary condition for achieving certain certifications, as ISO 27001 or SOC2. If you want to carry out continuous testing to constantly monitor a specific perimeter, then a bug bounty programme is more appropriate.
The ideal is to combine both approaches: regular pentests, with continuous testing in between. In security, zero risk is never achieved, but all solutions that reduce the level of risk are relevant. The amount of resources to be allocated to security testing also depends on the risks to which the company is exposed.
Finally, whether or not you have a bug bounty programme, it is recommended that you encourage responsible disclosure by setting up a means of being contacted by security researchers if any flaws are discovered in your solution. This means being easily reached, for example by having an email address, such as security[at]company.com, or a dedicated page, in order to create a framework for responsible reporting of flaws by researchers.