Choosing security scanners for your project
With literally dozens of code security scanners available on the market there’s a risk of throwing your money and resources at tool that will eventually prove to be of little use. How do you select the right tool for the job?
The notion that running code security scanners against your code in any software project is easy to defend: the earlier you catch security issues, the easier and cheaper it is to fix them. Today I routinely help client’s developer teams design a SDLC pipeline involving source code and dependency scanners triggered by repo commits, plus a run-time scanner in QA/UAT, all in continuous integration pipeline, and then we can go into a pentest at each stage of development with confidence that the app won’t get shamed with some low-hanging trivial vulnerabilities.
Two decades ago these scanners were just a few extremely expensive, Windows-only and awkward applications, which I only saw actually used at the largest banks: nobody else could afford them. Today the situation is much better: there are decent open-source code and dependency vulnerability scanners for Python, Ruby on Rails, Java and C/C++, and there is competitive market of commercial scanners out there too.
A side note: why there are so few open-source scanner projects? The answer is in economics: it’s easy and exciting to develop the scanner engine but an engine without rules is useless. Now, developing and updating the rules takes a lot of time, committment and is anything but exciting. This is why really good and up-to-date tools cost money: you’re not paying for the scanner, you’re paying for the time of people who patiently add detection rules for new vulnerabilities as they are published, day by day.
The rule of thumb is get any source-code scanner as early as possible into your pipeline. If you can’t get budget for a top-shelf commercial one, integrate an open-source one, if available for your programming language (more on this below). Then use anything found by the open-source scanner to build a case study to get budget for the top-shelf one.
The hard part starts when you actually get the budget: which scanner you buy? The market for security scanners is quite mature and competitive today, which unfortunately means a lot of marketing hype, false promises and yes, let’s say that straight, plain BS. The rule of thumb at this stage is always evaluate the candidate products at your actual code base. And yes, vendors facing potential at tens of thousands of dollars price tag will let you evaluate their products at least for a week or so. And this is more than enough.
A week is more than enough to notice the following surprises that are rarely highlighted in marketing leaflets:
- “the scanner does tick the JavaScript support check box, but hey, you didn’t say anything about Node or ES6”
- “yes, we fully support Java so if you wrote your multi-tier web app in pure Java rather than some modern Java framework we surely would find something useful”
- “well, that ORM abstraction you’re using is on our todo list but for now we can analyze SQL queries built using string concatenation”
- “your Linux-based build environment will surely enjoy a Windows 2003 Server and MS SQL because this is what our scanner requires”
These are all actual failed promises from my first-hand experience over the last decades where a scanner selected previuosly by the client was there, and kind of ticked the “we use SAST” audit box, but in fact it was completely useless. For example, because the client developed a decent DropWizard web application on JDK 1.8 and the scanner only happened to support J2EE web services up to JDK 1.4, so APIs that were used around 2003. This can be actually dangerous, because you kind of do SAST, and the “recognized” and “market leading” product kind of reports “no vulnerabilities”, but it’s only because it’s so outdated that it cannot really see any data flows in the application!
So do evaluate the shortlisted scanners and preferably evaluate them in your actual development environment. When doing this for clients, I follow a thorough two page checklist which I developed specifically for this purpose based on my past experience that covers everything from the actual (not only “declared”) language support to available integrations with ticketing systems like Jira, Slack etc.
Specifically for that purpose I have also developed this DropGoat Java web app with some intentionally planted vulnerabilities. Feel free to use it to evaluate Java scanners, but be aware of the fact that it’s a few years old now and scanner vendors might have easily optimized their products for it.
If you are looking for open-source scanners, these I can recommend from personal experience:
- Bandit for Python
- Brakeman for Ruby on Rails
- SpotBugs for Java (evolution of FindBugs and FindSecBugs)
- safety dependency scanner for Python
- OWASP Dependency Check dependency scanner for Java and JavaScript (client-side libraries)
- OWASP ZAP dynamic scanner which I have successfully scripted in headless mode to run against QA/UAT environment