Product Security Roadmap

Introduction

If you haven't seen it already, I wrote about some basic fundamentals to consider when building a Product Security program in an organization - Building a Product Security program from scratch.

Continuing the series, in this post, I will cover how one can reason about a multi-year ProdSec roadmap and also discuss a little about Asset Inventory. So, let's get started.

Product Security Workstreams

In the first few weeks as a founding ProdSec engineer, just wrapping your head around all the attack surface your org faces, identifying some of the biggest security gaps and improving visibility into those gaps will most likely consume the majority of your time. Not to mention, since you are now the resident AppSec/ProdSec expert, you are probably going to be inundated with all sorts of requests coming from all directions.

This ad-hoc approach of dealing with fires will only last a short time before you realize you cannot move the security needle and haven't been able to do any meaningful work that can have a long lasting impact across the org in maturing its application security posture. If you don't stop, take a step back and start strategizing how you want to approach the different sets of problems with different priorities, you risk burning out pretty quick. Burn out in the security world is actually quite common and I've been burnt out myself many a times before so I have first hand experience of it. It is important to realize you cannot solve all the problems and you are not the one going to be solving all of them. Not all problems need solving anyways. Unless you let some fires continue to burn and focus on some key priorities, it is going to be very challenging to get anything done. If you don't believe me, listen to what Caleb Sima has to say regarding fires in this podcast.

One approach that has helped me tremendously is to come up with a high level ProdSec roadmap (even though it's just that - a roadmap with nothing concrete set in stone) and socializing it with the rest of the security team. This will allow you to get more organized in your thought process, level set your priorities, track dependencies with other teams, get agreements on some common goals/milestones and communicate how you are thinking about building this program so that others can provide their feedback if they disagree with anything. You might have to revisit it a few times, change a few things here and there but doing this in the beginning will allow the key stakeholders to support you the best they can and will set you up for success more than anything else.

Hopefully, I have made it obvious by now that you need some sort of a ProdSec roadmap so I will shut up and share how I tend to think about it. I like to broadly categorize 3 workstreams in any ProdSec program:

Please note that each one of these workstreams below can be its own team and operate independently with its own leaders. My point is to get some projects started across each one of these workstreams so that you are not completely ignoring one and spending way more time on the other. Remember the idea is to work on tactical projects that eventually roll up under an overall strategic plan.

Vulnerability Management

• Continuous Vulnerability Identification & Discovery - continuous vulnerability scanning via different vulnerability scanners
• Vulnerability Triage - calculating severities/risk
• Vulnerability Reporting - attributing vulnerability tickets to the right owner
• Vulnerability Remediation - setting SLAs and enforcing them
• Centralized Vulnerability Tracking - track vulnerabilities reported from various scanners in one place

Security Partnerships

• Secure SDLC (Secure Software Development Life Cycle)
  • RRA (Rapid Risk Assessment)
  • Tech / Architecture Design Reviews
  • Threat Modeling (App + Deployment)
  • Secure Code Review
• ASVS (Application Security Verification Standard)
• Security Scorecards
  • Dashboard to get data from various tools and bubble up that data into a score → show this score in Github repos (as badges) and gamify with instructions on how to improve the score
  • Check this article from Chime on some ideas of metrics/data points you would want to collect for the security scorecard
• Security Champions
• Customized Security Training & Education

Security Tooling & Operations

• KPIs / Metrics - things you want to report on to the C-suite to show progress
• Security Scanning Tools
  • Setting up SAST, DAST, SCA, etc.
  • Maintaining and fine-tuning the tools
  • CI/CD Integration
• Secure Defaults - secure by default libraries & design patterns
  • Example - Some secure default policies could be set at Github org level using AllStar
• Bug Bounty
• External Pentests
• Product Security Incident Response - setting up ProdSec IR processes and runbooks
• Automated AppSec Platform - think of this like a one-stop security dashboard that has data about all things security constantly getting updated

High Level ProdSec Roadmap

Based on the 3 categories discussed above, if it's not obvious already, it is just not humanely possible to do all of these things without a multi-year plan by a team, let alone by a single individual. And, as noted above, you would want to work on projects that fall under each one of these categories so you are making progress on all fronts. Below is a rough breakdown of how that might look like for a 3 year roadmap and by the way, I think this is an extremely aggressive roadmap. Realistically speaking, this can easily be turned into a 5 year roadmap. On the contrary, if you can get some help i.e. hire folks with different skills, you can also get started on some of these initiatives sooner than later:

Please note that the roadmap below only highlights the start of a project/initiative. It is very much the case that most of these initiatives will likely turn into KTLO (Keep the lights on) activities once its established and constant improvements would have to be made to keep it running and successful.

Also note that the below rows are not in order of any priority. You can be tackling Secure Defaults before DAST. That is something you'd have to work with your stakeholders to determine what makes sense to do when.

Below is some of my thought process as to why this breakdown above:

Let's talk about the Vulnerability Management workstream first.

TL;DR - You can focus on establishing a process (Vulnerability Triage, Vulnerability Reporting and Vulnerability Remediation - Manual Followups) first and then dealing with Continuous Vulnerability Identification & Discovery, Centralized Vulnerability Tracking and Vulnerability Remediation - Automate SLA Enforcement at a later time. Remember to pick and choose your projects wisely so as to not completely overwhelm yourself.

As you are trying to establish a ProdSec program, you are probably wondering how to even find the vulnerabilities in the first place i.e. what scanner to run at what cadence, how to deploy it, where to deploy it, whether to buy or build a scanner, what to do with all the vulnerabilities it is going to find, etc. Let me make it simple for you. In the beginning, it is probably not going to be worth thinking about these things (just yet). Your org was functional before you joined and it will continue to function even if you don't do a single thing wrt vulnerability scanning and discovery. Please don't miscontrue this as Don't use vulnerability scanners. That is NOT what I am recommending. I am simply trying to help breakdown the priorities here. Vulnerability Scanners are good but only when you can dedicate your time and resources towards it. Otherwise, it will simply end up being a total drain and a deterrent from other higher priority items that you could be focussing on instead. Same thing applies to tracking vulnerabilities in one place. You need a good pipeline of incoming high fidelity vulnerabilities in the first place to be able to start worrying about tracking them all in one place.

Now, having said that, you will stumble upon vulnerabilities by digging into interesting areas - either via conversations or when you conduct Secure SDLC activities such as Threat Modeling and Design Reviews (more on this later). And, that's why, I believe defining a basic process to deal with vulnerabilities is paramount to any org so that you can speak the same language with your engineering counterparts. The basic process can simply mean that you define the vulnerability lifecycle (discovery -> triage -> reporting -> remediation) and socialize it with the engineering org. Thus, the most important things you could be doing in the beginning wrt Vuln Management are:

• Explain how you plan to triage the vulnerabilities (Vulnerability Triage)
  • How you calculate the severity - whether it is via CVSSv3 or something else
  • How you assign SLAs based on different severities
• Explain how you plan to report the vulnerabilities (Vulnerability Reporting)
  • How are you going to report these vulnerabilities to the correct owner
  • Where (which project, which bug tracker, etc.) will these vulnerabilities be reported
  • What are the different fields in the vulnerability ticket and what they mean
  • What is the expectation from the owner once they get a ticket assigned to them
• Follow up on these vulnerabilities manually (Vulnerability Remediation - Manual Followups) to track their remediation before their SLA due dates because the volume at this stage is going to be manageable without completely bogging you down. When you eventually get around implementing scanners to run on a cadence and have a steady incoming stream of vulnerabilities, you would need to ensure you have some way of auto-enforcing these SLAs based on their severities.

To be honest, Vulnerability Management deserves its own blog post because there are a lot of nuances which I haven't covered yet. I will get around writing more about it in the upcoming posts. Stay tuned!

Next, let's talk about the Security Partnerships workstream.

TL;DR - Invest in educating about the basics of threat modeling, design reviews, code reviews and risk assessments by conducting these consistently and manually with the engineering teams until there comes a point when it becomes second nature and starts to reflect in the engineering culture.

I believe the most important thing you can do on this front is to establish (define and implement) the various processes / activities that are integrated into the software development lifecycle to ensure security is considered from the first step and not as an after-thought. These activities are things like Threat Modeling, Design Reviews, Secure Code Reviews and Rapid Risk Assessments. I plan to write about these in more details in a separate post (most likely after this one).

Doing so will promote the right security culture from the get-go across the wider engineering org. Also, the more you do these activities with consistency and rigor, the better you will get at it and you will start noticing that it starts to percolate through the engineering culture as well. As an anecdote - In one of my previous workplaces, I wrapped up a threat modeling session with an engineering team and the developers were mindblown at how little they knew about the overall architecture of the system they have been working on. After the threat modeling session was over, the developers thanked me for showing how interesting threat modeling can be by just asking simple questions and having them think like the bad guys. A few months later, one of the developers ended up switching teams and working with me on the security team. I consider this the best way of recruiting security engineers (or passionate engineers who are interested in security).

Having said all of that, I firmly believe you should spend the majority of your time early on in doing these activities manually on the most risky products/features/systems as opposed to doing them for every feature request and every product release. The reason I think its worth calling this out is because you might have heard of terms like threat modeling as code and the fact that that's how you could scale threat modeling in your org. Based on my experience, I can tell you that threat modeling as code almost never works in a smaller org in a fashion that would allow you to actually have meaningful impact in improving the app security posture and shifting security left. Developers will struggle to understand the what/how/when behind it and you will struggle to keep it consistent across different architectures and tech stacks.

Overall, there is definitely going to be some hand holding early on as you drive these processes but I can guarantee that it will reap great benefits in the long run. As for how to continue maturing this workstream, once you've reached a point when you start sensing that engineers understand the importance of doing these activities and are aware of the processes, you can start investing in some bigger initiatives such as the ASVS, gamifying via Security Scorecards, instituting a Security Champions program and automating some mundane manual steps in the Secure SDLC program. I will try to cover those as well when we get there.

Finally, let's talk about the Security Tooling & Operations workstream.

TL;DR - Invest the most time here in the beginning. Set up some basic but high fidelity scanners with a constant feedback loop from the engineers. Focus on metrics that are meaningful and actually matters to you and the C-suite. Keep evolving them. Pay attention to any secure defaults that could be implemented as and when you learn about vulnerability classes affecting your org.

You will notice that the first year is pretty heavy with projects under this workstream and there is a reason behind it. Most of the work that you would be doing here will not only help in improving your visibility across your attack surface but also aid in your compliance initiatives. Things like setting up SAST and DAST are some of the basic things any ProdSec program is required to do and there are plenty of open source tools available these days that can help you get started with very minimal lift. If you want the maximum ROI, the key is to integrate these scanners in your CI/CD pipeline so that you setup the feedback loop (for the developers) in the beginning and then continue to fine tune your tools as and when necessary.

If you setup these scanners in any path that is new for developers, you will more than likely struggle to get the devs to pay attention to the results. Along the same lines, if you start blocking PRs when you are not sure about the findings being true positives, it will create friction with the developers. The trick is to enable the scanners and passively monitor it (not block PRs), keep fine tuning it until you are reasonably confident that the rate of true positive vs false positive is high enough for you to start blocking PRs. Achieving this balance is difficult but necessary for your engineers to take the results from any sort of security scanning seriously. Security scanners have a lot of bad reputation (and understandably so) and it is on us security folks to be empathetic and mindful when we implement them. If we are creating more work for our engineers (by cutting them vulnerability tickets), it is only fair we are doing this if its necessary and not wasting their time.

Establishing KPIs and metrics will also be an integral part in defining how successful your program is going to be. You can start with some basic metrics that you collect manually and then over time think about ways on automating their collection.

Also, as you implement the SAST and DAST scanners, you are going to stumble across some insecure patterns and also some secure-by-default libraries and frameworks for the languages in your org's tech stack. In order to make security easy and effortless for developers, you should be making these libraries/frameworks available to the developers, provide them code snippets and examples on how to integrate them and subsequently monitor their usage/non-usage. Taking this approach will allow you to understand the reason why certain libraries get adopted vs why certain don't. You could then use these insights to eventually help drive the adoption of secure defaults.

It is probably worth calling out a few other things under this workstream:

• For SCA (Software Composition Analysis) scanning, you could work with your Infra/SRE teams and enable Dependabot/Renovate on your repositories to get additional visibility into your 3rd party library ecosystem before deciding what to do with them. This is not just a security problem but a generic supply chain issue in the SDLC and with incidents like CodeCov and NPM, I hope it has proven that vulnerabilities in software supply chain should really be treated as an org priority and not just security.

• You could use your SAST scanner to scan for known and generic secrets in the code. It works fairly well but is obviously not the same as secrets detection at scale. I believe there are other priorities to be worked on before spending time on it hence I have it under the FY3 column.

• Some smaller companies like to start with bug bounty programs early on as a means of getting more visibility into their attack surface and leverage the power of crowd sourcing. If you have the $$ and the resources for it, I think this is a great idea and I would recommend it with the caveat that you should know what you are signing up for with bug bounties. Based on my experience, one should look to first build a strong understanding of the gaps and controls within their org and then tailor the scope of bounty programs such that you are incentivizing the researchers to find the things that matter the most to you.

And, if this isn't enough already, there are a few other things that I haven't included above. As you grow your Product Security team, these become important as well:

Other ProdSec Areas

• Vendor Management Program
• Hiring
• Evangelizing
  • Hackathon / CTFs
  • Lunch and Learn / Brown Bags
  • Office Hours

Asset Inventory

Shifting gears a little, one other thing that you should/could be doing simultaneously is to maintain an updated Asset Inventory. I am sure you have heard of the saying - You can't protect what you don't know.

I don't think there is a need to make Asset Inventory a complicated project in the beginning when you are the only one building the ProdSec program. It doesn't have to be manually updated every time but I am assuming you can't get it an automated fashion in the beginning anyways. So, feel free to use the template below to get started and gradually evolve/automate it as you mature. It goes without saying but having something like this is not only helpful for you as the founding ProdSec engineer but provides tremendous value to other folks in the org as well. Matter of fact, when I built something like this in one of my workplaces, a lot of folks from different business units reached out to me thanking me for building it and offered to help me keep it up-to-date. I couldn't have expected for more. Keeping it simple and accessible/search-able to anyone is the key here.

Now, having said that, I will admit that after a point, maintaining it will become a hasssle and you might not always have it up-to-date but I hope by that time, you know enough about the org that you don't have to refer to this table every time you need to know about a product or a tech stack. That is also a good indicator that you might need to invest in something (automation?) more than a manually updated asset inventory table.

...and, that's a wrap for this post! This post ended up being a lot longer than anticipated but hopefully I was able to get my message across. In the next post, I will dive into the Secure SDLC program and how to go about structuring the activities as the founding ProdSec engineer.

Until next time, ciao!