Building a Product Security program from scratch

Prologue

Hi! This is my attempt to reinvigorate this blog by sharing my experiences of building security programs from scratch (more specifically Product/Application Security programs). To provide more context, I have been very fortunate to have worked in multiple companies over the last decade or so - both big enterprises and startups so all my experience is a culmination of the various problems I've seen in these companies and the lessons learnt from trying to solve them. In some cases, I was successful in my approach and was able to have a big impact across the entire company and in some cases, not so much.

Regardless, I have learnt some invaluable lessons over the years and I hope that the things I share here can help folks who are either just getting started or are simply looking to learn more about other experiences. In return, I am looking forward to learning more about how others have approached similar problems in their organizations.

Feel free to reach out to me with any comments/feedback that you might have and I am happy to engage in a dialogue and collaborate together. If you find the content helpful and have ideas about what else you might be interested in reading about, please let me know that as well. You can find my contact information and little more about me here.

PS - For those who might have noticed a new look to this blog, this is because I am trying to learn about frontend frameworks (React et al) and how to make pretty websites in my spare time. I stumbled upon this tutorial from Vercel, the founders of Next.js and I can't recommend it enough. This blog is basically a result of me following that tutorial. It must have taken me a couple of hours to get it up and running with no prior knowledge of React whatsoever. The tutorial goes over the details of each and every concept including (what I consider, one of the best deployment experiences ever) how to deploy it within seconds.

Introduction

The way I envision writing about the topic Building a Product Security program from scratch is via a series of blog posts covering the different aspects of a product security program in more details. There will obviously be an overlap with other information security domains such as Cloud Security, Incident Response, etc. but I will try to stick to the ProdSec/AppSec domain as much as I can. It is also worth mentioning that many organizations use both Product Security and Application Security interchangeably. Some orgs consider AppSec a subset of ProdSec. I personally don't have a strong opinion on this and I am going to be using both the terms loosely in this blog.

My focus will be more on how to actually go about implementing these programs hands-on as opposed to general guidance and more philosophical approaches. I will be sharing real world examples/code snippets/what-have-you to provide as much context as possible so that you can make sense of it with respect to how it might relate to what you are trying to do at your organization.

Also, before I forget, some blogs/podcasts that have helped me immensely in shaping how I think about these programs that are worth mentioning are:

So, without further ado, lets dive right into it.

What it means to be the founding ProdSec/AppSec engineer

Okay, so lets say you've landed in a role where you are the founding Product Security engineer and are asked to build the Product Security program from the scratch or take it form 0 to 1. The first question that probably comes to your mind - Where do I even begin? Before even thinking about the foundations you are about to lay down, I think it is extremely important to understand what it means to be the founding ProdSec engineer.

Understanding the Organization

This goes without saying but every organization has different needs, risks and priorities so a product security program will need to be catered as per these requirements and there is no one-size-fits-all approach as such. Having said that, one thing that is paramount to a successful ProdSec program or for that matter any security initiative is whether you are bringing the Risk down for a particular org or not. Keeping this in mind, I recommend focusing on the following things:

Risk Appetite

TL;DR - You should be talking the same language with the C-suite wrt the risks that your org faces. You can do this by defining a risk model and building a risk register.

You should have a good understanding of what the risk appetite is for your org. In other words, how much risk are the C-suite willing to take in lieu of making decisions around business growth vs addressing security concerns. There are 2 key things you can do here:

  • Risk Model - This should really be the first step that you could/should do at your org if it doesn't exist already. Something like below should help you come to a shared understanding of how risk can potentially be quantified (albeit still be subjective) and how can it be calculated somewhat consistently without having all the information.

risk-model

  • Risk Register - While socializing the risk model and understanding the risk appetite of your organization, you will encounter some things that should be documented in a Risk Register for posterity. Consider this a good starting point in your Risk Reduction journey. You can do this in whatever tracking system your org uses, whether it is Jira or Asana or something else. Once you define the risks and add them to the risk register, the following becomes equally important:

    • Having the right owners assigned to each risk
    • Communicating the risks to them and getting their buy-in/agreement on it - whether it means they have accepted the risk or would like to address it or something else (Transfer/Avoid/etc)
    • Assigning SLAs and following up on them

    Having this Risk Register accessible and available to the key stakeholders is extremely helpful in ensuring you are talking the same language with the C-suite and anybody else involved. Not to mention, this will be your source of truth for all things Risk moving forward and will help later on with things like hiring, budgeting, etc.

Priorities

TL;DR - You are the only ProdSec engineer so you have to align your priorities with the business priorities in a way that doesn't disrupt the business, reduce developer velocity and introduce additional risk to the org. And, you do this while ensuring you have a long term strategy of improving the security posture of the org by implementing short term tactical approaches.

I am sure the word priority gets thrown around quite a lot in the InfoSec world and there is a reason behind it. Security in any org is very much an operational/support business unit that doesn't necessarily bring in any revenue to the org. And, hence there is a constant battle trying to get budget and resourcing for security teams. Unless you ruthlessly prioritize, this is a losing battle, one that I don't recommend fighting. This is very much a known problem and shouldn't come as a surprise to anybody. This is similar to the security vs usability discussion, although not quite the same.

It is important to understand what the priorities are for any org and then make security a feature for that particular priority - whether that priority is launching a product within a deadline or an entire business unit that is mission critical to the business. In my experience, considering security an important aspect and a feature doesn't necessarily happen unless you lead folks to come to that conclusion themselves.

To give you an example:

Consider a scenario where you conducted a gray box pentest for an upcoming Product X (not yet launched) and found a glaring authorization vulnerability, one that might potentially affect other products and risk exposing customer data by unauthorized actors, if it were to be exploited. It is difficult to say for sure without further investigation. But, you haven't had much luck getting it addressed because it was not a priority for that particular product team and so getting resources committed to fixing it or even investigating it had proven to be a challenge in the past.

Now, there is a push within the org to launch a Product Y in the next 30 days. As the only ProdSec engineer, your challenge is to ensure the launch goes smoothly yet ensuring there is some sort of security assurance that this product doesn't get shipped with any high/critical severity vulnerabilities. You conduct a time-boxed gray-box pentest of Product Y only to find out that the same authorization vulnerability you found in Product X also affects Product Y. Not just that, since it is a homegrown authorization scheme, the vulnerability exists in the middleware that is consumed by all products and thus it affects the entire org (all products), not just these 2 products. Some things that you might immediately start thinking about:

  • How do you get this addressed?
  • Do you get this addressed for all products or just Product Y?
  • Can you even get this addressed within the next 30 days?
  • Are there more such issues that are currently unknown?

You can see it gets overwhelming pretty soon and these are just a handful questions that I can think of off the top of my head. Situations like these are not uncommon in the security world. Balancing the risk along with business priorities is something that doesn't have a right or a wrong answer and you just have to work with the appropriate stakeholders to come to a decision that is mutually agreeable and understood by all parties.

In this situation, realistically speaking, it is impossible to fix the root cause of the authorization issue across all products before Product Y gets launched. And so, the focus should be to stop the bleeding in the short term (tactical) while continuing to make progress to fix the root cause in the long term (strategic). You might say - That's fine and makes sense but how do you actually do that? Well, let me try to break down a more realistic situation for you:

  • The first thing you should be doing is to build a proof of concept of how a bad actor might exploit this vulnerability to gain unauthorized access to the customer data.

  • Next, consider the Risk Model shared above to calculate the Impact and Likelihood and then come up with the overall Risk that is introduced to the org as a result of this vulnerability.

  • Communicate this with all the stakeholders so that everyone understands how bad it is and what's at stake here. Get their sign off on either addressing the risk or accepting it so that it's all documented and visible.

  • If the decision is to address the risk before the launch of Product Y, work with the appropriate dev teams to come up with a temporary mitigation strategy. For example - decorate/guard the vulnerable endpoints in question with a function that sits on top of the vulnerable middleware framework and is responsible for performing the authorization checks that are missing but are needed. Also, make sure you have the logs necessary to investigate any abuse of this temporary hack, if need be. This should be implemented before the launch.

    The crux here is that by aligning your priority (of getting the authZ issue addressed) with the business priority (of launching Product Y in the next 30 days), you have now made security an important feature, without which the product cannot be launched and has now become a priority for everyone which is something you've struggled with in the past. Although the way it is getting addressed might not be ideal, it still ensures the customer data is not at risk and you are making some progress towards your end goal.

  • Investigate any other exploitation activity in existing products (already launched and in production) that might have been exposed by the same vulnerability following the same approach to mitigate. One way to do this would be to run your static analysis scan (more on this in upcoming blogs so stay tuned!) to identify usage of the same middleware authorization framework in other repositories and triaging it as you find them. Also, work with your incident response counterpart to ensure you are following the right procedure to disclose to your customers if their data was breached and fulfill any other legal obligations, as applicable.

  • Conduct an in-depth root cause analysis / postmortem of what the issue is, how it was introduced, etc. Try to get as close as possible to the root cause so that it can be addressed there. Simultaneously, monitor any PRs (via static analysis) that might be using the same middleware in any new code being committed. Make sure there is a strategy to explain to devs what they should be doing in the interim while the root cause of the issue gets addressed. Don't let them hanging and confused. Definitely, don't block them from committing code.

  • Start a guild/workstream comprising at least 1 person from each team that might have been affected by this vulnerability and work with them in coming up with a long term strategy on how to remediate it (not mitigate) and how to go about rolling it out in phases with a constant feedback loop.

    As an example - the long term strategy could be replacing the current authorization middleware by implementing something like OPA (Open Policy Agent) for all things authorization (specially if your environment is primarily composed of microservices) and you could start rolling this out with one team, learning from that roll out and gradually expanding it to the rest of the org. Getting commitment to dedicate resources from each stakeholder involved for such a herculean effort is going to be crucial to successfully address this problem.

Lay of the land

Apart from building a risk model for your org and coming to terms with its priorities, one thing that can also be very helpful and complement your efforts of improving the security maturity of the org would be look at all the previous vulnerabilities that were known/found, whether they were discovered in a pentest or in an internal bug tracker. Having some idea about the kinds of vulnerabilities or bug classes affecting your org could potentially armor you with some good starting points to dig into straight off the bat.

Don't just stop there. Continue documenting the core technologies, platforms, languages and stacks used by your org as you learn about them. You will need all this information later on when you start looking into secure defaults and solving security for developers so that they don't have to worry about it.

Building Relationships

TL;DR - Sorry, no TL;DR here. This section is prolly the most important thing you can do as a founding ProdSec engineer

The second pillar (this should really be the first) that I feel the founding ProdSec engineer should really drill into is to build long lasting relationships with all the stakeholders in an organization. Being the only security person responsible for the product security of an entire organization can be overwhelming enough and you should be looking for your stakeholders to champion your initiatives as much as you can muster them to. Think about it like an un-official Security Champions program.

One thing that has really worked for me is to have an empathetic attitude towards engineering teams when approaching these conversations with the engineering leaders. Explaining your vision and your approach to that vision in a way that conveys the right message to these leaders can make your job a lot easier. This might sound like a cliche to some seasoned AppSec folks but stressing on things mentioned below really helps in driving your point home:

  • Security is everyone's responsibility
  • Security folks should be considered as enablers vs blockers
  • Advocating for Guardrails vs Gates
  • Trusting your developers to do the right thing

A few key things I would recommend to help build these relationships:

Know your EMs and Staff+ engineers

I know some security folks like to sit on engineering scrum calls so as to integrate themselves into the dev lifecycle to understand the pain points and identify points of integration but in my experience, I've found that that's just not scaleable, specially when you are the only one doing product security. Instead, what has worked for me is to build relationships with the engineering managers of those dev teams and key staff+ engineers who are likely more integrated with the dev teams than you can ever be. Think of these folks like your security champions if you will. Share your roadmap with them, learn about their roadmap, talk priorities, ask them what keeps them awake at night wrt security, etc. You get the idea! A bi-weekly 30 min meeting can do wonders here to further your mission.

Know your organization chart

This might sound unusual but one of the struggles I've had in my experiences is to constantly wonder which team a particular engineer is in, what are they working on that eventually falls under an org priority, who do they report to, etc. It might not sound as important but trust me, if you have these answers readily available without spending a lot of time asking folks, you will cut a lot of your time simply chasing down the right folks to have a productive discussion.

Consider this more like bottom - top approach where an engineer who worked on a particular feature and is the closest to it can help you with something in a few mins as opposed to you trying to approach their EM and trying to have the EM help you direct to the said engineer, only to be told that their sprint is packed with no room for any additional work. The EMs will try to protect their engineers - which btw is totally fair. You just have to be smart about how you approach the situation. As a bonus point, if you approach this engineer directly with the right attitude (knowing exactly what you want, doing you research beforehand and respecting them for their time), you are going to build a personal relationship with them which will go a long way.

Get involved with an ongoing project

I firmly believe a security engineer can evangelize developer empathy as much as they want to but to truly feel it, you've got to directly drink from the firehose along with the developers. In other words, identifying an ongoing project where security might play a key role and participating in the sprint with other developers (whether committing code or just shadowing somebody) will help you understand the lifecycle and the processes and things that you otherwise wouldn't know about by just talking to folks. You can then tailor your approach accordingly and the dev teams will start considering you as an extension of their team - which is where you want to be really. This approach is great for building rapport with any team and has worked wonderfully for me.

What's next?!

I think I have written enough about some general but extremely important things above that you should be cognizant about as a founding ProdSec engineer. Next, in the upcoming blogs, I will continue to walk through some actual initiatives/programs that might be helpful for you as a reference to start your own Product Security program. Here is how I am thinking I will break it down (Disclaimer: The order might change):

  • Asset Inventory
  • Product Security Roadmap
  • Vulnerability Management
  • Secure SDLC - Crawl
  • Secure SDLC - Walk & Run
  • KPIs/metrics

I hope you enjoyed reading this and could resonate with it! Stay tuned for more..

If you like the content and don't want to miss out on new posts, enter your email and hit the Subscribe button below. I promise I won't spam. Only premium content!
← Back to home
AB © 2024