Focal security team principles
1. Fix Technologies Hindering Your People & Processes
When discussing the people, processes, & technologies (PPT) of security, most leaders will speak fanatically about how the people that make up their team are the most valuable resource and make the real difference. I agree – which is why you need to do whatever you can to improve the technologies & processes that you are asking them to endure 40 hours a week.
The technology stack of a security team – especially in a startup – can become stale to the future of the organization if you aren’t paying constant attention. For instance, a number of custom microservices and tools were used at Fullstory when I arrived, and over the past months, we’ve sunset entirely without decreasing our security program’s coverage and value.
Over the past two years we’ve replaced every key technology with a focus on coverage of resources, data quality, value for the cost, and ease of automation. The right technologies allow our team to maximize their expertise, reduce frustration, and secure the business at scale.
2. Security teams must take on the pain first
Security teams have two crucial decisions when it comes to enabling any new technology: what do we show our engineers and when do we do it? I see many security leaders buy new technologies (e.g. SAST, SCA) and immediately shim them into the CI/CD pipeline with the expectation that engineers will reach out to the security team if the findings are incorrect.
I’ve always operated teams with a different model: security needs to take on the pain first. This means that we triage a new technology’s findings, tune our configuration, identify bugs, improve features with the vendor, remediate common anti-patterns at scale – and then we give the signal to our engineers. This ensures that we are accountable to our customers and understand what we are asking of them. A security team that is empathetic to the pain is incentivized to lessen it.
3. Promotion-path your security technologies & processes
In an additive way to the prior principle, when your security team is taking on the pain first, you are more able to determine when a given technology, feature, or process is ready to be exposed to your engineering customers. Importantly, this is unlikely to be a boolean and instead is more likely to be a granular promotion path of a feature where you increase visibility slowly as confidence grows that alerts are well-tuned, severities make sense, and rules are high signal.
At Fullstory, we’ve been – over months – rolling out automated security alerts for our SAST & SCA features to engineers by severity level as the findings we had to initial triage – remember, it was our pain first – became healthy enough that we didn’t expect to see e.g. high, critical alerts regularly anymore. As that technology maturity continued to grow, we expanded the number of code bases it applied to and kept lowering the severity threshold. Over time, engineers will receive most alerts in real-time, and we expect to revert that if our quality bar is not being met.
4. Build a program that matches your engineers’ speed
An anti-pattern of security teams that I often see play out is that the program is implemented without considering the actual software development life cycle (SDLC) of the company. This is readily observed at a company where engineering teams are shipping updates to production on a weekly or bi-weekly basis. Of course there are companies that ship thousands of releases a day – but is that your company? Is your pace of security too fast? Does it really add value?
Each alert sent. Each “Red X” shown. Each Jira task created. Each CI/CD failure. It all adds up.
Security Engineering at Fullstory is specifically focused on meeting our engineers at the best parts of the SDLC so that we balance over-informing them about issues that don’t need urgency, reducing frustration, stress, and feeling like every mistake will be a new headache for them.
We tend to take a “pipeline-less” approach where most of the visibility engineers receive comes from findings associated with pull requests so that we can have a discussion with them before a merge, if necessary. Ideally, when resources are being built (let alone shipped), there’s been plenty of time in our typical SDLC to think thoughtfully through the topic and collaborate fully.
5. Focus on what actually matters
Let’s face it: cybersecurity has become an industry of inundation. More tools. More issues. More alerts. More frameworks. More best practices. The painful reality? 99% of what we see doesn’t matter that much. Those 7,000 NodeJS transitive dependencies you just patched? Probably didn’t actually change your business risk in the least. But it felt really good to get it done, yeah?
One of my favorite phrases to repeat is, “Not everything interesting is important.” Hearing about a cool new CVE or a fantastic research project at Black Hat has to be pressure-tested against the reality of your own circumstances. Understand your threat model and secure it.
Maturity guardrails
Each of the programs I’ve built in security engineering has been built up with a maturity model at their foundation. Importantly, a maturity model is not a checklist – it acts as a guardrail to your program building so that you don’t have an unknowing imbalance in your investments. At Fullstory, we are leveraging OWASP’s Software Assurance Maturity Model (SAMM), and I have also used the Building Security In Maturity Model (BSIMM) from Synopsys with great outcomes.
On occasion, I hear security leaders talk about their amazing red team function – all while they lack comprehensive security visibility and high-leverage controls. Having a red team isn’t a bad thing; it’s simply that needing one is best served when the program is already at a high level of maturity overall. If your red team is able to do a touchdown dance every afternoon, you’ve likely over-invested in offensive testing before you’ve focused on resolving large program gaps.
In this way, using a maturity model helps keep you honest to yourself and the organization – blind spots in security are a real risk and “checking in” with yourself about the scope of your program with something beyond your own experiences and implicit biases is a healthy thing.
Most importantly, though, teams that use maturity models can speak the same language about the areas necessary to improve, the progress being made, and the “next steps” to chase after. Still, a maturity model is not a rigid structure and should be used as a tool, not as a checklist.
Currently, we update our maturity model status in a simple spreadsheet, tracking each of the 294 individual security activities in OWASP SAMM with a maturity score and annotation. Each of these updates is added to a new tab for a given assessment period, allowing aggregation and analysis to understand the changes over time and keep an eye out for any stagnant areas.
While that may seem daunting, the first assessment is by far the most time-consuming. Part of that time should be spent validating assumptions with stakeholders who may have competing insight. The goal of that baseline assessment is to truly understand the current state of your program so that all updates are founded in reality with good data and perspective to work from. As your program matures, you’ll be able to map your quarterly projects to maturity activities, which creates a tight relationship between your program status, planning cycle, and outcomes.
Measuring the details
If you consider a maturity model as a foundational element of your broader program, you can think of reasonable key performance indicators (KPIs) as the granular companion to show the granular attributes for larger, more general security activities. For instance, you may decide that your vulnerability management program is perhaps early in maturity, but unless you break down that “boiling the ocean” problem into smaller areas of focus (e.g., containers, virtual machines), you are likely to make less progress on creating a strategy that truly resolves that macro issue.
At Fullstory, the KPIs we track often scope to an individual control area from one of our security technologies (e.g., ASPM, CSPM), and we use an approximate process such as the following:
Prioritize the right work – based on risk profile and scale, what should our next focus be?
Understand the data – determine the root cause(s) for the gap, including historical reasons
Ideate solutions – don’t play whack-a-whole; how can we simply & uniformly solve this?
Coordinate with stakeholders – few problems can be solved with just the security team
Work the plan – keep an eye on a single target metric that informs this KPI’s progress
Stay healthy – once we make a concerted effort, we have hyperfocus
to not let it drift
In many cases, we try to make smaller KPIs to start that eventually get consumed by broader KPIs later on. For example, if you want to track a KPI for unresolved High + Critical results in third-party dependencies, start by focusing on core product repositories first, and as time goes on, increase the scope to include more repositories until you’ve gotten to 100% coverage. Overwhelming problems are only that way when you are trying to take on too much at one time.
Lastly, there’s often an industry fixation that KPIs are used to appease executives and boards. While they may demand certain KPIs, don’t conflate what is asked of your team by stopping the team from curating their own high-value KPIs that are tracked only for your own purposes. There is a psychological value in teams that track their progress – security is a slog many days, and unless you can observe when the wins are wins, it can be hard to feel the momentum growing.
Our security stack & partners
Security leaders coming into a new organization have a pretty daunting task of determining which existing security-related vendors are meeting the needs of the business – and can scale, too. That scale isn’t just about technical limitations; it’s also often about the cost of operating the stack. There’s always a risk that a new leader applies the same stack they’ve used in previous years but without considering how that matches the needs (e.g., technologies, budget, maturity) for this current role. Notably, new leaders may also have to deal with burning down prior multi-year agreements that effectively lock them into a technology that perhaps isn’t the right one anymore.
One tip I’ve used in multiple roles is to track my security capabilities to find solutions to spot gaps or duplicative investments when defining what a go-forward strategy may need to look like. So instead of saying, “I have Solution X for SCA,” assert that, “I can perform third-party dependency vulnerability analysis on Cargo-managed packages for Rust.”
Sharing your vendor list can be controversial for a few reasons, but in general, we'd just qualify that the following list should not be taken as an exhaustive list of our security capabilities or act as an implicit endorsement of any vendor. The information shown is accurate as of April, 2024.
Cloud Security Posture Management (CSPM): Wiz – check our case study
Application Security Posture Management (ASPM): Arnica – check out our case study
Dynamic Application Security Testing (DAST): Burp Suite Professional & Enterprise – check out our Burp Suite Enterprise + GCP blog post
Web Application Firewall (WAF): Fastly Signal Sciences
(Private) Bug Bounty Program: HackerOne
Identity & Access Management: Okta
End-point Detection & Response (EDR): CrowdStrike Falcon
Mobile Device Management (MDM): Kandji
End-user Secret Management: Keeper
Security Awareness Platform: KnowBe4
Third-party Security Assessments: Aon
Customer Trust Center: SafeBase
Governance, Risk & Compliance Platform: Risk3sixty fullCircle GRC
Third-party Audit & Compliance: Risk3sixty & A-LIGN
General team workflows
Complexity makes security really, really hard. But so can well-intentioned processes that are inefficient by trying to be too clever. Automation everywhere only matters if your teams can keep up with the flow and if the signal versus noise is healthy enough not to get ignored constantly. As a team expands, contracts, and evolves, processes should keep being checked in on to see which small changes may make a big difference in reducing your team’s frustration in doing their best work. A bad process can soak up hours, achieving very little, and cannot be continued long.
At Fullstory, we make use of the Jira Service Desk to help track internal requests for specific types of canned support tasks for our customers inside the organization. The goal isn’t to make every single possible work task a service desk ticket but rather to simplify common patterns. We also tend to help prompt a person asking a question in some of our Slack channels to create a support ticket if they need something more robust than a quick yes/no answer back in a thread.
Speaking of, Slack is a place where we do some of our best work with the help of channels such as #ask-security-engineering or #ask-security-grc, which helps to track Q&A to a granular level. We also leverage some Slack integrations in the spirit of ChatOps but try not to just make entire channels giant alert backlogs that nobody looks at. There are tactical cases we do today, but in our overall process management, we’re converting those into Jira tasks for our Kanban boards to aggregate work into one place where the team can see high-signal work to focus.
Each security team has a bit of a different way of working, but for instance our Security Engineering team will leverage three 30-minute team standups a week (M, W, F) and also provides weekly “office hours” where employees can show up if they want to chat real-time without having to schedule something directly with everyone. Free time during those meeting slots usually just turns into ad-hoc discussions within the team or follow-up on prior topics.
Lastly, when we create project plans for a quarter, we focus on single-threaded owners. The goal here isn’t to have people work in silos; rather, it is to allow team members to have something they can “call their own” while also simplifying how we plan, track, status, and deliver work. This is integrated into the day-to-day operations of the team, and rapid adjustments are made, such as deprioritization of a workstream, when the beliefs of what the project needs to solve change.
Security engineering + Security GRC alignment
One anti-pattern I’ve experienced in other organizations is a bifurcation between the team responsible for core technical security (e.g., Security Engineering) and the team that is the guardrail for the business to ensure that program meets an industry-standardized bar (e.g., Security GRC). At Fullstory, both of these functions end up reporting to the same leader (read: me). Because I am able to directly lead the Security Engineering team and then partner with our Head of Security GRC, Anne Turner, there are multiple syncs a week that ensure the needs of the business are being addressed with excellent technical security that will also be able to exceed expectations on security audits and certifications that we share to customers.
Arguably, we spend very little time trying to “meet compliance requirements,” and instead we focus on building the right kind of security program for our business to securely thrive. Because of the tight partnership between Security Engineering and Security GRC, this enables us to share a comprehensive understanding of our program to the great auditors we work with.
In fact, as Fullstory has grown the number of third-party certifications and attestations we hold, the amount of “work” to achieve those has stayed the same – or even been reduced – because of the diligent focus, expertise, and coordination that our Security GRC team brings to the table.
Additionally, these two teams partner with our IT Information Security team, which is responsible for end-user and end-point security at Fullstory. We coordinate with our Legal team on privacy, security, and trust topics regularly, as well as other stakeholder groups to maximize efforts. This includes a bi-weekly senior security, privacy, legal, and compliance sync and a weekly security all-hands meeting so that all of our key partners across the business have visibility.
This year at Fullstory
I’ve had the great privilege of working on refining a robust security program here at Fullstory for the past two years with partners across the business who sincerely care about our responsibility to our customers and employees alike. Combined with exceptional vendor relationships that we’ve tuned over that time period, we’re more fully positioned than ever to continue acceleration on material security improvements that give our engineers high-quality signal, reduce our attack surface, simplify the process of “being secure,” and observe all of that effort via insights derived from third-party validation of our program, our maturity model, & KPIs.
I want to thank you for taking the time to learn more about our program and would encourage you to reach out to markstanislav@fullstory.com if you have any follow-up questions that I can answer. As a reminder, our customers have access to our Trust Center, where a wealth of information is available to observe the policies, certifications, attestations, and other details you may value.
