BlackHat USA DevSecOps

This expanded 50-slide presentation by Anant Shrivastava at Black Hat USA 2019 provides a comprehensive guide to DevSecOps — integrating security into DevOps pipelines through both automation and cultural transformation. Building on the core “What, Why, and How” framework, this version adds significant depth with detailed breakdowns of each pipeline stage, language-specific tooling matrices, cloud-native DevSecOps implementations across AWS/Azure/GCP, sample lab demonstrations for Ruby/PHP/Python/Node.js, real-world case studies from organizations like Fannie Mae and ABN Amro, and a critical discussion on securing the security toolchain itself (“Who Watches the Watcher”). The presentation positions DevSecOps as a multi-pillar strategy encompassing People, Process, and Technology.

Key Topics Covered

• DevSecOps Defined: An effort to achieve “Secure by Default” by integrating security via tools, creating a Security as Code culture, and promoting cross-skilling. DevSecOps makes it easier to manage the rapid pace of development and enables smoother scaling of secure deployments.

• The Shift Left Business Case: Finding a single SQL injection earlier in the pipeline (at the developer/source code stage) requires fewer man-days of effort, eliminates the need for new deployments, and can be caught through automated source code review — dramatically reducing cost compared to discovering it in production penetration testing.

• Complete DevSecOps Pipeline Architecture: The presentation details security integration at every stage:

  • Pre-Commit Hooks: Regex-based filtering of sensitive data (access keys, tokens, SSH keys) on developer workstations. A defense-in-depth measure since developers can circumvent it.
  • IDE Security Plugins: Quick actionable pointers to stop basic security blunders during coding.
  • Secrets Management: Tokenizing credentials stored in config files to prevent leakage and abuse.
  • Software Composition Analysis (SCA): Checking third-party libraries and modules (PIP, NPM, Gems, go get, CPAN, PHP Packager) for known vulnerabilities, since the biggest portion of modern software is third-party code.
  • Static Analysis Security Testing (SAST): White-box automated testing for low-hanging fruits like SQL injection and XSS. Requires manual oversight for managing false positives.
  • Dynamic Analysis Security Testing (DAST): Black/grey-box testing that catches deployment-specific issues SAST may miss. Results from DAST and SAST can be compared to weed out false positives.
  • Security in Infrastructure as Code: Auditing Docker/K8s infrastructure, ensuring base images are minimal and assessed for inherited vulnerabilities.
  • Compliance as Code: Converting industry standards (PCI DSS, HIPAA, SOX) or organization-specific compliance rules into version-controlled test cases.
  • Vulnerability Management: Centralizing and normalizing reports from multiple tools into a dashboard integrated with bug tracking systems.
  • Alerting and Monitoring: Understanding security control effectiveness — when attacks occurred, whether they were blocked, access level achieved, and data exfiltration.
  • Asset Monitoring: Continuous inventory of all assets including cloud resources, not a static list.

• Tools of the Trade: Organized by category with preference for open-source:

  • Pre-Commit: Git Hound, truffleHog
  • IDE Plugins: CAT.net
  • Threat Modeling: ThreatSpec, Microsoft Threat Modeling Tool
  • Secret Management: Keywhiz
  • SCA: Retire.js, and language-specific tools
  • SAST/DAST: Multiple open-source options
  • Infrastructure: Docker Bench for Security
  • WAF, Vulnerability Management: Various tools listed

• Pipeline Tool Criteria: API/CLI access, 15-minute maximum execution time, containerizable/scriptable, minimal licensing limitations, machine-readable output (JSON/XML), and configurable for false positive/negative management.

• Pipeline Optimization Strategies: Tailor the pipeline based on the type of change — skip SCA for CSS-only changes, skip SAST for dependency-file-only changes (pom.xml/gradle), fast-track infrastructure scans when IaC has zero changes. Run the full unoptimized pipeline periodically to catch anything missed.

• Language-Specific Tool Matrix: Detailed breakdown across Java, PHP, Python, Ruby/Rails, .NET, and Node.js covering both SCA and SAST tools:

  • Java: FindBugs, FindSecBugs (SAST); OWASP Dependency-Check (SCA)
  • PHP: graudit (SAST); various SCA
  • Python: Bandit, graudit (SAST); Safety (SCA)
  • Ruby/Rails: Brakeman, graudit (SAST); Bundler-Audit (SCA)
  • .NET: DotNet Security Guard (SAST); DotNET Retire, SafeNuGet (SCA)
  • Node.js: NodeJsScan, npm-check (SAST/SCA); Retire.js, ClearlyDefined

• Cloud-Native DevSecOps: Comprehensive comparison table mapping conventional tools to cloud-native equivalents across AWS, Azure, and GCP for source code management, IaC, CI/CD, artifact repositories, staging/production servers, monitoring, firewalls, DLP, threat detection, vulnerability scanning, and secrets management.

• Cloud Security Considerations: The threat landscape changes — focus on IAM, asset inventory, billing, security groups, permissions, rogue/shadow admins, and forgotten resources. Different providers approach security differently, but SAST, DAST, SCA, and vulnerability management tools still need to be sourced regardless of provider.

• Security Enablers — People, Process, Technology:

  • People: Build relationships between teams, identify and nurture security-conscious individuals, empower Dev/Ops to deliver securely, focus on solutions instead of blaming.
  • Process: Involve security from the design/ideation phase, fix by priority rather than attempting to fix everything, make security controls programmable and automated, ensure smooth feedback governance.
  • Technology: Templatize scripts/tools per language/platform, adapt security to the DevOps flow rather than expecting others to adopt security, stay pragmatic about testing and adopting new tools.

• Security Champions Program: A single person per team who bridges Dev, Sec, and Ops. Incentivize collaboration through internal bug bounties, sponsored social interactions, and cross-skilling training.

• Real-World Case Studies: Positive examples from Fannie Mae and ABN Amro demonstrating successful DevSecOps adoption, plus negative case studies showing cloud asset misconfiguration and the consequences of inadequate monitoring.

• “Who Watches the Watcher”: A critical point about securing the security toolchain itself — if an attacker controls security tools or the build chain, they gain limitless power (DevSecOops). The same security practices must be applied to security tools: secure configuration, patching, and basic security hygiene.

Actionable Takeaways

1. Map your full DevOps pipeline and inject security at every stage from pre-commit through production monitoring — use the detailed pipeline architecture as a reference blueprint.
2. Optimize your pipeline for speed by tailoring security scans to the type of change (skip SCA for CSS changes, skip SAST for dependency-only changes), but ensure a full unoptimized scan runs periodically.
3. Select language-specific SAST and SCA tools for each technology in your stack rather than relying solely on multi-language tools like SonarQube.
4. When migrating to cloud, map your existing security tooling to cloud-native equivalents using the AWS/Azure/GCP comparison matrix, and identify gaps that still require third-party tools.
5. Implement the three-pillar Security Enablers model (People, Process, Technology) — involve security from the design phase, build a Security Champions program, and templatize security tooling per language/platform.
6. Secure your security toolchain with the same rigor you apply to production systems — an attacker who controls your CI/CD pipeline or security tools has unlimited power.
7. Validate DevSecOps effectiveness through periodic penetration testing and continuous bug bounty programs, and treat feedback as actionable items rather than filing risk acceptance documentation as a first resort.
8. Study the Fannie Mae and ABN Amro case studies for practical lessons on organizational DevSecOps adoption at scale.