Practical advice for securing secrets across the SDLC

When planning for secrets management, there are many considerations. Each area of the logical architecture of the Software Development Lifecycle (SDLC) has unique areas of concern, tooling, and pathways to properly use secrets. In addition to the considerations covered here, you should also consider the specific software and data-flow pathways in your environment for additional mitigation efforts. Avoid logging secrets at any point through your environment. If a secret is logged during troubleshooting, bug-fix, etc., the log should be immediately scrubbed and that secret should be rotated!

Develop phase

Development secrets should be treated with the same respect and constraints as production secrets.

  • Secrets used for development should be stored in a secure store separate from production stores. If using Key Vault for production, store development secrets in a different vault.
  • Include an expiration date for development secrets. Create a shorter lifespan for development secrets to ensure they do not outlive the development process and to test rotation strategies outside of production.
  • Maintain a dedicated inventory near the development environment. This ensures engineers have appropriate content and context when considering secrets. Reflect this inventory in or derive it from a broader organizational inventory.

Enforce secret-detection as part of the pull request process for every change. This should be a pre-commit hook to ensure no secrets can be merged into the source control history. Preventing secrets from being pushed to a branch under source control helps avoid a breach of source control from becoming a source of historical data on secrets that may still be active.

Encourage consideration for how an application under development should access any required secrets and, where possible, use appropriate SDKs to enable the application to access secrets securely. This isn't always feasible, but it provides the highest level of control over access pathways. Avoid storing secrets as environment variables to the greatest extent possible, considering the language and tooling used.

Build phase

The build system is generally responsible for several operations around building your software and preparing it for packaging. Some primary responsibilities of the build system revolve around testing the software. These may include code and image scanning as well as quality tests. Your build system should also include scanning the source code for secrets as part of final preparation. If you have a well-configured code repository, this likely seems redundant, but it is an additional layer of protection to ensure secrets are not leaked into the build system.

Ensure your build system uses appropriately configured secret stores to access any credentials required for the build process. When possible, follow the best practice of using managed identities (as discussed), but using secrets may be necessary. If secrets are part of the build steps, follow these rules:

  • Logging across the build system is vital for continuous build processes, but it can also be a potential area for secret leaks. Ensure that secrets used in the build system are not logged in plain text at any point.
  • Ensure that any rotation is accounted for in the build system. As you move towards continuous builds, it is paramount that refreshed secrets are available to the build system. This is generally less complex than keeping code-required secrets up-to-date, but may be more challenging in complex build pipelines.
  • Ensure that access to secret stores is time-boxed to the build process's needs. The secret should be available to the build system with the tightest possible limitations that enable the build processes to run. If the secret must be accessed multiple times during the build process, this is preferable to having it accessible for the entire duration of all build steps.
  • Use one of the recipes suggested in secrets-store to configure access to secrets properly.
  • When using multi-stage build pipelines, consider which stage needs access and limit access as much as possible to operations within that pipeline stage.
  • Use ancillary scans for secrets to ensure no secrets are hard-coded into the source or build environments. The secret-detection guidance provides useful resources for detecting secrets.

Deploy phase

Much of the guidance for the deploy layer of the SDLC also applies to deploying software, with additional considerations for new tools. When planning deployment strategy, you may deal with different types of secrets:

  • Avoid logging secrets during deployment. Deployment often includes an augmentation or completely separate set of tools. Ensure that the passwords used for these tools are provisioned with only the permissions required for deployment.
  • Be mindful of logging and network traversal required to access secrets during deployment.
  • If possible, use different secrets for deployment rather than reusing build secrets.
  • Never use development secrets to deploy software to staging and production environments.
  • Configure deployment pipelines using the suggestions in secrets-store.
  • Account for deployment in secrets-rotation strategies.

Operate phase

Software running in production deserves prudent consideration. Even after testing for secrets in other layers of the SDLC, it still requires diligence to ensure that operational team members have high confidence in a secret-free environment and can update secrets as needed.

  • Operations must be aware of automated secret rotation strategies and have access to guidance on break-glass rotation scenarios.
  • Secrets across the software supply chain should include operational insights into secret fidelity.
  • Production systems involve extensive logging operations. Apply secret detection across logging as a final stop-gap to ensure secrets are not logged during normal operations and to protect against malicious logging triggered by vulnerabilities.
  • Any software requiring manual updates to secrets for rotation should be known, and the update process documented. This should be included in the secrets inventory for ease of access and should outline steps for ensuring the secret is updated in any secret store and within reliant applications.

One final reminder is that using managed identities is always preferred over secrets. This guidance is meant to be implemented when managed identity usage is not possible.

Additional articles in this solution