Environment Variables and Secrets

In modern application development, security and efficiency are paramount. Environment variables and secrets are crucial components in ensuring that applications can be securely configured and deployed without exposing sensitive information. These mechanisms are widely used in DevOps, CI/CD pipelines, and cloud-based application management.

This article explores the role of environment variables and secrets, their importance in application security, and how to implement them effectively in DevOps pipelines, cloud platforms, and local development environments.

What Are Environment Variables?

Environment variables are key-value pairs that are used by the operating system, software applications, or services to configure various settings or provide runtime information. They are typically used to manage configuration settings that vary across environments, such as development, testing, and production.

Common Use Cases for Environment Variables:

Configuration settings: Environment variables store settings like database connection strings, API keys, or logging levels.
Deployment configurations: Variables can specify deployment environments (e.g., development, staging, production) and the respective configurations.
Cloud infrastructure variables: Cloud services often use environment variables to configure settings related to cloud providers, like AWS, Azure, or Google Cloud Platform.

Environment variables allow for easy changes in configuration without needing to modify source code, reducing the risk of security flaws caused by hardcoded credentials or settings.

What Are Secrets?

Secrets refer to sensitive data (e.g., passwords, API keys, encryption keys, tokens) that must be kept confidential to prevent unauthorized access. Unlike environment variables, secrets are highly sensitive and require additional security measures for storage, retrieval, and usage.

Secrets often include:

Database passwords
OAuth tokens
API keys
Encryption certificates
Private keys

Managing secrets effectively is critical to avoid exposing them through logs, configuration files, or incorrect permissions.

Why Use Environment Variables and Secrets?

1. Separation of Configuration and Code

By using environment variables and secrets, you can separate configuration from application code. This practice follows the 12-factor app methodology, which recommends that an app’s configuration be stored in environment variables, making it easier to deploy across multiple environments without code changes.

2. Security and Privacy

Storing sensitive information like passwords or API keys directly in code repositories exposes these credentials to risks, including unauthorized access or accidental leaks. Environment variables and secrets provide a secure way to store and use sensitive information without hardcoding it into the codebase.

3. Easier Configuration Management

Managing configurations becomes easier because you can adjust variables without touching the source code. Whether it’s changing an API URL, updating an authentication token, or modifying the behavior of a service, environment variables allow for centralized and consistent configuration across environments.

4. Flexible Deployment to Multiple Environments

You can use different sets of environment variables for different environments (development, staging, production). For instance, your production environment may use different database credentials or API keys than your development or testing environments.

Environment Variables vs. Secrets

While environment variables and secrets can both store configuration data, there are key differences between them:

Environment Variables	Secrets
Typically used for non-sensitive data	Used exclusively for sensitive data
Stored in system environment or configuration files	Stored in secret management systems (e.g., AWS Secrets Manager, Azure Key Vault)
Can be accessed by any process running in the environment	Access to secrets is strictly controlled and encrypted
Less secure but easy to manage for non-sensitive settings	High security and encryption are required

Environment variables are more appropriate for non-sensitive configuration (e.g., app version, service endpoints), while secrets should be used for sensitive information.

How to Use Environment Variables in DevOps

1. Defining Environment Variables in CI/CD Pipelines

In CI/CD pipelines, environment variables are often used to store configuration data that changes based on the environment or pipeline stages. Most CI/CD tools, such as GitHub Actions, Azure DevOps, or Jenkins, allow you to define environment variables within the pipeline.

Example in GitHub Actions:

name: CI/CD Pipeline

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      DB_PASSWORD: ${{ secrets.DB_PASSWORD }}
      API_KEY: ${{ secrets.API_KEY }}
      
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Run tests
        run: |
          echo "Running tests..."
          # Use DB_PASSWORD and API_KEY from environment variables

In this example, DB_PASSWORD and API_KEY are securely stored in GitHub secrets and accessed as environment variables during the build.

2. Environment Variables in Cloud Platforms

Cloud platforms like AWS, Azure, and Google Cloud provide native ways to configure environment variables and secrets.

For example, in Azure, you can configure environment variables using the Azure App Service settings:

Go to your App Service in the Azure portal.
Under Configuration, you can add environment variables (key-value pairs).
These variables are automatically available to your application during runtime.

3. Local Development with Environment Variables

For local development, you can use a .env file to store environment variables. Tools like dotenv (for Node.js), python-dotenv (for Python), or dotenv for Ruby help load these environment variables into the environment.

DB_PASSWORD=mysecretpassword
API_KEY=myapikey

In your application code, you can access these variables using:

import os
db_password = os.getenv('DB_PASSWORD')

This approach keeps sensitive data out of source control and makes local development configuration flexible.

How to Handle Secrets Securely

1. Using Secret Management Systems

Instead of hardcoding secrets or storing them as environment variables, use secret management systems that are specifically designed to store sensitive information securely.

AWS Secrets Manager

AWS Secrets Manager enables you to store, manage, and retrieve secrets (such as database credentials, API keys, etc.).
Secrets are encrypted at rest and can be programmatically accessed using the AWS SDK.

import boto3
from botocore.exceptions import NoCredentialsError

def get_secret():
    client = boto3.client('secretsmanager')
    secret_name = "my_secret_key"
    
    try:
        get_secret_value_response = client.get_secret_value(SecretId=secret_name)
        return get_secret_value_response['SecretString']
    except NoCredentialsError:
        print("Credentials not available")
        return None

Azure Key Vault

Azure Key Vault helps you safeguard and control access to secrets, keys, and certificates.
Secrets can be accessed via REST API, SDKs, or integrated directly into applications running in Azure.

# Store secret in Azure Key Vault
az keyvault secret set --vault-name <your-keyvault-name> --name <secret-name> --value <secret-value>

Google Cloud Secret Manager

Google Cloud Secret Manager is a fully managed service that enables you to store and manage sensitive data securely.
It provides IAM-based access control and encryption at rest.

# Store secret in Google Cloud Secret Manager
gcloud secrets create my-secret --data-file=my-secret.txt

2. Access Control and Auditing

Always enforce least privilege access for managing secrets. Only the necessary applications or users should have access to specific secrets.

Most secret management tools also offer auditing capabilities, allowing you to track who accessed secrets and when.

Best Practices for Using Environment Variables and Secrets

1. Use a Secret Management System

For sensitive data, avoid using plain environment variables. Instead, leverage secret management systems like AWS Secrets Manager, Azure Key Vault, or Google Secret Manager to securely store and access secrets.

2. Secure Access to Secrets

Ensure that only authorized applications or services can access secrets. Use Identity and Access Management (IAM) policies to control permissions.

3. Never Hardcode Secrets

Never hardcode secrets, passwords, or API keys directly into your source code. Use environment variables or secret management systems to keep secrets safe.

4. Use Different Configurations for Different Environments

Use separate configurations for development, staging, and production environments. For instance, use different database credentials for your development and production environments.

5. Rotate Secrets Regularly

To minimize the impact of potential leaks, rotate secrets periodically. Many secret management systems offer automatic secrets rotation.

6. Encrypt Sensitive Data

Always encrypt sensitive information—both at rest and in transit. This applies to secrets stored in secret management systems as well as data transferred over networks.

7. Use Environment Variables for Non-sensitive Configurations

Environment variables are best suited for non-sensitive configuration data like app version, logging level, or service endpoints.