Separating Product Secrets from Ops Secrets
Secrets are everywhere in modern software systems.
API keys authenticate requests to third-party services. Database passwords allow infrastructure components to communicate. OAuth credentials enable integrations. Encryption keys protect sensitive data.
Despite looking similar, not all secrets serve the same purpose.
In Upblit, we intentionally separate customer-managed product secrets from internal operational secrets. Product credentials are managed through a custom Lua-based Secrets Manager exposed to the application layer, while infrastructure credentials remain under the control of HashiCorp Vault.
This architectural boundary reduces risk, simplifies operations, and prevents a surprisingly common design mistake: treating every secret as if it belongs in the same system.
Two Categories of Secrets
The first step is recognizing that secrets have fundamentally different ownership models.
Product Secrets
These are created and managed on behalf of customers.
Examples include:
- GitHub personal access tokens
- Slack webhook URLs
- Stripe API keys
- AWS credentials for customer integrations
- SMTP passwords
- External API tokens
Applications frequently need to read and use these values during normal business operations.
Operational Secrets
These exist solely to operate the platform itself.
Examples include:
- PostgreSQL credentials
- Redis passwords
- Kafka SASL credentials
- Internal TLS private keys
- Service account tokens
- Infrastructure encryption keys
These secrets are consumed by the platform rather than by customer workflows.
Why Keep Them Separate?
Although both categories are "secrets," they have different lifecycles, access patterns, and trust requirements.
A simplified architecture looks like this:
Customer
│
▼
Lua-Based Secrets Manager
│
▼
Customer Integration Secrets
--------------------------------------------
Internal Services & Operators
│
▼
HashiCorp Vault
│
▼
Infrastructure Credentials
The boundary is intentional: customer data and infrastructure credentials are managed independently.
Product Secrets Behave Like Application Data
A user connecting an external service expects the application to:
- store credentials securely,
- retrieve them during execution,
- rotate or revoke them,
- audit access,
- and associate them with tenants or projects.
Conceptually:
User
│
▼
Save API Key
│
▼
Secrets Manager
│
▼
Encrypted Storage
These secrets are part of the product experience and often require APIs, user interfaces, permissions, and tenancy controls.
Operational Secrets Behave Like Infrastructure
Infrastructure credentials are different.
Applications typically read them during startup or deployment:
Application
│
▼
HashiCorp Vault
│
▼
Database Password
│
▼
Connect to PostgreSQL
End users never interact with these values directly.
Their management belongs to platform engineering rather than application logic.
The Security Boundary
Maintaining separate systems creates an important security boundary.
Suppose an application bug exposes customer-facing APIs.
With separation:
- customer secrets remain within the product domain,
- infrastructure credentials remain protected by Vault,
- and compromise of one system does not automatically imply compromise of the other.
Likewise, infrastructure administrators responsible for operational credentials do not necessarily require unrestricted access to customer-managed integration secrets.
This separation follows the principle of least privilege.
Why a Custom Lua-Based Secrets Manager?
Product secrets often require application-specific behavior that general-purpose infrastructure secret stores are not optimized for.
Examples include:
- tenant-aware access control,
- project-level permissions,
- metadata attached to secrets,
- lifecycle management,
- customer-facing APIs,
- integration-specific validation,
- usage tracking,
- and product-specific business rules.
A lightweight Lua-based implementation allows these workflows to be embedded directly into the application's execution model while keeping policy enforcement close to the business logic.
For customer-facing integrations, this flexibility can be more valuable than exposing infrastructure-oriented tooling directly to end users.
Why Use HashiCorp Vault for Ops Secrets?
Operational credentials benefit from mature infrastructure tooling.
Vault provides capabilities such as:
- centralized secret storage,
- policy-based access control,
- audit logging,
- encryption at rest,
- dynamic credentials,
- lease management,
- and automated rotation.
These features make it well suited for managing platform infrastructure and service-to-service authentication.
By reserving Vault for operational concerns, platform teams can take advantage of these capabilities without exposing unnecessary complexity to product workflows.
The Anti-Pattern: One Store for Everything
A tempting design is to place all secrets into a single system.
Everything
│
▼
One Giant Secret Store
│
┌──────────┴──────────┐
▼ ▼
Customer APIs Infrastructure
Credentials
Initially this appears simpler.
In practice, it creates several problems.
Mixed Access Policies
Customer-facing applications need controlled access to customer-owned credentials.
Infrastructure components need privileged access to operational secrets.
Combining both categories often leads to broad permissions that violate least-privilege principles.
An application intended only to retrieve integration tokens may inadvertently gain access to database credentials or internal service keys.
Different Rotation Requirements
Infrastructure secrets often rotate automatically according to operational policies.
Customer-managed credentials may rotate only when users explicitly update them or when external providers issue replacements.
Treating both categories identically complicates automation and increases operational risk.
Different Audit Requirements
Product teams frequently need audit trails such as:
- who created a secret,
- which tenant owns it,
- when it was last updated,
- and when it was accessed by the application.
Infrastructure teams, meanwhile, focus on service authentication, deployment events, and operational access.
Separate systems enable audit logs tailored to each domain.
Tenant Isolation
Because product secrets belong to customers, they naturally inherit tenancy boundaries.
A conceptual schema might resemble:
CREATE TABLE secrets (
id UUID PRIMARY KEY,
tenant_id TEXT NOT NULL,
encrypted_value BYTEA NOT NULL
);
Every lookup is scoped by tenant_id, ensuring that applications retrieve only credentials belonging to the active organization.
Infrastructure credentials typically do not follow this ownership model.
Defense in Depth
Separation also improves resilience during security incidents.
Imagine a vulnerability affecting product APIs.
If customer secrets and infrastructure secrets share the same backend, attackers may gain a pathway to internal credentials.
With distinct systems and access controls, compromise in one domain is less likely to escalate into complete platform compromise.
Defense in depth is achieved not through any single mechanism but through carefully designed boundaries.
Operational Simplicity
Keeping operational credentials in Vault allows platform teams to continue using established tooling for deployment, rotation, and policy management.
Meanwhile, product engineers can evolve customer-facing secret workflows independently without coupling user experience to infrastructure operations.
Each system remains optimized for its own purpose.
Final Thoughts
Not all secrets are created equal. Customer integration tokens and infrastructure credentials may both require encryption and careful handling, but they differ fundamentally in ownership, lifecycle, and access patterns.
By implementing a dedicated Lua-based Secrets Manager for client-facing credentials while reserving HashiCorp Vault for operational secrets, Upblit establishes a clear security boundary that supports least privilege, tenant isolation, and defense in depth. Avoiding the temptation to collapse both categories into a single store keeps responsibilities well defined and reduces the blast radius of failures or compromises.
For modern SaaS platforms, treating product secrets and operational secrets as separate architectural concerns is often the cleaner—and safer—design choice.