How to store sensitive configuration data

Introduction

In this post I would like to discuss places where we can store configuration of our application and warn against keeping sensitive configuration data in the code repository. Additionally, I will show example of configuration based on User Secrets and Environment Variables in .NET Core.

Types of configuration data

There are plenty of configuration data types. I will try to name a few of them:

• store configurations (connection strings, providers, timeouts)
• external API configurations (urls, endpoints)
• caching
• hosting configuration (url, port, schema)
• file system paths (storing files, loading other resources)
• framework configuration
• libraries configuration
• business logic parameters
• …

Apart from the configuration type, we can add additional categorization:

Sensitive vs Insensitive

Sensitive data that must be protected from unauthorized access to safeguard the security of an our application or user’s data. Examples of sensitive data are private key for external API or username and password in database connection string. Example of insensitive configuration data could be database timeout length setting.

Local vs Global

Local configuration is dependent on environment. For example connection string to database is different for local development, test or production environments because they are using their own storage.

Global configuration is something independent from environment. For example it could be list of predefined file extensions which user can upload.

Required vs Optional

Configuration data for specific component could be required or optional. For database connection string server and database name are required parameters but for example max connection per pool is not.

Handling type

There are 3 types of handling configuration data:

1. Fetched once, used many time
2. Fetched every now and then, used many times
3. Fetched each time it is used

Configuration sources

How many configuration types exist there are so many sources of it. Diagram below presents possible variations:

Application configuration file

This is the most common option. Configuration data is included in application and can be changed by editing/replacing configuration file.

External configuration file

It is also configuration file but situated outside the application - on the same computer or somewhere on the network.

Environment variable

Configuration data is stored in operating system environment variables. This option has become popular by the trend on containerization.

Store

Sometimes there is a need to store configuration data in the database. The reason for this is externalization of configuration (when more than one process needs this data) or we need easy way to edit that configuration.

System / Service

When configuration is more complicated and dynamic it is good approach to get configuration from external system / service. An example of such a service is the Oculus which is service discovery mechanism in microservices environment.

Storing sensitive configuration data

If you would have to remember one thing from this entry, that would be it: Don’t store sensitive data in your code repositories. Never. Period.

Even if you have private repositories on Github or Bitbucket . Even if you have private repositories hosted on your internal servers and whole infrastructure is hidden behind the NAT. Even if you created only local repository which is not published to remote server.

Don’t do it.

How many times you thought in this way:

Hmm…Ok, I have to add credentials configuration for database/api/whatever now. For now I will simply add them to configuration file and when I get ready to deploy new feature to test/prod I will change it. Or I will do it before committing my changes. Yes, I will only check if this connection works..

…and the data suddenly appears in the repository. Because we forgot. Because we took care of something else. Because…

Too much “becauses”. The right solutions are simple and I will show two of them: storing configuration in external configuration file (User Secrets) and Environment variables.

User Secrets

User Secrets is nothing else than JSON configuration file stored in a system-protected user profile folder on the local machine in path %APPDATA%\Microsoft\UserSecrets<user_secrets_id>\secrets.json. User secrets id is GUID generated by Secret Manager tool.

Process of adding User Secrets is pretty simple. Firstly, we need right click on the project and select Manage User Secrets option.

After this step Visual Studio will generate secrets.json file and add entry to .csproj file:

<PropertyGroup>
  <TargetFramework>netcoreapp2.1</TargetFramework>
  <UserSecretsId>a54b8aa0-b40a-426d-9ba9-28d74b72a0e9</UserSecretsId>
</PropertyGroup>

Then we can add needed configuration to generated secrets.json file, for example:

{
  "DbConnectionString":  "secretConnString" 
}

Last thing to do is reference Microsoft.Extensions.Configuration.UserSecrets provider configuration package and register it in application startup:

// Register UserSecrets
public Startup()
{
    var builder = new ConfigurationBuilder()
        .AddUserSecrets<Startup>();

    Configuration = builder.Build();
}

Environment variables

User Secrets are good in development scenarios. For other scenarios when we need deploy our application to other environments (test, stage, prod) we should use Environment variables instead. First step is to define variable for hosting operating system / container. For Windows it looks like:

setx SampleKey SampleValue /M

Second and last step is to configure provider referenced from Microsoft.Extensions.Configuration.EnvironmentVariables:

// Environment variables provider configuration
public Startup()
{
    var builder = new ConfigurationBuilder()
        .AddEnvironmentVariables();

    Configuration = builder.Build();
}

Extension method AddEnvironmentVariables() has overload with prefix parameter. It can be useful for variables filtering or when we have multiple environments on single host. This is example configuration:

// Get configuration for multiple environments
string environment = GetEnvironment();
_configuration = new ConfigurationBuilder()
    .AddEnvironmentVariables($"ASPNETCORE_{environment}_")
    .Build();

How to implement GetEnvironment() method depends on your preferences. It can be determined based on project configuration or other factor like external file.

Summary

In this post I described the diversity of configuration data as well as their possible sources. In addition, I showed how to keep sensitive configuration data securely on the .NET Core runtime environment using User Secrets and Environment variables. The security of applications and our data is often overlooked in the initial phase of the project, which is a big mistake and can be tragic in consequences. It’s better to think about it from the very beginning.

Comments