At one point during development, my team and I realized that our web application, running in Azure Kubernetes Service (AKS), is not rotating secrets correctly. Specifically, when a connection string was changed in Azure Key Vault, the .NET app didn't pick up the change, despite having reloadOnChange enabled and working fine during local development on Windows.

Setup:

• ASP.NET Core 6 API
• running as Docker container in AKS
• using Secrets Store CSI driver to get secrets from Azure Key Vault as files

Turns out that file changes in Docker (and Linux, according to my experiments) are not always properly reported to the file system watcher and polling needs to be enabled.

tl;dr

Use the DOTNET_USE_POLLING_FILE_WATCHER environment variable and set it to 1 or true to make the application poll for config file changes every 4 seconds.

Getting secrets from Key Vault - the CSI driver

The application is using Secrets Store CSI (Container Storage Interface) driver for Kubernetes. There's an official guide on how to set it up.

The driver gets secrets from the configured Azure Key Vault instance (or multiple) and mounts them to Kubernetes pods as a volume. These secrets are then available to the application as files.

Using files as configuration in .NET Core

Using individual files as configuration can be easily set up in .NET with the Key-per-file configuration provider. File names will become **keys ** and file contents will become values.

To generate nested configuration (ApplicationInsights:InstrumentationKey), use __ (two underscores) like this: ApplicationInsights__InstrumentationKey.

Program.cs

public static IHostBuilder CreateHostBuilder(string[] args) =>
	Host.CreateDefaultBuilder(args)
    .ConfigureAppConfiguration((_, configuration) =>
    {
    	configuration.AddKeyPerFile(directoryPath: "/mnt/secrets-store", optional: true, reloadOnChange: true);
    })
    // ... other stuff

Note the reloadOnChange: true parameter. This should ensure that configuration gets reloaded every time the underlying files change. Running the app in Kubernetes and also locally in WSL Ubuntu revealed that it's not always the case. Which brings us to the solution.

Automatically refreshing configuration

After some investigation I found out that:

Some file systems, such as Docker containers and network shares, may not reliably send change notifications.

And the solution is to set the DOTNET_USE_POLLING_FILE_WATCHER environmental variable to either 1 or true, which makes the application poll the filesystem for changes every 4 seconds. Then the configuration gets properly updated once changed in Key Vault and propagated all the way to the cluster.