Zstandard compression in .NET 11

.NET has had DeflateStream, GZipStream, ZLibStream, and BrotliStream for a while now. In .NET 11, a new one joins the party: ZstandardStream. And now we get to say "Zstd" in .NET.

What is Zstandard?

Zstandard (or Zstd) is a compression algorithm published by Facebook in 2015. To quote Wikipedia:

Zstandard was designed to give a compression ratio comparable to that of the DEFLATE algorithm (developed in 1991 and used in the original ZIP and gzip programs), but faster, especially for decompression. It is tunable with compression levels ranging from negative 7 (fastest)[6] to 22 (slowest in compression speed, but best compression ratio).

^{Source: Wikipedia}

"Users" of Zstd: the Linux kernel (btrfs, SquashFS), FreeBSD coredumps, AWS Redshift, Fedora and ArchLinux package managers, and Nintendo Switch game archives. And maybe soon as part of the .NET runtime/tooling.

The new API

The design follows the existing Brotli* pattern, so if you've used BrotliStream or BrotliEncoder, this will feel familiar.

There are four main new types in System.IO.Compression:

• ZstandardStream — the stream wrapper
• ZstandardEncoder / ZstandardDecoder — struct-based, Span-friendly
• ZstandardDictionary — for compressing many small files efficiently
• ZstandardCompressionOptions — quality, window size, checksum, and more

The simplest usage looks just like GZip:

// Compress
using var output = new MemoryStream();
using (var zstd = new ZstandardStream(output, CompressionMode.Compress))
{
    await inputStream.CopyToAsync(zstd);
}

// Decompress
using var zstd = new ZstandardStream(compressedStream, CompressionMode.Decompress);
await zstd.CopyToAsync(outputStream);

If you don't need streaming, there are also Span-based one-shot APIs:

byte[] source = File.ReadAllBytes("data.bin");

// Compress
int maxLength = ZstandardEncoder.GetMaxCompressedLength(source.Length);
var compressed = new byte[maxLength];
ZstandardEncoder.TryCompress(source, compressed, out int bytesWritten);

// Decompress
int decompressedSize = ZstandardDecoder.GetMaxDecompressedLength(compressed.AsSpan(0, bytesWritten));
var decompressed = new byte[decompressedSize];
ZstandardDecoder.TryDecompress(compressed.AsSpan(0, bytesWritten), decompressed, out int bytesRead);

Quality levels

Zstd has a quality range from ZSTD_minCLevel() (which can be negative, for ultra-fast modes) up to 22, with 3 as the default. CompressionLevel.Fastest maps to the minimum, CompressionLevel.SmallestSize to 22.

var options = new ZstandardCompressionOptions
{
    Quality = 6 // 1–22, default is 3
};

using var zstd = new ZstandardStream(output, options);

Plugging it into ASP.NET Core

Response compression is one line of real code:

// Program.cs
builder.Services.AddResponseCompression(x =>
{
    x.EnableForHttps = true;
    x.Providers.Add<ZstandardCompressionProvider>();
});

The ZstandardCompressionProvider connects the encoding: zstd content negotiation header automatically. Request decompression is equally straightforward via IDecompressionProvider.

Dictionary compression

This is the part that makes Zstd a bit unique: When you're compressing a lot of small, similar payloads (JSON API responses, serialized events, config blobs), a shared dictionary can improve the compression ratio.

// Train a dictionary from samples
byte[] allSamples = ...; // all your sample payloads concatenated
int[] lengths = ...;     // length of each sample

using ZstandardDictionary dict = ZstandardDictionary.TrainFromSamples(allSamples, lengths, maxDictionarySize: 100_000);

// Compress with dictionary — both sides must use the same one
using var encoder = new ZstandardEncoder(dict);
encoder.Compress(payload, compressed, out int consumed, out int written, isFinalBlock: true);

Store the dictionary somewhere both producer and consumer can reach it. The recommended size is up to 100 kB, trained on around 100x that size of representative data.

Ressources

The original issue tracking this: https://github.com/dotnet/runtime/issues/59591