support http zstd compression.

[mbtolou]

hi.

can you support zstd compression in http call ?

https://github.com/ClickHouse/ClickHouse/blob/512786346c5d615e512b8cea1c237699d1fe7153/tests/queries/0_stateless/00302_http_compression.sh

This algorithm (zstd) is 3 times faster than gzip.

https://github.com/ClickHouse/ClickHouse/blob/512786346c5d615e512b8cea1c237699d1fe7153/tests/queries/0_stateless/00302_http_compression.sh#L29

[DarkWanderer]

It is possible, but comes with following downsides

• I would need to create a custom wrapper for HttpResponseStream to read zstd, which I'd like to avoid
• ZstdNet seems to be a native library wrapper

Is there a way to have zstd compression supported in HttpClient natively? Are there pure managed implementation of zstd compression?

[mbtolou]

Hello
Thanks for your attention .

I recommend the ZStd library as a compressor.
Because in this library, compression is implemented using the unsafe class.
To make it clear, I will send you the example of the advanced codes of this library in the form of an image so that I can convey my meaning better.

In the following, I will compare the efficiency of this library with native libraries.

ZstdSharp is a port of zstd compression library to С#
Based on Zstandard v1.5.2
Supports .NET Core 3.1, .NET 5, .NET 6, .NET Standard 2.0+, .NET Framework 4.6.1+

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19044.1566 (21H2)
12th Gen Intel Core i7-12700, 1 CPU, 20 logical and 12 physical cores
.NET SDK=6.0.200
[Host] : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT
DefaultJob : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT

Compression level 1
|           Method |      Mean |     Error |    StdDev | Ratio | InstructionRetired/Op |
|----------------- |----------:|----------:|----------:|------:|----------------------:|
|   CompressNative | 25.565 ms | 0.2983 ms | 0.2491 ms |  1.00 |           397,370,833 |
|    CompressSharp | 35.616 ms | 0.0593 ms | 0.0495 ms |  1.39 |           523,222,222 |
|                  |           |           |           |       |                       |
| DecompressNative |  6.821 ms | 0.0097 ms | 0.0086 ms |  1.00 |           154,583,333 |
|  DecompressSharp |  8.178 ms | 0.0198 ms | 0.0176 ms |  1.20 |           184,807,292 |


Compression level 5
|           Method |      Mean |     Error |    StdDev | Ratio | RatioSD | InstructionRetired/Op |
|----------------- |----------:|----------:|----------:|------:|--------:|----------------------:|
|   CompressNative | 79.408 ms | 0.1196 ms | 0.0933 ms |  1.00 |    0.00 |         1,096,971,429 |
|    CompressSharp | 96.018 ms | 0.3943 ms | 0.3292 ms |  1.21 |    0.00 |         1,528,277,778 |
|                  |           |           |           |       |         |                       |
| DecompressNative |  8.156 ms | 0.1620 ms | 0.1663 ms |  1.00 |    0.00 |           205,591,912 |
|  DecompressSharp | 10.637 ms | 0.0937 ms | 0.0831 ms |  1.31 |    0.03 |           250,476,042 |

[mbtolou]

In the next step, I developed a library that can replace the current functions by using Extension functions.

I also uploaded the complete code of the library in the Repository below.
Thank you again for your attention.

https://github.com/mbtolou/ZStdHttpClient

using System.Net.Http.Headers;
namespace ZstdHttpClient;

public static class HttpExtensions
{
    private static readonly StringWithQualityHeaderValue ZStdHeader = new StringWithQualityHeaderValue("zstd", 1);

    public static async Task<HttpResponseMessage> GetAsync(
        this HttpClient client,
        string url,
        CompressionType cp
        )
    {
        if (cp == CompressionType.Zstd)
            client.DefaultRequestHeaders.AcceptEncoding.Add(ZStdHeader);

        var response = await client.GetAsync(url);

        if (cp == CompressionType.Zstd)
        {
            if (response.Content.Headers.ContentEncoding.LastOrDefault() == "zstd")

            {
                response.Content = new ZStdHttpContent(response.Content);
            }
        }

        return response;
    }
}

[mbtolou]

Hi.

Given the compression power and reduced memory and processor usage, can you switch the library compressor to ZSTD?