Need to integrate zstandard compression algorithm to compress/decompress API request/response

Question

Need to integrate zstandard compression algorithm to compress/decompress API request/response

Hardik Bhadania 0

I need to integrate zstandard compression algorithm to compress/decompress API request/response.

I've started doing this by CustomCompressionProvider class which implements ICompressionProvider interface.

While implementing CreateStream method, I'm confused from where we can get actual API response data which I need to compress. Also didn't get what is the use of argument of this CreateStream method.

Anonymous

2023-09-04T03:46:29.6633333+00:00

Hi @Hardik Bhadania

Can you share the related code? So that we can know which kind of CreateStream method you are using and help understand what is the use of argument of this CreateStream method?

Besides, since you are using zstandard, which is a third-party package, if the issue relates it, you'd better to ask for help from the zstandard forum. Thanks for your understanding.

For the stream in Asp.net core request and response, you can refer to Request and response operations in ASP.NET Core.

Hardik Bhadania 0

Hi @Anonymous , Sure. Here is the code snippet. Please do let me know your thoughts.

using ImpromptuNinjas.ZStd;
using Microsoft.AspNetCore.ResponseCompression;
using System.IO;
using System.IO.Compression;

namespace Client.Custom {
    public class CustomCompressionProvider : ICompressionProvider {
        public string EncodingName => "zstd";
        public bool SupportsFlush => true;
        public int CompressionLevel => 3;
        //var compressionLevel = ZStdCompressor.MinimumCompressionLevel; // probably 1
        // Note: negative levels do exist as per zstd documentation, but this just calls ZSTD_minCLevel
        //var compressionLevel = ZStdCompressor.MaximumCompressionLevel; // probably 22

        public Stream CreateStream(Stream outputStream) {
            // Replace with a custom compression stream wrapper.

            // have somewhere to put the compressed output, any stream will do
            using var compressed = new MemoryStream();

            // create a compression context over the output stream
            using var compressStream = new ZStdCompressStream(compressed);

            // ZStdCompressStream.Compressor is just a ZStdCompressor, so you can set parameters and use dictionaries
            compressStream.Compressor.Set(CompressionParameter.CompressionLevel, CompressionLevel);

            byte[] bytes = ReadFully(outputStream);

            // write some data into it
            compressStream.Write(bytes, 0, bytes.Length);

            // if you're done writing data and you're not at the end of a frame, it'd be wise to flush
            compressStream.Flush();
            return outputStream;
        }

        private static byte[] ReadFully(Stream input) {
            using var ms = new MemoryStream();
            input.CopyTo(ms);
            return ms.ToArray();
        }
    }
}

Hardik Bhadania 0

Hi @Anonymous , Here is the code snippet.

using ImpromptuNinjas.ZStd;
using Microsoft.AspNetCore.ResponseCompression;
using System.IO;
using System.IO.Compression;

namespace Client.Custom {
    public class CustomCompressionProvider : ICompressionProvider {
        public string EncodingName => "zstd";
        public bool SupportsFlush => true;
        public int CompressionLevel => 3;
        //var compressionLevel = ZStdCompressor.MinimumCompressionLevel; // probably 1
        // Note: negative levels do exist as per zstd documentation, but this just calls ZSTD_minCLevel
        //var compressionLevel = ZStdCompressor.MaximumCompressionLevel; // probably 22

        public Stream CreateStream(Stream outputStream) {
            // Replace with a custom compression stream wrapper.

            // have somewhere to put the compressed output, any stream will do
            using var compressed = new MemoryStream();

            // create a compression context over the output stream
            using var compressStream = new ZStdCompressStream(compressed);

            // ZStdCompressStream.Compressor is just a ZStdCompressor, so you can set parameters and use dictionaries
            compressStream.Compressor.Set(CompressionParameter.CompressionLevel, CompressionLevel);

            byte[] bytes = ReadFully(outputStream);

            // write some data into it
            compressStream.Write(bytes, 0, bytes.Length);

            // if you're done writing data and you're not at the end of a frame, it'd be wise to flush
            compressStream.Flush();
            return outputStream;
        }

        private static byte[] ReadFully(Stream input) {
            using var ms = new MemoryStream();
            input.CopyTo(ms);
            return ms.ToArray();
        }
    }
}

Anonymous

Hi @Hardik Bhadania

To get Http response in Asp.net core API, you can create a custom middleware.

Refer to the following code:

public class RequestResponseLoggerMiddleware  
{  
    private readonly RequestDelegate _next;  
    private readonly bool _isRequestResponseLoggingEnabled;  
  
    public RequestResponseLoggerMiddleware(RequestDelegate next, IConfiguration config)  
    {  
        _next = next;  
        _isRequestResponseLoggingEnabled = config.GetValue<bool>("EnableRequestResponseLogging", false);  
    }  
  
    public async Task InvokeAsync(HttpContext httpContext)  
    {  
        // Middleware is enabled only when the EnableRequestResponseLogging config value is set.  
        if (_isRequestResponseLoggingEnabled)  
        {  
            Console.WriteLine($"HTTP request information:\n" +  
                $"\tMethod: {httpContext.Request.Method}\n" +  
                $"\tPath: {httpContext.Request.Path}\n" +  
                $"\tQueryString: {httpContext.Request.QueryString}\n" +  
                $"\tHeaders: {FormatHeaders(httpContext.Request.Headers)}\n" +  
                $"\tSchema: {httpContext.Request.Scheme}\n" +  
                $"\tHost: {httpContext.Request.Host}\n" +  
                $"\tBody: {await ReadBodyFromRequest(httpContext.Request)}");  
  
            // Temporarily replace the HttpResponseStream, which is a write-only stream, with a MemoryStream to capture it's value in-flight.  
            var originalResponseBody = httpContext.Response.Body;  
            using var newResponseBody = new MemoryStream();  
            httpContext.Response.Body = newResponseBody;  
  
            // Call the next middleware in the pipeline  
            await _next(httpContext);  
  
            newResponseBody.Seek(0, SeekOrigin.Begin);  
            var responseBodyText = await new StreamReader(httpContext.Response.Body).ReadToEndAsync();  
  
            Console.WriteLine($"HTTP request information:\n" +  
                $"\tStatusCode: {httpContext.Response.StatusCode}\n" +  
                $"\tContentType: {httpContext.Response.ContentType}\n" +  
                $"\tHeaders: {FormatHeaders(httpContext.Response.Headers)}\n" +  
                $"\tBody: {responseBodyText}");  
  
            newResponseBody.Seek(0, SeekOrigin.Begin);  
            await newResponseBody.CopyToAsync(originalResponseBody);  
        }  
        else  
        {  
            await _next(httpContext);  
        }  
    }  
  
    private static string FormatHeaders(IHeaderDictionary headers) => string.Join(", ", headers.Select(kvp => $"{<!-- -->{<!-- -->{kvp.Key}: {string.Join(", ", kvp.Value)}}}"));  
  
    private static async Task<string> ReadBodyFromRequest(HttpRequest request)  
    {  
        // Ensure the request's body can be read multiple times (for the next middlewares in the pipeline).  
        request.EnableBuffering();  
  
        using var streamReader = new StreamReader(request.Body, leaveOpen: true);  
        var requestBody = await streamReader.ReadToEndAsync();  
  
        // Reset the request's body stream position for next middleware in the pipeline.  
        request.Body.Position = 0;  
        return requestBody;  
    }  
}

More detail information, check these links:

ASP.NET Core Web API how to log requests and responses into database.

Request Response Logging Middleware ASP.NET Core

Logging HTTP Request and Response in .NET Web API

After getting the stream from the response, you can call the custom compression method to compression the steam.

Best regards,
Dillion.

Your answer

Anonymous

2023-09-04T03:46:29.6633333+00:00

Hi @Hardik Bhadania

Can you share the related code? So that we can know which kind of CreateStream method you are using and help understand what is the use of argument of this CreateStream method?

Besides, since you are using zstandard, which is a third-party package, if the issue relates it, you'd better to ask for help from the zstandard forum. Thanks for your understanding.

For the stream in Asp.net core request and response, you can refer to Request and response operations in ASP.NET Core.
Hardik Bhadania 0 Reputation points

2023-09-12T11:22:44.1166667+00:00

Hi @Anonymous , Sure. Here is the code snippet. Please do let me know your thoughts.

using ImpromptuNinjas.ZStd; using Microsoft.AspNetCore.ResponseCompression; using System.IO; using System.IO.Compression; namespace Client.Custom { public class CustomCompressionProvider : ICompressionProvider { public string EncodingName => "zstd"; public bool SupportsFlush => true; public int CompressionLevel => 3; //var compressionLevel = ZStdCompressor.MinimumCompressionLevel; // probably 1 // Note: negative levels do exist as per zstd documentation, but this just calls ZSTD_minCLevel //var compressionLevel = ZStdCompressor.MaximumCompressionLevel; // probably 22 public Stream CreateStream(Stream outputStream) { // Replace with a custom compression stream wrapper. // have somewhere to put the compressed output, any stream will do using var compressed = new MemoryStream(); // create a compression context over the output stream using var compressStream = new ZStdCompressStream(compressed); // ZStdCompressStream.Compressor is just a ZStdCompressor, so you can set parameters and use dictionaries compressStream.Compressor.Set(CompressionParameter.CompressionLevel, CompressionLevel); byte[] bytes = ReadFully(outputStream); // write some data into it compressStream.Write(bytes, 0, bytes.Length); // if you're done writing data and you're not at the end of a frame, it'd be wise to flush compressStream.Flush(); return outputStream; } private static byte[] ReadFully(Stream input) { using var ms = new MemoryStream(); input.CopyTo(ms); return ms.ToArray(); } } }
Hardik Bhadania 0 Reputation points

2023-09-12T11:27:26.7233333+00:00

Hi @Anonymous , Here is the code snippet.

using ImpromptuNinjas.ZStd; using Microsoft.AspNetCore.ResponseCompression; using System.IO; using System.IO.Compression; namespace Client.Custom { public class CustomCompressionProvider : ICompressionProvider { public string EncodingName => "zstd"; public bool SupportsFlush => true; public int CompressionLevel => 3; //var compressionLevel = ZStdCompressor.MinimumCompressionLevel; // probably 1 // Note: negative levels do exist as per zstd documentation, but this just calls ZSTD_minCLevel //var compressionLevel = ZStdCompressor.MaximumCompressionLevel; // probably 22 public Stream CreateStream(Stream outputStream) { // Replace with a custom compression stream wrapper. // have somewhere to put the compressed output, any stream will do using var compressed = new MemoryStream(); // create a compression context over the output stream using var compressStream = new ZStdCompressStream(compressed); // ZStdCompressStream.Compressor is just a ZStdCompressor, so you can set parameters and use dictionaries compressStream.Compressor.Set(CompressionParameter.CompressionLevel, CompressionLevel); byte[] bytes = ReadFully(outputStream); // write some data into it compressStream.Write(bytes, 0, bytes.Length); // if you're done writing data and you're not at the end of a frame, it'd be wise to flush compressStream.Flush(); return outputStream; } private static byte[] ReadFully(Stream input) { using var ms = new MemoryStream(); input.CopyTo(ms); return ms.ToArray(); } } }

Share via

Need to integrate zstandard compression algorithm to compress/decompress API request/response

Your answer