OutputFormatter provides no way to disable text encoding

[timothy-shields]

Title

OutputFormatter provides no way to disable text encoding

Functional impact

It seems impossible to write a class derived from OutputFormatter for handling a custom binary media type that uses no text encoding.

Minimal repro steps

1. Create new Web API app (using "Microsoft.AspNet.Mvc": "6.0.0-beta4")

2. Add a MyBinaryOutputFormatter class:

   public class MyBinaryOutputFormatter : OutputFormatter
   {
       public MyBinaryOutputFormatter()
       {
           SupportedMediaTypes.Add(new MediaTypeHeaderValue("application/x-mybinary"));
           SupportedEncodings.Add(Encoding.UTF8);
       }
   
       public override bool CanWriteResult(OutputFormatterContext context, MediaTypeHeaderValue contentType)
       {
           return context.Object is byte[];
       }
   
       public override async Task WriteResponseBodyAsync(OutputFormatterContext context)
       {
           using (var stream = new System.IO.MemoryStream())
           {
               byte[] data = (byte[])context.Object;
               await context.ActionContext.HttpContext.Response.Body.WriteAsync(data, 0, data.Length);
           }
       }
   }
   

3. Add the output formatter in Startup.cs:

   services.AddMvc().Configure<MvcOptions>(options =>
   {
       options.OutputFormatters.Add(new MyBinaryOutputFormatter());
   });
   

4. Add a controller and action like the following:

   [HttpGet]
   public byte[] Test()
   {
       return new byte[] { 0xFF };
   }
   

5. Start the service and do a GET request to the action method using a HTTP client that allows you to inspect the response body contents as hex.

Expected result

Since I write to the response body stream directly, I would expect the bytes I write to be exactly those seen in the response body client-side. That is the response body seen client side should be the single byte FF (hex).

Actual result

The response body seen client side is the single byte FD (hex).

Further technical details

Using Encoding.ASCII instead in the MyBinaryOutputFormatter constructor seems to be a hacky workaround to this problem, but then all responses have a Content-Type header of application/x-mybinary; charset=us-ascii, which doesn't look great to clients.

If the single-byte response is not convincing that the problem is the encoding, try returning a larger byte[] from the Test action method. It quickly becomes evident that the encoding is corrupting the data.

[danroth27]

@Tratcher Why would what got written to the body change in this case?

[timothy-shields]

I actually forgot to add one other point: what I think the user expectation would be here. My expectation was that the SupportedEncodings and the OutputFormatterContext.SelectedEncoding would never be automatically applied, that they would only be automatically negotiated and selected, and that I should be responsible for encoding any textual information according to the SelectedEncoding.

Furthermore, it should be possible to have no encoding at all - e.g. Content-Type: application/x-mybinary instead of Content-Type: application/x-mybinary; charset=.... Is it possible to do this? Most binary formats have no concept of choosing a charset.

[Tratcher]

I wouldn't expect the above code to encode the body in any way. I can only assume that WriteResponseBodyAsync isn't being called and some other code is sending the body. @rynowak?

[timothy-shields]

@Tratcher WriteResponseBodyAsync is definitely being called, I have a breakpoint and can step through it all with the debugger. This is a minimal example that reproduces the problem, but in my original scenario I had a "real" serialization going on in my custom OutputFormatter for which there was no way something else could be sending the body. Examining the hex of the response body client-side revealed I was definitely getting the results of my custom serialization done in WriteResponseBodyAsync - just they were corrupted because of a UTF8 encoding.

[timothy-shields]

I figured out what the problem is... The Chrome App "DHC" which I am using as my test client is failing to render the correct hex for the response body! It appears that it is stripping the charset off of the Content-Type on the response and then modifying the response body based on the charset. I made the request with a different test client and the hex was all correct.

In summary: The problem was with the DHC client, not MVC 6. Apologies for not figuring that out without bothering you all.

I still am perplexed as to how I can entirely omit any encodings and charsets from my OutputFormatter and response headers.

[rynowak]

@timothy-shields thanks for the feedback. We're looking into making a fix for OutputFormatter to make it leave out a charset if SupportedEncodings is empty (instead of throwing).

As a workaround, you might want to look into implementing IOutputFormatter directly. Take a look at StreamOutputFormatter. It's not a 100% fit for what you're doing - StreamOutputFormatter will write any content-type, but you should be able to implement what you want.