Bug 1656 - System.Net.HttpListenerException: I/O operation aborted. While sending a chunked POST request.
Summary: System.Net.HttpListenerException: I/O operation aborted. While sending a chun...
Status: RESOLVED FIXED
Alias: None
Product: Class Libraries
Classification: Mono
Component: System ()
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: Untriaged
Assignee: Gonzalo Paniagua Javier
URL:
Depends on:
Blocks:
 
Reported: 2011-10-24 04:48 UTC by Andrej Pregl
Modified: 2012-02-14 02:54 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Test case (7.93 KB, text/x-csharp)
2011-10-24 04:48 UTC, Andrej Pregl
Details
Updated test case (7.93 KB, text/x-csharp)
2012-02-13 10:28 UTC, Andrej Pregl
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Andrej Pregl 2011-10-24 04:48:44 UTC
Created attachment 757 [details]
Test case

The problem seems to arise when the chunks are being split up in multiple tcp packets. We sniffed the traffic with Wireshark and split it up according to the tcp packets which we're coming in.

Our guess is that the ChunkedReader always expect a length in the start of the packets and doesn't handle the case where the length can end up being in the middle of a tcp packet.

We're able to reproduce the bug with the attached test code running in mono 2.6.7 on Ubuntu 11.04.

Here's a link to our pull request https://github.com/mono/mono/pull/182
Comment 1 Andrej Pregl 2011-10-24 05:25:52 UTC
This bug also is reproducible in Mono 2.10.6 on OSX and Mono (Debian 2.10.5-1) on Ubuntu 11.10.
Comment 2 Gonzalo Paniagua Javier 2011-11-12 13:22:59 UTC
The output I get, with OR without the patch from the pull request is:
Server started, connecting listener in 1000 msec
Sending data
Writing 0
Writing 1
Writing 2
Writing 3
System.Net.HttpListenerException: I/O operation aborted.
  at System.Net.ChunkedInputStream.EndRead (IAsyncResult ares) [0x00072] in /home/gonzalo/mono/git/wd/mono/mcs/class/System/System.Net/ChunkedInputStream.cs:165 
  at System.Net.ChunkedInputStream.Read (System.Byte[] buffer, Int32 offset, Int32 count) [0x0000c] in /home/gonzalo/mono/git/wd/mono/mcs/class/System/System.Net/ChunkedInputStream.cs:74 
  at System.IO.StreamReader.ReadBuffer () [0x00012] in /home/gonzalo/mono/git/wd/mono/mcs/class/corlib/System.IO/StreamReader.cs:394 
  at System.IO.StreamReader.Read (System.Char[] buffer, Int32 index, Int32 count) [0x00062] in /home/gonzalo/mono/git/wd/mono/mcs/class/corlib/System.IO/StreamReader.cs:471 
  at System.IO.StreamReader.ReadToEnd () [0x0002b] in /home/gonzalo/mono/git/wd/mono/mcs/class/corlib/System.IO/StreamReader.cs:573 
  at ChunkReaderTest.MainClass.HandleRequest (IAsyncResult result) [0x00023] in /tmp/Main.cs:97 
Data sent
Comment 3 Björn Gustafsson 2011-11-16 07:19:41 UTC
That is very peculiar.

We're currently running that patch as a temporary fix on 3 Debian systems with 2.6.7, and on 2 dev laptops with the version in Ubuntu 11.04.

There is a deeper exception here, specifically referencing "count < 0", which I believe is in System/Net/RequestStream.cs (throw new ArgumentOutOfRangeException ("count", "< 0")) , and it's being replaced in ChunkedInputStream at this point:

if (my_ares.Error != null)
     throw new HttpListenerException (400, "I/O operation aborted.");

The "count" in RequestStream here is directly passed from ChunkStream.ChunkLeft, which in the case forced with the test code is < 0, instead of the expected 0 or above.
Comment 4 Gonzalo Paniagua Javier 2012-02-10 02:14:13 UTC
Running the program on windows with MS.NET yields: 
------------------------------------
Server started, connecting listener in 1000 msec
Sending data
Writing 0
Writing 1
Writing 2
Writing 3
System.Net.HttpListenerException: Incorrect function
   at System.Net.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 size)

   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadToEnd()
   at ChunkReaderTest.MainClass.HandleRequest(IAsyncResult result)
Data sent
-----------------------------------

I have added 2 lines to reset chunkRead and chunkSize before attempting to read the chunk size and now Mono outputs:
---------------------------------
Server started, connecting listener in 1000 msec
Sending data
Writing 0
Writing 1
Writing 2
Writing 3
System.Net.HttpListenerException: I/O operation aborted: Expecting \r
  at System.Net.ChunkedInputStream.EndRead (IAsyncResult ares) [0x00000] in <filename unknown>:0 
  at System.Net.ChunkedInputStream.Read (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0 
  at System.IO.StreamReader.ReadBuffer () [0x00000] in <filename unknown>:0 
  at System.IO.StreamReader.Read (System.Char[] buffer, Int32 index, Int32 count) [0x00000] in <filename unknown>:0 
  at System.IO.StreamReader.ReadToEnd () [0x00000] in <filename unknown>:0 
  at ChunkReaderTest.MainClass.HandleRequest (IAsyncResult result) [0x00000] in <filename unknown>:0 
Data sent
------------------------------



which gives you a better idea of what's wrong with your test.
Comment 5 Andrej Pregl 2012-02-10 08:14:46 UTC
Hi Gonzalo.

It's good that you've added those lines but it still doesn't fix the issue.

We're currently using an embedded device running J2ME and using the HttpConnection which automatically forces chunked transfers when the string you're sending is longer than 2016 chars. This is done by the API itself and we haven't found a way to disable it. 

The actual HTTP packet is valid, and mono seems to have no problem with if it is either
a) split up in tcp packets on the "logical" boundaries (i.e. break before length of next chunk), or
b) sent as one continuous packet.

However when the tcp packet break is "in the middle", the ChunkReader misbehaves and returns a negative length when asked how much is left to read.

The test we wrote is based on the HTTP packets we were able to sniff with Wireshark in order to provide you with the test code we attached here. To clarify: the test is not an attempt to force an error in mono, it is the data we have sniffed which highlights it. If it could be avoided, we would naturally not send the data this way.

We're aware that the fix we committed (https://github.com/mono/mono/pull/182) might just mask the issue it self so it would be great if the main issue itself could be fixed.
Comment 6 Gonzalo Paniagua Javier 2012-02-10 12:39:37 UTC
What I said is that the test does not work on Microsoft .NET and does not work on Mono. The end result is that either there is no test case or there is no bug because Microsoft.NET behaves the same way.
Comment 7 Andrej Pregl 2012-02-13 10:28:13 UTC
Created attachment 1354 [details]
Updated test case

Finally a working test case which runs on Microsoft.NET but fails on Mono.
Comment 8 Andrej Pregl 2012-02-13 10:28:35 UTC
So we were quite confused by why the test case wasn't working in Microsoft.NET but we managed finally to find the issue.

The last part of the package in the case that fails was missing a \r\n in the end which made it always fail. 

I'v attached the updated test code which runs on Microsoft.NET and fails on Mono.
Our patch should solve the issue but might be masking the problem itself as we earlier mentioned.
Comment 9 Gonzalo Paniagua Javier 2012-02-13 10:36:09 UTC
Mono 2.10.5 and 2.10.8.1 fail.
Mono from master works fine.

I'm going to check if there's any missing patch between master and the mono-2-10 branch.
Comment 10 Gonzalo Paniagua Javier 2012-02-13 11:19:11 UTC
Either the patch I backported fixed it or the latest code in mono-2-10 already fixes the issue:
-----------------------------
$ mono --version
Mono JIT compiler version 2.10.8.1 (mono-2-10/bb45dcf Mon Feb 13 11:07:38 EST 2012)
Copyright (C) 2002-2011 Novell, Inc, Xamarin, Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            Included Boehm (with typed GC and Parallel Mark)
$ mono /tmp/Main-patched.exe 
Server started, connecting listener in 1000 msec
Sending data
Writing 0
Writing 1
Writing 2
Writing 3
Data sent
----------------------------

Master works fine.

Thanks for the test cases!
Comment 11 Andrej Pregl 2012-02-14 02:54:28 UTC
I'm glad we could help and thanks for investigating the issue. Thankfully we finally managed to pinpoint the issue and that it's been resolved.

Since our servers are running the latest stable release of Debian which is using Mono 2.6.7 we're interested in hearing if this will be patched in 2.6.7?