Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.
Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.
Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.
Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.
We found that too many quick calls to HttpWebRequest.GetResponse() causes the request not to be sent, and the call blocks until the configured timeout is reached. I reproduced this easily in a small program by making requests to a noop server: while (true) { var client = (HttpWebRequest)WebRequest.Create("http://localhost:5000/"); client.Timeout = (int)TimeSpan.FromSeconds(10); client.GetResponse(); } The first few (6-8) requests succeed quickly, and then calls fail with a timeout error despite not having sent the request to the server. Once in this state it appears that the framework cannot recover, so all calls fail due to timeout from that point on until the application restarts. The value of the timeout doesn't matter, and if no timeout is specified the call fails after around 100 seconds (which I think is the default). I also tried using the GetResponseAsync() method and that one exhibits the same behavior except that it never times out, so the first call that runs into this situation hangs forever.
Created attachment 6840 [details] --trace=N:System.Net output for call that hangs I ran mono with the --trace=N:System.Net option, and this log is the bottom portion of the log right after the last successful web request.
After looking at the trace I think what might be happening is that the request gets queued in WebConnection.cs, line 819. I see no obvious place where this queue is being processed in this scenario, so the request probably just sits in the queue indefinitely.
In my code above, I missed the TotalMilliseconds property, so the code should be: while (true) { var client = (HttpWebRequest)WebRequest.Create("http://localhost:5000/"); client.Timeout = (int)TimeSpan.FromSeconds(10).TotalMilliseconds; client.GetResponse(); } Also I found where the SendNext should be called, which is on Close of the previous connection. However I don't know yet whether or not it's actually being called in this case.
We figured out that the test case is invalid. According to the .NET documentation, failure to close the response can lead to running out of connections. This bug can be closed as not an issue. Adding a call to Close in the test code makes it all work just fine.
I added some code to Mono in my local system to watch for any connections that are left open for more than 10 seconds and reproduced the issue using our application. The new code did not detect any connections left open, so somehow the connections are being used up or getting hung some other way. This leads me to believe there is in fact a defect in Mono that is causing this despite the earlier test case being invalid.
Created attachment 6866 [details] New test case Here's a new test case. This works fine in .NET but in Mono many of the calls fail with a WebException "The request timed out". After the first iteration of the while loop almost all calls fail with the timeout error.
It's also worth noting with the attached test case that you can run into errors in .NET by kicking off ~1000 requests before calling EndGetResponse on any of them. However most of the calls still succeed, and the error in that case is a WebException "Unable to connect to the remote server". I can reproduce the timeout in Mono with as few as 4 outstanding requests, and 5 hits the issue regularly.
You are starving the ThreadPool: http://www.mono-project.com/Article:ThreadPool_Deadlocks
@Miguel - With this new test case I run into the issue with only a handful of threads (I saw 9 when I attached in gdb). Setting MONO_THREADS_PER_CPU to 2000 does not change the behavior. In our full application we've been running with MONO_THREADS_PER_CPU=200 and I observed 214 threads at the time the issue occurred, with two logical processors given to the VM.
Any updates on this?
I'm fairly sure I am experiencing this bug. It only started when I did a distribution release update (to Ubuntu 12.04 to 14.04 - Mono 2.10.8 to 3.2.8). I'm not sure if that helps narrow down the problem.
I think I finally have some understanding of what is going on here. In .NET, the timeout value on a request is not in effect until the request gets sent. Therefore you see some calls to GetResponse() taking longer than the configured timeout if the request can't be made for a while due to heavy load. In Mono on the other hand, the timeout is in effect as soon as GetResponse is called. So if a request needs to wait for a while, you could see a timeout error before the request gets sent, or at least sooner than you would see it in .NET. The workaround we plan to implement is to throttle requests at a higher level to avoid request queuing in Mono so that we never (or at least very rarely) get into this situation. That said it would be best if Mono behaved like .NET in this scenario.
As for today, we are facing the same problem running on Ubuntu 14.04 and Mono 3.2.8, Does any one have found out how to fix or avoid this? We also have a configuration for MONO_THREADS_PER_CPU Thanks in advance :)
Same issue is experienced on Ubuntu 14.04 and Mono 3.2.8 Is this fixed in a later version?
I got the same problem using Web References with Xamarin Android 6.1.2.21. Two or three requests work well, then there are only WebExceptions due to timeouts. Restart the app and all works again. Priority should probably be higher, since this is an elementary function for many projects.
One quick remark: I solved my problems downgrading to Xamarin 4.0.3 (Xamarin.Android 6.0.3.5).
The error seem to be introduced again since Xamarin 4.1 4.1.2.18 = Problem 4.1.0 = Problem 4.0.4.4 = OK 4.0.3 = OK
The error seems to be adressed in the latest beta release Xamarin 4.2.0.675. https://releases.xamarin.com/beta-preview-4-cycle-8/ The release notes state: 42864 – “System.Net.WebException: Error: NameResolutionFailure” on second web request to certain raw IP addresses with HttpClient I tested it and it seem to adress this problem, too.
Bug has come back in version 4.2.1 (Debian 4.2.1.102+dfsg2-7ubuntu4) Was working before I update my distribution to xenial from trusty.
This sees to be related to https://bugzilla.xamarin.com/show_bug.cgi?id=45761 which is in the latest Stable Release: Cycle 8 Service Release 2: [Mono Framework] – 45761 – After network reconnected, web request fails for a couple of minutes with a NameResolutionFailure https://bugzilla.xamarin.com/show_bug.cgi?id=45761 And may be fixed in Alpha Preview 7: Cycle 9. But I have not found te time to test this, since Alpha Preview 7: Cycle 9 has other issues with Android and appcompat in my project.
Could you please try to update to any recent version and try to reproduce the issue again. If the issue still persists please include the version information and change the bug status to NEW.
I'm afraid I do not have a specific reproduction scenario, but am getting a lot of reports for this from users of my app. App runs on macOS (various versions), with Mono v5.2.0.224 (Now updating to latest stable 5.4.0.201, but haven't tested this yet) My app does a lot of web requests to a server. Occasionally, we see errors of this type: : send - : Exception!! Error getting response stream (ReadDone2): ReceiveFailure at System.Net.WebClient.DownloadDataInternal (System.Uri address, System.Net.WebRequest& request) [0x00072] in <5042234bf3a846de97c5c528912576b6>:0 at System.Net.WebClient.DownloadString (System.Uri address) [0x00020] in <5042234bf3a846de97c5c528912576b6>:0 at System.Net.WebClient.DownloadString (System.String address) [0x00016] in <5042234bf3a846de97c5c528912576b6>:0 at (wrapper remoting-invoke-with-check) System.Net.WebClient:DownloadString (string) at Client.Business.API.ApiRequest`1[T].HttpRequest (System.String pUrl) [0x000a0] in <5152f9f14f9d4a04a8fbb13d79a33276>:0 at Client.Business.API.ApiRequest`1[T].send (System.String pUrl) [0x0000e] in <5152f9f14f9d4a04a8fbb13d79a33276>:0 When retrying, the connection works. However, a while later, after some more of the 'ReadDone2' errors, users start seeing errors like the one reported in this issue: WebException: The request timed out at System.Net.WebClient.DownloadDataInternal (System.Uri address, System.Net.WebRequest& request) [0x00072] in <5042234bf3a846de97c5c528912576b6>:0 at System.Net.WebClient.DownloadString (System.Uri address) [0x00020] in <5042234bf3a846de97c5c528912576b6>:0 at System.Net.WebClient.DownloadString (System.String address) [0x00016] in <5042234bf3a846de97c5c528912576b6>:0 at (wrapper remoting-invoke-with-check) System.Net.WebClient:DownloadString (string) at Client.Business.API.ApiRequest`1[T].HttpRequest (System.String pUrl) [0x000a0] in <30c6f32f074d4fc28bd2fac4820ada2c>:0 at Client.Business.API.ApiRequest`1[T].send (System.String pUrl) [0x0000e] in <30c6f32f074d4fc28bd2fac4820ada2c>:0 ==> once this happens, all further web requests fail, it never 'recovers'. The only solution is to restart the full app. While I can't provide you with a full reproduction scenario (working on that), I can confirm this issue still exists in Mono v5.2.0.224 I can't be sure, but I think the 2 problems I see are related: is it possible that a WebResponse isn't properly closed/disposed when the ReadDone2 error occurs, and as a result, eventually, Mono stops handling new connections? (running out of threads/memory/...?)
Same issue on 5.4.1.6 Immediate timeout of some requests shows this is caused by the queue and the effect described by David Straw.
Not sure if this is related, but I'm having the opposite effect. Sending a request to a potentially non-reachable endpoint takes exactly one minute for the System.Net.WebClient to throw an exception. I extended the WebClient by overriding the GetWebRequest API and set the timeout before returning the new WebRequest. The value is ignored regardless... Anyone else experiencing this? ------------------------------------------------------------------- Microsoft Visual Studio Professional 2017 Version 15.6.4 VisualStudio.15.Release/15.6.4+27428.2015 Microsoft .NET Framework Version 4.6.01590 Xamarin 4.9.0.752 (0faa808ee) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android. Xamarin Designer 4.10.58 (cee1369d0) Visual Studio extension to enable Xamarin Designer tools in Visual Studio. Xamarin.Android SDK 8.2.0.16 (HEAD/a78295902) Xamarin.Android Reference Assemblies and MSBuild support. Xamarin.iOS and Xamarin.Mac SDK 11.8.1.28 (3979d08) Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
I am seeing exactly the same problem on Mono 5.10 on CentOS 7. This is the error: The operation has timed out., Stack Trace: at System.Net.HttpWebRequest+<RunWithTimeoutWorker>d__244`1[T].MoveNext () [0x000ba] in <93075ab5c03242e39aa9261622b472b8>:0 --- End of stack trace from previous location where exception was thrown --- at System.Net.HttpWebRequest.GetResponse () [0x00013] in <93075ab5c03242e39aa9261622b472b8>:0 I have a web app running on Apache that runs the following code often: Uri myUri = new Uri(...); WebRequest myWebRequest = HttpWebRequest.Create(myUri); HttpWebRequest request = (HttpWebRequest)myWebRequest; request.Method = "GET"; request.ContentType = "application/json; charset=utf-8"; using (Stream s = request.GetResponse().GetResponseStream()) { using (StreamReader sr = new StreamReader(s)) { var jsonData = sr.ReadToEnd(); return; } } After a while it simply stops working, throwing the error mentioned above.