Bug 32978 - Race condition in HttpListener.Close
Summary: Race condition in HttpListener.Close
Status: RESOLVED ANSWERED
Alias: None
Product: Class Libraries
Classification: Mono
Component: System.Net.Http ()
Version: 2.10.x
Hardware: PC Linux
: --- normal
Target Milestone: Untriaged
Assignee: Marek Safar
URL:
Depends on:
Blocks:
 
Reported: 2015-08-11 17:09 UTC by Christian Fahr
Modified: 2015-08-18 09:41 UTC (History)
2 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED ANSWERED

Description Christian Fahr 2015-08-11 17:09:15 UTC
The following program crashes while terminating with Unhandled Exception: System.InvalidOperationException: Hashtable.Enumerator: snapshot out of sync.
To reproduce the bug use something like ab -t11 -c10 localhost:8080/ to generate many requests.
You may find that you need to run this several times, as the bug is non-determenistic.
It occurs if the Close() gets called under heavy load.
Futhermore, I'm sorry but I cannot rule out that there is a bug in the program itself, as I found the msdn does not fully specify the behavior of the class. (May there be requests running while closing? When exactly do callbacks get called? It does not even mention Close() calls the callback, etc. https://msdn.microsoft.com/en-us/library/system.net.httplistener.begingetcontext%28v=vs.110%29.aspx even implies the IAsyncResult WaitHandles get signaled when the callback is done, while they really get signaled when the after request processing together with the callback call, it seems.)

using System;
using System.Net;
using System.Threading;

namespace HttpAsync
{
	class MainClass
	{
		static HttpListener listener = new HttpListener();
		const int n = 10;
		static IAsyncResult[]  ares = new IAsyncResult[n];
		// AutoResetEvent's which signal if request handler i is done.
		static AutoResetEvent[] whandle = new AutoResetEvent[n];
		static Object running_lock = new Object();
		static volatile Boolean running = false;
		static Thread schedThread = new Thread(Schedule);
		
		public static void Main (string[] args)
		{
			Console.WriteLine ("Startup.");
			for(int j = 0; j < n; ++j)
			{
				// signal initially, so request handlers get created by schedule thread
				whandle[j] = new AutoResetEvent(true);
			}
			listener.Prefixes.Add(String.Format (@"http://+:{0}/", 8080));
			running = true;
			listener.Start();
			Console.WriteLine("Scheduler starting...");
			schedThread.Start();
			
			Thread.Sleep(10 * 1000);
			
			lock(running_lock)
			{
				running = false;
			}
			Console.WriteLine("Starting to shut down...");
			
			listener.Close();
			schedThread.Join();

			Console.WriteLine("Terminated.");
		}
		
		public static void Schedule()
		{
			Console.WriteLine("Request scheduler thread started.");
			
			while(running)
			{
				int j = System.Threading.WaitHandle.WaitAny(whandle);
				Console.WriteLine(String.Format("Request handler {0} done.", j));
				// the listener really MUST be open during the BeginGetContext call
				// we need a read lock here
				lock(running_lock)
				{
					if(running)
					{
						ares[j] = listener.BeginGetContext(new AsyncCallback(ListenerCallback), j);
					}
					else
					{
						// We just consumed whandle[j] and there is no new request to set it again:
						// restore it, so we can terminate.
						whandle[j].Set();
					}
				}
			}
			
			Console.WriteLine("Scheduler shutting down: waiting for all requests to finish.");
			for(int j = 0; j < n; ++j)
			{
				whandle[j].WaitOne();
				Console.WriteLine(String.Format("Request handler {0} terminated.", j));
			}
			Console.WriteLine("Request scheduler thread terminates.");
		}
		
		public static void ListenerCallback(IAsyncResult result)
		{
			// important: listener.Stop() (and hence Close)
			// closes the socket and fires all callbacks,
			// hence EndGetContext will block in this case.
			// Check if we closed the listener.
			// If so, do nothing and just signal the request.
			// This is a safe way to guarantee both that there is
			//   a) no request that started and is still running
			//   b) check a request really is not started
			int reqid = (int) result.AsyncState;
			if(running)
			{
				// note: this call is always ok, if the listener was running
				// while we got into this callback, that is
				// because of a request, not because it was closed.
				// This means its not a problem if the server gets closed before
				// this gets executed.
				HttpListenerContext context = listener.EndGetContext(result);
				HttpListenerRequest request = context.Request;
				HttpListenerResponse response = context.Response;
				
				Console.WriteLine(String.Format("Request: {0} {1}", request.Url, reqid));
				
				// you would do something here
				//Thread.Sleep(40);
				string responseString = String.Format("<HTML><BODY> Hello from request handler {0}!</BODY></HTML>\n", reqid);
				byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
				response.ContentLength64 = buffer.Length;
				
				System.IO.Stream output = response.OutputStream;
				output.Write(buffer, 0, buffer.Length);
				output.Close();
			}
			// signal this request is done, so we allow a new one
			whandle[reqid].Set();
		}	
	}
}
Comment 1 Christian Fahr 2015-08-11 17:12:30 UTC
Stacktrace:

Unhandled Exception: System.InvalidOperationException: Hashtable.Enumerator: snapshot out of sync.
  at System.Collections.Hashtable+Enumerator.FailFast () [0x00000] in <filename unknown>:0 
  at System.Collections.Hashtable+Enumerator.MoveNext () [0x00000] in <filename unknown>:0 
  at System.Net.EndPointListener.Close () [0x00000] in <filename unknown>:0 
  at System.Net.EndPointManager.RemoveEndPoint (System.Net.EndPointListener epl, System.Net.IPEndPoint ep) [0x00000] in <filename unknown>:0 
  at System.Net.EndPointListener.CheckIfRemove () [0x00000] in <filename unknown>:0 
  at System.Net.EndPointListener.RemovePrefix (System.Net.ListenerPrefix prefix, System.Net.HttpListener listener) [0x00000] in <filename unknown>:0 
  at System.Net.EndPointManager.RemovePrefixInternal (System.String prefix, System.Net.HttpListener listener) [0x00000] in <filename unknown>:0 
  at System.Net.EndPointManager.RemoveListener (System.Net.HttpListener listener) [0x00000] in <filename unknown>:0 
  at System.Net.HttpListener.Close (Boolean force) [0x00000] in <filename unknown>:0 
  at System.Net.HttpListener.Close () [0x00000] in <filename unknown>:0 
  at HttpAsync.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Hashtable.Enumerator: snapshot out of sync.
  at System.Collections.Hashtable+Enumerator.FailFast () [0x00000] in <filename unknown>:0 
  at System.Collections.Hashtable+Enumerator.MoveNext () [0x00000] in <filename unknown>:0 
  at System.Net.EndPointListener.Close () [0x00000] in <filename unknown>:0 
  at System.Net.EndPointManager.RemoveEndPoint (System.Net.EndPointListener epl, System.Net.IPEndPoint ep) [0x00000] in <filename unknown>:0 
  at System.Net.EndPointListener.CheckIfRemove () [0x00000] in <filename unknown>:0 
  at System.Net.EndPointListener.RemovePrefix (System.Net.ListenerPrefix prefix, System.Net.HttpListener listener) [0x00000] in <filename unknown>:0 
  at System.Net.EndPointManager.RemovePrefixInternal (System.String prefix, System.Net.HttpListener listener) [0x00000] in <filename unknown>:0 
  at System.Net.EndPointManager.RemoveListener (System.Net.HttpListener listener) [0x00000] in <filename unknown>:0 
  at System.Net.HttpListener.Close (Boolean force) [0x00000] in <filename unknown>:0 
  at System.Net.HttpListener.Close () [0x00000] in <filename unknown>:0 
  at HttpAsync.MainClass.Main (System.String[] args) [0x00000] in <filename unknown>:0
Comment 2 Marek Safar 2015-08-17 13:07:34 UTC
Already fixed in recent Mono.

Relevant commit eb4e581e880be39082db6ad49b877b2d868f8580
Comment 3 Christian Fahr 2015-08-17 17:44:06 UTC
Hi again,

I checked again against openSUSE "Mono JIT compiler version 3.8.0 (tarball Fri Oct 10 15:00:11 UTC 2014)" which seems to be newer than eb4e581e880be39082db6ad49b877b2d868f8580. The runtime crashes with several different exceptions and segfaults in native code under load with ab2.
This happens during runtime not only during shutdown.
I have yet to check against the mono alpha.

Here is one:

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00000] in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
Comment 4 Christian Fahr 2015-08-18 09:41:09 UTC
I checked again with the newest mono from git.
Now it crashes during runtime.
I opened another bug for this: https://bugzilla.xamarin.com/show_bug.cgi?id=33191.