Bug 37288 - Deadlock issues in new thread pool system
Summary: Deadlock issues in new thread pool system
Status: RESOLVED ANSWERED
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: 4.2.0 (C6)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Ludovic Henry
URL:
Depends on:
Blocks:
 
Reported: 2015-12-25 23:51 UTC by Mark Laws
Modified: 2016-01-29 13:18 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
running with MONO_THREADS_PER_CPU=1 (21.89 KB, text/plain)
2016-01-05 22:00 UTC, Mark Laws
Details
running with MONO_THREADS_PER_CPU=8 (25.15 KB, text/plain)
2016-01-05 22:01 UTC, Mark Laws
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED ANSWERED

Description Mark Laws 2015-12-25 23:51:54 UTC
Mono 4.2.x appears to have a bug in which code that worked on <= 4.2.x is now susceptible to deadlock, presumably due to differences in behavior with the new thread pool system.

I have a Linux virtual machine with a single virtual CPU core assigned to it.  On Mono 4.0.4, the program worked fine, but it deadlocks 100% of the time on 4.2.1 and 4.2.2.  Increasing the minimum worker thread count with ThreadPool.SetMinThreads does not help; however, running the program with MONO_THREADS_PER_CPU=2 set in the environment enables the program to run a bit further, and larger values (I've been testing with 8 for this case) resolve the issue entirely.

It is, of course, undesirable that (potentially non-technical) users should have to modify runtime settings in order to make things work, especially since a system upgrade that brings them onto Mono 4.2.x has the potential to break programs that have not been upgraded.  Furthermore, this problem does not appear to occur on .NET.

To reproduce the issue, build:

https://github.com/fsharp/FsAutoComplete

Then, on a system/VM with only a single core (because Mono defaults to MONO_THREADS_PER_CPU=1), run fsautocomplete.exe, and try typing "help" followed by return into it.  You should get nothing in response, and furthermore, you should find that the program cannot be terminated via ^C.  Try running it again with MONO_THREADS_PER_CPU=2, and you should get an error message in reply to "help" (this is what we would expect to see).

For reference, I initially reported the issue at https://github.com/fsharp/FsAutoComplete/issues/88 before determining it was a Mono bug.
Comment 1 Ludovic Henry 2016-01-04 17:04:35 UTC
Hello Mark,

Thank you for such a detailed report! I will look into this issue right away.

Thank you very much,
Happy new year,

Ludovic
Comment 2 Ludovic Henry 2016-01-04 17:37:13 UTC
I looked at your issue at https://github.com/fsharp/FsAutoComplete/issues/88 and you said having MONO_LOG_LEVEL=debug helped you. Could you please send me the output of that? Thank you
Comment 3 Mark Laws 2016-01-05 22:00:49 UTC
Created attachment 14452 [details]
running with MONO_THREADS_PER_CPU=1

I guess bugzilla simply ate my mail with the logs attached.  Sorry for the delay.  Thank you for looking into this!
Comment 4 Mark Laws 2016-01-05 22:01:26 UTC
Created attachment 14453 [details]
running with MONO_THREADS_PER_CPU=8
Comment 5 Ludovic Henry 2016-01-25 15:15:06 UTC
Hi Mark,

Sorry to come back to you so late. I think it's highly unlikely we will backport a fix for this issue to 4.2, as it would be a pretty invasive change to the threadpool heuristic.

If you wouldn't mind retrying setting `ThreadPool.SetMinThreads` at the beginning of your program, that should fix it. We had a bug in there that made it inefficient but it's been fixed in mono 4.2.2, and you can download it from http://download.mono-project.com/archive/4.2.2/macos-10-x86/MonoFramework-MDK-4.2.2.30.macos10.xamarin.x86.pkg

Thank you!
Comment 6 Mark Laws 2016-01-27 11:00:27 UTC
Is this a patch release of Mono 4.2.2?  I already previously tried on some version of 4.2.2.
Comment 7 Ludovic Henry 2016-01-27 11:40:48 UTC
Yes it's the Service Release 1, so the link above will contain the appropriate fix to `ThreadPool.SetMinThreads`
Comment 8 Mark Laws 2016-01-27 11:41:55 UTC
OK, I'll try it.  Thanks!
Comment 9 Mark Laws 2016-01-28 06:45:46 UTC
I can confirm that this version:

Mono JIT compiler version 4.2.2 (Stable 4.2.2.30/996df3c Wed Jan 27 20:19:41 UTC 2016)

fixes the issue when using ThreadPool.SetMinThreads (no need for MONO_THREADS_PER_CPU)!