Bug 29692 - Mono 4.0.1 SIGSEGV crash on Ubuntu 14.04 with libpthread.so.0
Summary: Mono 4.0.1 SIGSEGV crash on Ubuntu 14.04 with libpthread.so.0
Status: RESOLVED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2015-05-04 01:51 UTC by apollo_maverick
Modified: 2017-01-29 17:42 UTC (History)
8 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description apollo_maverick 2015-05-04 01:51:39 UTC
Repro code here: https://github.com/helios-io/DedicatedThreadPool/tree/dev/src/benchmark/Helios.DedicatedThreadPool.VsThreadpoolBenchmark


Envrironment:

Ubuntu 14.04
details: 3.16.0-31-generic #41~14.04.1-Ubuntu SMP Wed Feb 11 19:30:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Microsoft Azure Standard A2 virtual machine
Xamarin mono stable repository for Debian

Mono JIT compiler version 4.0.1 (tarball Tue Apr 28 11:47:58 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        LLVM:          supported, not enabled.
        GC:            sgen


Output from Azure Standard A2 virtual machine:

Comparing Helios.Concurrency.DedicatedThreadPool vs System.Threading.ThreadPool for 10000 items
DedicatedThreadFiber.NumThreads: 2
System.Threading.ThreadPool
00:00:00.0110000
Helios.Concurrency.DedicatedThreadPool
00:00:00.0070000
Comparing Helios.Concurrency.DedicatedThreadPool vs System.Threading.ThreadPool for 100000 items
DedicatedThreadFiber.NumThreads: 2
System.Threading.ThreadPool
00:00:00.1200000
Helios.Concurrency.DedicatedThreadPool
00:00:00.0610000
Comparing Helios.Concurrency.DedicatedThreadPool vs System.Threading.ThreadPool for 1000000 items
DedicatedThreadFiber.NumThreads: 2
System.Threading.ThreadPool
00:00:01.2380000
Helios.Concurrency.DedicatedThreadPool
Stacktrace:


Native stacktrace:

        mono() [0x4b1d6c]
        mono() [0x50833e]
        mono() [0x428bfd]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd6da0d9340]
        mono() [0x4aedd0]
        mono() [0x4af13e]
        mono() [0x4b2ad7]
        mono() [0x5076bd]
        mono() [0x50775e]
        [0x41dc9bfc]

Debug info from gdb:


=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================
Comment 1 apollo_maverick 2015-05-05 03:35:21 UTC
Comparing Helios.Concurrency.DedicatedThreadPool vs System.Threading.ThreadPool for 10000 items
DedicatedThreadFiber.NumThreads: 2
System.Threading.ThreadPool
00:00:00.0110000
Helios.Concurrency.DedicatedThreadPool
00:00:00.0090000
Comparing Helios.Concurrency.DedicatedThreadPool vs System.Threading.ThreadPool for 100000 items
DedicatedThreadFiber.NumThreads: 2
System.Threading.ThreadPool
00:00:00.1280000
Helios.Concurrency.DedicatedThreadPool
00:00:00.0820000
Comparing Helios.Concurrency.DedicatedThreadPool vs System.Threading.ThreadPool for 1000000 items
DedicatedThreadFiber.NumThreads: 2
System.Threading.ThreadPool
00:00:01.2030000
Helios.Concurrency.DedicatedThreadPool

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at Helios.Concurrency.DedicatedThreadPool+PoolWorker.Failover (Boolean errorRecovery) [0x00000] in <filename unknown>:0 
  at Helios.Concurrency.DedicatedThreadPool+PoolWorker.<.ctor>b__4_0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.CancellationTokenSource.Dispose () [0x00000] in <filename unknown>:0 
  at System.Collections.Concurrent.BlockingCollection`1+<GetConsumingEnumerable>c__Iterator0[System.Action].<>__Finally0 () [0x00000] in <filename unknown>:0 
  at System.Collections.Concurrent.BlockingCollection`1+<GetConsumingEnumerable>c__Iterator0[System.Action].Dispose () [0x00000] in <filename unknown>:0 
  at Helios.Concurrency.DedicatedThreadPool+PoolWorker.<.ctor>b__4_0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.CancellationTokenSource.Dispose () [0x00000] in <filename unknown>:0 
  at System.Collections.Concurrent.BlockingCollection`1+<GetConsumingEnumerable>c__Iterator0[System.Action].<>__Finally0 () [0x00000] in <filename unknown>:0 
  at System.Collections.Concurrent.BlockingCollection`1+<GetConsumingEnumerable>c__Iterator0[System.Action].Dispose () [0x00000] in <filename unknown>:0 
  at Helios.Concurrency.DedicatedThreadPool+PoolWorker.<.ctor>b__4_0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
Comment 2 Christian Hüning 2015-05-22 04:38:35 UTC
Got the same issue here.

I also make heavy use of thread pools (parallel.ForEach etc.) in my application and it crashes repeatably with the above mentioned error.

Running Mono 4.0.1 amd64 on Debian Linux, Kernel 3.13.0-53-generic.
Comment 3 Christian Hüning 2015-05-22 05:08:13 UTC
whoops, seems to be fixed for me. Just did an update to the latest build (4.0.1 (tarball Tue May 12 15:39:23 UTC 2015) and it's working now.
Comment 4 Josip Bakic 2015-05-25 09:42:34 UTC
Same issue. In the course of testing my project I run a lot of Tasks. The SIGSEGV happens only sometimes, and in different tests. It is unpredictable. Never happened pre-4.0 (although pre-4.0 versions had the tendency to keep creating new threads for the tasks, which made running the tests in the debugger impossibly slow), and still happens for me with the latest version out of the official PPA. Ubuntu 14.04.

Even stranger is that sometimes I get a NullRef exception from a piece of code where such an exception should be impossible. It's as if a ThreadStatic field lost its value and is suddenly null. However, I have not seen this bug occur since the 4.1 upgrade, so that may be resolved.
Comment 5 Christian Hüning 2015-05-27 09:01:40 UTC
Update here: It's not fixed with the build from 12th May. It just doesn't occur that often anymore. :(
Comment 6 Christian Hüning 2015-05-27 09:21:20 UTC
Which 4.1 update? can't find any Mono 4.1 update, except in the nightly builds (which already are at 4.3)
Comment 7 Josip Bakic 2015-05-27 09:25:17 UTC
My apologies, a typo - I meant to say 4.0.1, the latest official release (I presume). I am using the PPA recommended for Ubuntu on the Mono Project homepage, and have not installed anything manually.
Comment 8 Christian Hüning 2015-05-27 09:46:23 UTC
Well I am still getting the error, just double checked. 

In this run it's been the NullRef Exception mentioned earlier by you: 

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00000] in <filename unknown>:0
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0


I also tried the latest nightly Mono 4.3.0 and 4.1.0. It still seems to be present there.
Comment 9 Christian Hüning 2015-06-07 11:46:29 UTC
Any updates on this one? Actually at least for my part this is a pretty urgent issue! Usage of the Parallel-API is quite essential!
Comment 10 Christian Hüning 2015-06-07 11:46:52 UTC
Any updates on this one? Actually at least for my part this is a pretty urgent issue! Usage of the Parallel-API is quite essential!
Comment 11 Josip Bakic 2015-06-19 17:38:14 UTC
I am no longer able to reproduce the issue. Since it occurs sporadically, it's hard for me to say when it went away (or even if it is really gone), but I have not seen it for some time now, couple of weeks maybe. Everything is running smoothly.

Mono JIT compiler version 4.0.1 (tarball Thu May 28 09:08:28 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen
Comment 12 Christian Hüning 2015-06-20 03:05:04 UTC
Ok, you're using a newer build as it seems. I will give it a shot later today! Thanks for answering!
Comment 13 Christian Hüning 2015-06-24 06:19:54 UTC
Ok , the 4.0.1 build from May 28 solves this!
Comment 14 Taloth Saldono 2015-07-16 15:17:15 UTC
No guys, probably not fixed in mono. But a kernel upgrade (3.13.0-54 and 3.16.0-40).
See https://bugzilla.xamarin.com/show_bug.cgi?id=29212.

Seems like it's reoccurring in newer 4.1 kernels again coz they somehow decided to revert the whole thing https://github.com/torvalds/linux/commit/73459e2a1ada09a68c02cc5b73f3116fc8194b3d.
Comment 15 dtravin 2016-10-17 14:35:44 UTC
What kernel version should I have to have latest mono 4.6 working properly regarding this issue?
Comment 16 apollo_maverick 2017-01-29 17:41:38 UTC
the bug has been gone for a while tested on Mono 4.8 and latest snapshot