Bug 39948 - Crashed while GCing.
Summary: Crashed while GCing.
Status: RESOLVED NORESPONSE
Alias: None
Product: Runtime
Classification: Mono
Component: GC ()
Version: 4.2.0 (C6)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Kevin.Kline
URL:
: 39932 ()
Depends on:
Blocks:
 
Reported: 2016-03-28 14:20 UTC by Kevin.Kline
Modified: 2016-11-04 19:37 UTC (History)
7 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NORESPONSE

Description Kevin.Kline 2016-03-28 14:20:07 UTC
#Initializing KAE internal system resource factory...
	#Initializing network resources...
		#Initialized network resource.	(P: Metadata,	URL: tcp://192.168.38.130:8081)
		#Initialized network resource.	(P: INTERNAL_SPECIAL_RESOURCE,	URL: tcp://192.168.38.130:6679)
	#Loading KPPs...
KAE Agent Started!
[New Thread 0x7fff973fe700 (LWP 82307)]
[New Thread 0x7fff96dfb700 (LWP 82310)]
[New Thread 0x7fff975ff700 (LWP 82306)]
[New Thread 0x7fff971fd700 (LWP 82308)]
[New Thread 0x7fff96ffc700 (LWP 82309)]
[New Thread 0x7fff96bfa700 (LWP 82311)]
[New Thread 0x7fff969f9700 (LWP 82312)]
[New Thread 0x7fff967f8700 (LWP 82313)]
[New Thread 0x7fff965f7700 (LWP 82314)]
[New Thread 0x7fff963f6700 (LWP 82315)]
[New Thread 0x7fff961f5700 (LWP 82316)]
[New Thread 0x7fff95ff4700 (LWP 82317)]
[New Thread 0x7fff95df3700 (LWP 82318)]
[New Thread 0x7fff95bf2700 (LWP 82319)]
[New Thread 0x7fff959f1700 (LWP 82320)]
[New Thread 0x7fff957f0700 (LWP 82321)]
[New Thread 0x7fff955ef700 (LWP 82322)]
[New Thread 0x7fff953ee700 (LWP 82323)]
* Assertion: should not be reached at sgen-scan-object.h:101


Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff95ff4700 (LWP 82317)]
0x00007ffff71255f7 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff71255f7 in raise () from /lib64/libc.so.6
#1  0x00007ffff7126ce8 in abort () from /lib64/libc.so.6
#2  0x0000000000627ea6 in monoeg_log_default_handler (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, 
    message=0x7fff60008980 "* Assertion: should not be reached at sgen-scan-object.h:101\n", unused_data=0x0) at goutput.c:233
#3  0x0000000000627c93 in monoeg_g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, 
    format=0x6339f0 "* Assertion: should not be reached at %s:%d\n", args=0x7fff95ff2428) at goutput.c:113
#4  0x0000000000627df4 in monoeg_assertion_message (format=0x6339f0 "* Assertion: should not be reached at %s:%d\n") at goutput.c:133
#5  0x00000000005f4476 in simple_nursery_serial_scan_object (object=0x7ffff0467058, desc=<optimized out>, queue=0x964920 <gray_queue>)
    at sgen-scan-object.h:101
#6  0x00000000005e281a in sgen_drain_gray_stack (max_objs=max_objs@entry=-1, ctx=...) at sgen-gc.c:563
#7  0x00000000005e409e in collect_nursery (unpin_queue=unpin_queue@entry=0x0, finish_up_concurrent_mark=finish_up_concurrent_mark@entry=0)
    at sgen-gc.c:1590
#8  0x00000000005e606f in sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x6d83f5 "Nursery full", wait_to_finish=0)
    at sgen-gc.c:2367
#9  0x00000000005d9b2a in sgen_alloc_obj_nolock (vtable=vtable@entry=0x13f3480, size=size@entry=24) at sgen-alloc.c:291
#10 0x00000000005d9e13 in sgen_alloc_obj (vtable=0x13f3480, size=24) at sgen-alloc.c:457
#11 0x00000000005c7cb9 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:933
#12 0x0000000040018978 in ?? ()
#13 0x00007ffff07c0d00 in ?? ()
#14 0x00007fff95ff46e8 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb) p mono_pmip (0x0000000040018978)
$1 = 0x7fff600371d0 " (wrapper managed-to-native) object:__icall_wrapper_mono_gc_alloc_obj (intptr,intptr) + 0x58 (0x40018920 0x400189a9) [0x9ad0a0 - KAEAgent.exe]"
(gdb) p mono_pmip (0x00007ffff07c0d00)
$2 = 0x0
(gdb) p mono_pmip (0x00007fff95ff46e8)
$3 = 0x0
(gdb) p mono_pmip (0x00007fff95ff46e8)




mono version 4.2.0 (stable)
OS: CentOS 7.
Comment 1 Alexander Köplinger [MSFT] 2016-04-05 01:10:30 UTC
*** Bug 39932 has been marked as a duplicate of this bug. ***
Comment 2 Alexander Köplinger [MSFT] 2016-04-05 01:11:20 UTC
It would be super helpful to get a sample app/code which shows the problem. Thank you!
Comment 3 Eric Roller 2016-06-08 16:38:52 UTC
I think I am running into the same issue using mono version 4.2.3. I'll see if it is a frequent occurrence and send along an example program if possible.

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_obj (intptr,intptr) <0xffffffff>
  at (wrapper alloc) object.AllocSmall (intptr,intptr) <0xffffffff>
  at string.System.Collections.Generic.IEnumerable<char>.GetEnumerator () <0x00018>
  at System.Linq.Enumerable.FirstOrDefault<char> (System.Collections.Generic.IEnumerable`1<char>) <0x0011d>
  at SequencingFiles.FastaLoader.TryAdvanceStreamToSequence (string) <0x00073>
  at SequencingFiles.FastaLoader.AdvanceStreamToSequence (string) <0x00017>
  at SequencingFiles.FastaLoader.LoadFastaSequence (string) <0x00017>
  at SequencingFiles.FastaLoader.LoadFastaSequence (string,string) <0x000cb>
  at CanvasBin.CanvasBin.InitializeAlignmentArrays (string,string,CanvasCommon.CanvasCoverageMode,System.Collections.Generic.IDictionary`2<string, System.Collections.BitArray>,System.Collections.Generic.IDictionary`2<string, CanvasBin.HitArray>,System.Collections.Generic.IDictionary`2<string, int16[]>) <0x0003b>
  at CanvasBin.CanvasBin.BinOneGenomicInterval (CanvasBin.CanvasBinParameters,System.Collections.Generic.Dictionary`2<string, System.Collections.BitArray>,System.Collections.Generic.Dictionary`2<string, CanvasBin.HitArray>,System.Collections.Generic.Dictionary`2<string, int16[]>) <0x0004b>
  at CanvasBin.CanvasBin.Run (CanvasBin.CanvasBinParameters) <0x00233>
  at CanvasBin.Program.Main (string[]) <0x000e3>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_int_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x4935d5]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x4ed14a]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x41425f]
        /lib64/libpthread.so.0(+0xf7e0) [0x2aaaab36d7e0]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5f4bf2]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5e266a]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5e4ad7]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5e60ef]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5d979e]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5d9894]
        /illumina/development/Isas/packages/mono-4.2.3/bin/mono() [0x5ca639]
        [0x40017aa8]

Debug info from gdb:

Mono support loaded.
[New LWP 21820]
[New LWP 21813]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00002aaaab36d37d in waitpid () from /lib64/libpthread.so.0
  Id   Target Id         Frame
  3    Thread 0x2aaab1e00700 (LWP 21813) "mono" 0x00002aaaab36968c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x2aaab3adf700 (LWP 21820) "Finalizer" 0x00002aaaab36ba00 in sem_wait () from /lib64/libpthread.so.0
* 1    Thread 0x2aaaab910b80 (LWP 21810) "mono" 0x00002aaaab36d37d in waitpid () from /lib64/libpthread.so.0

Thread 3 (Thread 0x2aaab1e00700 (LWP 21813)):
#0  0x00002aaaab36968c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005f793c in thread_func (thread_data=0x0) at sgen-thread-pool.c:118
#2  0x00002aaaab365aa1 in start_thread () from /lib64/libpthread.so.0
#3  0x00002aaaab66493d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x2aaab3adf700 (LWP 21820)):
#0  0x00002aaaab36ba00 in sem_wait () from /lib64/libpthread.so.0
#1  0x000000000061afe8 in mono_sem_wait (sem=0x949cc0 <finalizer_sem>, alertable=1) at mono-semaphore.c:107
#2  0x00000000005a1e92 in finalizer_thread (unused=<optimized out>) at gc.c:1096
#3  0x000000000058008f in start_wrapper_internal (data=<optimized out>) at threads.c:725
#4  start_wrapper (data=<optimized out>) at threads.c:772
#5  0x00000000006223d0 in inner_start_thread (arg=0x7fffffffc8e0) at mono-threads-posix.c:97
#6  0x00002aaaab365aa1 in start_thread () from /lib64/libpthread.so.0
#7  0x00002aaaab66493d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2aaaab910b80 (LWP 21810)):
#0  0x00002aaaab36d37d in waitpid () from /lib64/libpthread.so.0
#1  0x0000000000493668 in mono_handle_native_sigsegv (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2193
#2  0x00000000004ed14a in mono_arch_handle_altstack_exception (sigctx=0x2aaaaabb7ac0, siginfo=0x2aaaaabb7bf0, fault_addr=<optimized out>, stack_ovf=0) at exceptions-amd64.c:805
#3  0x000000000041425f in mono_sigsegv_signal_handler (_dummy=11, _info=0x2aaaaabb7bf0, context=0x2aaaaabb7ac0) at mini-runtime.c:2471
#4  <signal handler called>
#5  0x00000000005f4bf2 in simple_nursery_serial_scan_object (object=<optimized out>, desc=<optimized out>, queue=0x95b8a0 <gray_queue>) at sgen-scan-object.h:82
#6  0x00000000005e266a in sgen_drain_gray_stack (max_objs=-1, ctx=...) at sgen-gc.c:563
#7  0x00000000005e4ad7 in collect_nursery (unpin_queue=0x0, finish_up_concurrent_mark=<optimized out>) at sgen-gc.c:1592
#8  0x00000000005e60ef in sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x6d908e "Nursery full", wait_to_finish=0) at sgen-gc.c:2369
#9  0x00000000005d979e in sgen_alloc_obj_nolock (vtable=0xbcfb58, size=32) at sgen-alloc.c:291
#10 0x00000000005d9894 in sgen_alloc_obj (vtable=0xbcfb58, size=32) at sgen-alloc.c:457
#11 0x00000000005ca639 in mono_gc_alloc_obj (vtable=<optimized out>, size=<optimized out>) at sgen-mono.c:936
#12 0x0000000040017aa8 in ?? ()
#13 0x00002aaab1800238 in ?? ()
#14 0x00002aaaab910b58 in ?? ()
#15 0x00002aaab1800190 in ?? ()
#16 0x0000000000bcfb58 in ?? ()
#17 0x00002aaab1800258 in ?? ()
#18 0x00000000009c9560 in ?? ()
#19 0x0000000000a081a0 in ?? ()
#20 0x00007fffffffc400 in ?? ()
#21 0x00007fffffffc2f0 in ?? ()
#22 0x0000000040013234 in ?? ()
#23 0x00002aaab1800190 in ?? ()
#24 0x00002aaab1808948 in ?? ()
#25 0x0000000000000000 in ?? ()
Comment 4 Alexander Köplinger [MSFT] 2016-06-16 14:50:26 UTC
You can also try upgrading to the new stable Mono 4.4 version that was recently released.
Comment 5 Alex Rønne Petersen 2016-08-29 09:14:36 UTC
It is very difficult (next to impossible) to debug GC crashes without a repro. Since there hasn't been any activity on this bug for a while, I'm going to close it as NORESPONSE. If anyone can provide a repro for this issue, please feel free to attach it and reopen. Thanks!
Comment 6 Ben Rush 2016-11-04 19:37:18 UTC
So I'm seeing this as well, but unfortunately it's for a very complex program I'm working on (that requires the installation of many different software packages to work - it's a weather modelling application)  

I do have the source code publicly available at : https://github.com/kwende/wrfsharp. Even though it would take you quite a while to set it up and run it. 

The application automates a number of external processes, parses their output, and so forth. Periodically I get this crash (FYI, I also posted something on stack overflow here [http://stackoverflow.com/questions/40252080/occasional-crash-during-object-allocation-mono-4-6-1?noredirect=1#comment67767308_40252080] about it). 

Here is the stack trace: 

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0x0005d>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <0x00143>
  at string.SplitInternal (char[],int,System.StringSplitOptions) <0x00081>
  at string.Split (char[]) <0x00018>
  at Mono.Unix.UnixPath.GetPathComponents (string,string[]&,int&) <0x00060>
  at Mono.Unix.UnixPath.GetCanonicalPath (string) <0x00033>
  at Mono.Unix.UnixPath.GetFullPath (string) <0x0001b>
  at Mono.Unix.UnixFileSystemInfo..ctor (string) <0x00053>
  at Mono.Unix.UnixSymbolicLinkInfo..ctor (string) <0x00013>
  at Mono.Unix.UnixFileSystemInfo.CreateSymbolicLink (string) <0x00053>
  at WrfSharp.Runner.Implementations.FileSystem.CreateSymLink (string,string) <0x00073>
  at WrfSharp.Helpers.FileSystem.FileSystemHelper.CreateMetEmSymlinksInRealDirectory (WrfSharp.DataStructures.WrfConfigura                                                   tion,WrfSharp.Interfaces.IFileSystem) <0x001db>
  at WrfSharp.Runner.Program.PrepStage (WrfSharp.Interfaces.IFileSystem,WrfSharp.Interfaces.INetwork,WrfSharp.Interfaces.I                                                   Logger,WrfSharp.Interfaces.IProcessLauncher,WrfSharp.DataStructures.WrfConfiguration) <0x005d3>
  at WrfSharp.Runner.Program.Main (string[]) <0x00217>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0x000c8>

Native stacktrace:

        mono() [0x4accca]
        mono() [0x5040fe]
        mono() [0x4275d7]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x113e0) [0x7f1702bfd3e0]
        mono() [0x603d29]
        mono() [0x60789b]
        mono() [0x607f5c]
        mono() [0x5fb80d]
        mono() [0x5e8ea1]
        [0x409696de]

Debug info from gdb:

[New LWP 16709]
[New LWP 16710]
[New LWP 16711]
[New LWP 16715]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f1702bfcfcb in __waitpid (pid=17313, stat_loc=0x7f170362711c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29      ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id         Frame
* 1    Thread 0x7f170374f740 (LWP 16708) "Main" 0x00007f1702bfcfcb in __waitpid (pid=17313, stat_loc=0x7f170362711c, optio                                                   ns=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  2    Thread 0x7f1701bff700 (LWP 16709) "SGen worker" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86                                                   _64/pthread_cond_wait.S:185
  3    Thread 0x7f170232b700 (LWP 16710) "Finalizer" 0x00007f1702bfb877 in futex_abstimed_wait_cancelable (private=0, abst                                                   ime=0x0, expected=0, futex_word=0x97c100) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7f16ff0ff700 (LWP 16711) "Timer-Scheduler" 0x00007f1702639826 in __GI___sigsuspend (set=0x97c640) at ../sy                                                   sdeps/unix/sysv/linux/sigsuspend.c:30
  5    Thread 0x7f16ff252700 (LWP 16715) "Threadpool work" 0x00007f17026feb5d in poll () at ../sysdeps/unix/syscall-templa                                                   te.S:84

Thread 5 (Thread 0x7f16ff252700 (LWP 16715)):
#0  0x00007f17026feb5d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00000000005a2b9f in ?? ()
#2  0x00000000005a3bb3 in ?? ()
#3  0x000000000059c68c in ?? ()
#4  0x0000000000653b4c in ?? ()
#5  0x00007f1702bf370a in start_thread (arg=0x7f16ff252700) at pthread_create.c:333
#6  0x00007f170270a82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f16ff0ff700 (LWP 16711)):
#0  0x00007f1702639826 in __GI___sigsuspend (set=0x97c640) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000005e71e4 in ?? ()
#2  0x00000000005e742d in ?? ()
#3  <signal handler called>
#4  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#5  0x0000000000629f41 in ?? ()
#6  0x000000000063fca4 in ?? ()
#7  0x000000000059c3c7 in ?? ()
#8  0x000000000059d686 in ?? ()
#9  0x00000000409b9168 in ?? ()
#10 0x00000000017ae948 in ?? ()
#11 0xffffffffffffffff in ?? ()
#12 0x7fffffffffffffff in ?? ()
#13 0x00007f1701e99478 in ?? ()
#14 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f170232b700 (LWP 16710)):
#0  0x00007f1702bfb877 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x97c100) at ../s                                                   ysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x97c100, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f1702bfb924 in __new_sem_wait_slow (sem=0x97c100, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f1702bfb9ca in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00000000005bb8f9 in ?? ()
#5  0x000000000059c68c in ?? ()
#6  0x0000000000653b4c in ?? ()
#7  0x00007f1702bf370a in start_thread (arg=0x7f170232b700) at pthread_create.c:333
#8  0x00007f170270a82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f1701bff700 (LWP 16709)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000623a27 in ?? ()
#2  0x00007f1702bf370a in start_thread (arg=0x7f1701bff700) at pthread_create.c:333
#3  0x00007f170270a82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f170374f740 (LWP 16708)):
#0  0x00007f1702bfcfcb in __waitpid (pid=17313, stat_loc=0x7f170362711c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.                                                   c:29
#1  0x00000000004acd59 in ?? ()
#2  0x00000000005040fe in ?? ()
#3  0x00000000004275d7 in ?? ()
#4  <signal handler called>
#5  0x0000000000603d29 in ?? ()
#6  0x000000000060789b in ?? ()
#7  0x0000000000607f5c in ?? ()
#8  0x00000000005fb80d in ?? ()
#9  0x00000000005e8ea1 in ?? ()
#10 0x00000000409696de in ?? ()
#11 0x00007f1701f3c9d0 in ?? ()
#12 0x00000000013de1d0 in ?? ()
#13 0x0000000000000130 in ?? ()
#14 0x00000000014975a8 in ?? ()
#15 0x00000000013ddcb0 in ?? ()
#16 0x0000000001406b00 in ?? ()
#17 0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted (core dumped)
root@vortex:~/wrf/wrfsharp# mono -V
Mono JIT compiler version 4.6.1 (Stable 4.6.1.5/ef43c15 Wed Oct 12 09:10:37 UTC 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen