Bug 46511 - Random crash - presumably bug in GC - when running mono app
Summary: Random crash - presumably bug in GC - when running mono app
Status: RESOLVED INVALID
Alias: None
Product: Runtime
Classification: Mono
Component: GC ()
Version: 4.6.0 (C8)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-11-04 19:44 UTC by Ben Rush
Modified: 2018-05-05 07:47 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED INVALID

Description Ben Rush 2016-11-04 19:44:00 UTC
I posted a question about this on stack overflow the other day. The person who responded told me that, if I could get this reproducible that I should open up a bug on bugzilla. I can get this to consistently occur with my application, however the application is quite complex and requires quite a bit of installation of third party applications to get it running (it's actually a weather modelling application). Still, when asking the application to run for a long time I get 100% duplication of this bug. 

My source is available here: https://github.com/kwende/wrfsharp. 

My setup is: 

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:        16.04
Codename:       xenial

Mono JIT compiler version 4.6.1 (Stable 4.6.1.5/ef43c15 Wed Oct 12 09:10:37 UTC 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

======================
The crash, which I'm quite convinced is GC-related, leaves the following stack trace (or something very near it, usually during object allocation). Now I understand that getting my application to run on your machines might be quite time consuming, so if there's a debug version of *whatever* I can run to help you guys track this down please have me do so, I'm quite capable of helping you dig into this if needed. At the moment it's becoming quite a roadblock for me! 

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <0x0005d>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <0x00143>
  at string.SplitInternal (char[],int,System.StringSplitOptions) <0x00081>
  at string.Split (char[]) <0x00018>
  at Mono.Unix.UnixPath.GetPathComponents (string,string[]&,int&) <0x00060>
  at Mono.Unix.UnixPath.GetCanonicalPath (string) <0x00033>
  at Mono.Unix.UnixPath.GetFullPath (string) <0x0001b>
  at Mono.Unix.UnixFileSystemInfo..ctor (string) <0x00053>
  at Mono.Unix.UnixSymbolicLinkInfo..ctor (string) <0x00013>
  at Mono.Unix.UnixFileSystemInfo.CreateSymbolicLink (string) <0x00053>
  at WrfSharp.Runner.Implementations.FileSystem.CreateSymLink (string,string) <0x00073>
  at WrfSharp.Helpers.FileSystem.FileSystemHelper.CreateMetEmSymlinksInRealDirectory (WrfSharp.DataStructures.WrfConfigura                                                   tion,WrfSharp.Interfaces.IFileSystem) <0x001db>
  at WrfSharp.Runner.Program.PrepStage (WrfSharp.Interfaces.IFileSystem,WrfSharp.Interfaces.INetwork,WrfSharp.Interfaces.I                                                   Logger,WrfSharp.Interfaces.IProcessLauncher,WrfSharp.DataStructures.WrfConfiguration) <0x005d3>
  at WrfSharp.Runner.Program.Main (string[]) <0x00217>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0x000c8>

Native stacktrace:

        mono() [0x4accca]
        mono() [0x5040fe]
        mono() [0x4275d7]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x113e0) [0x7f1702bfd3e0]
        mono() [0x603d29]
        mono() [0x60789b]
        mono() [0x607f5c]
        mono() [0x5fb80d]
        mono() [0x5e8ea1]
        [0x409696de]

Debug info from gdb:

[New LWP 16709]
[New LWP 16710]
[New LWP 16711]
[New LWP 16715]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f1702bfcfcb in __waitpid (pid=17313, stat_loc=0x7f170362711c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29      ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id         Frame
* 1    Thread 0x7f170374f740 (LWP 16708) "Main" 0x00007f1702bfcfcb in __waitpid (pid=17313, stat_loc=0x7f170362711c, optio                                                   ns=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  2    Thread 0x7f1701bff700 (LWP 16709) "SGen worker" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86                                                   _64/pthread_cond_wait.S:185
  3    Thread 0x7f170232b700 (LWP 16710) "Finalizer" 0x00007f1702bfb877 in futex_abstimed_wait_cancelable (private=0, abst                                                   ime=0x0, expected=0, futex_word=0x97c100) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7f16ff0ff700 (LWP 16711) "Timer-Scheduler" 0x00007f1702639826 in __GI___sigsuspend (set=0x97c640) at ../sy                                                   sdeps/unix/sysv/linux/sigsuspend.c:30
  5    Thread 0x7f16ff252700 (LWP 16715) "Threadpool work" 0x00007f17026feb5d in poll () at ../sysdeps/unix/syscall-templa                                                   te.S:84

Thread 5 (Thread 0x7f16ff252700 (LWP 16715)):
#0  0x00007f17026feb5d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x00000000005a2b9f in ?? ()
#2  0x00000000005a3bb3 in ?? ()
#3  0x000000000059c68c in ?? ()
#4  0x0000000000653b4c in ?? ()
#5  0x00007f1702bf370a in start_thread (arg=0x7f16ff252700) at pthread_create.c:333
#6  0x00007f170270a82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f16ff0ff700 (LWP 16711)):
#0  0x00007f1702639826 in __GI___sigsuspend (set=0x97c640) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000005e71e4 in ?? ()
#2  0x00000000005e742d in ?? ()
#3  <signal handler called>
#4  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#5  0x0000000000629f41 in ?? ()
#6  0x000000000063fca4 in ?? ()
#7  0x000000000059c3c7 in ?? ()
#8  0x000000000059d686 in ?? ()
#9  0x00000000409b9168 in ?? ()
#10 0x00000000017ae948 in ?? ()
#11 0xffffffffffffffff in ?? ()
#12 0x7fffffffffffffff in ?? ()
#13 0x00007f1701e99478 in ?? ()
#14 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f170232b700 (LWP 16710)):
#0  0x00007f1702bfb877 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x97c100) at ../s                                                   ysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x97c100, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f1702bfb924 in __new_sem_wait_slow (sem=0x97c100, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f1702bfb9ca in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29
#4  0x00000000005bb8f9 in ?? ()
#5  0x000000000059c68c in ?? ()
#6  0x0000000000653b4c in ?? ()
#7  0x00007f1702bf370a in start_thread (arg=0x7f170232b700) at pthread_create.c:333
#8  0x00007f170270a82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f1701bff700 (LWP 16709)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000623a27 in ?? ()
#2  0x00007f1702bf370a in start_thread (arg=0x7f1701bff700) at pthread_create.c:333
#3  0x00007f170270a82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f170374f740 (LWP 16708)):
#0  0x00007f1702bfcfcb in __waitpid (pid=17313, stat_loc=0x7f170362711c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.                                                   c:29
#1  0x00000000004acd59 in ?? ()
#2  0x00000000005040fe in ?? ()
#3  0x00000000004275d7 in ?? ()
#4  <signal handler called>
#5  0x0000000000603d29 in ?? ()
#6  0x000000000060789b in ?? ()
#7  0x0000000000607f5c in ?? ()
#8  0x00000000005fb80d in ?? ()
#9  0x00000000005e8ea1 in ?? ()
#10 0x00000000409696de in ?? ()
#11 0x00007f1701f3c9d0 in ?? ()
#12 0x00000000013de1d0 in ?? ()
#13 0x0000000000000130 in ?? ()
#14 0x00000000014975a8 in ?? ()
#15 0x00000000013ddcb0 in ?? ()
#16 0x0000000001406b00 in ?? ()
#17 0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted (core dumped)
Comment 2 Vlad Brezae 2016-11-07 16:00:48 UTC
Hello Ben,

      The crash site is not very helpful. Could you install mono-runtime-dbg and let us know when you get a stacktrace that has debug symbols included ?

Vlad
Comment 3 Ben Rush 2016-11-11 03:55:38 UTC
I removed a bunch of Console.Writeline() from my application which dramatically reduced the frequency of the crashes; however it just now happened. I ran my application with 

mono --debug myapp.exe

and this is the stacktrace I got (after installing the mono debug runtime as you suggested, let me know if you need me to do anything else): 

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_gc_alloc_vector (intptr,intptr,intptr) <IL 0x00009, 0x0005d>
  at (wrapper alloc) object.AllocVector (intptr,intptr) <IL 0x000b3, 0x00143>
  at string.SplitInternal (char[],int,System.StringSplitOptions) <0x00081>
  at string.Split (char[]) <0x00018>
  at WrfSharp.Helpers.Namelists.NamelistParser.ParseFromString (string) <IL 0x000af, 0x00376>
  at WrfSharp.Helpers.Namelists.NamelistHelper.UpdatePhysicsParameters (WrfSharp.DataStructures.WrfConfiguration,WrfSharp.DataStructures.PhysicsConfiguration,WrfSharp.Interfaces.IFileSystem) <IL 0x0000e, 0x00057>
  at WrfSharp.Runner.Program.ComputeStage (WrfSharp.Interfaces.IFileSystem,WrfSharp.Interfaces.ILogger,WrfSharp.Interfaces.IProcessLauncher,WrfSharp.Interfaces.IEnvironment,WrfSharp.DataStructures.WrfConfiguration,WrfSharp.DataStructures.PhysicsConfiguration) <IL 0x00010, 0x00067>
  at WrfSharp.Runner.Program.Main (string[]) <IL 0x000a8, 0x00293>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <IL 0x00051, 0x000c8>

Native stacktrace:

        mono() [0x4accca]
        mono() [0x5040fe]
        mono() [0x4275d7]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x113e0) [0x7fb584dff3e0]
        mono() [0x603d29]
        mono() [0x60789b]
        mono() [0x607f5c]
        mono() [0x5fb80d]
        mono() [0x5e8ea1]
        [0x41aec6de]

Debug info from gdb:

[New LWP 604]
[New LWP 605]
[New LWP 606]
[New LWP 611]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fb584dfefcb in __waitpid (pid=pid@entry=10903, stat_loc=stat_loc@entry=0x7fb58582711c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29      ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id         Frame
* 1    Thread 0x7fb585951740 (LWP 603) "Main" 0x00007fb584dfefcb in __waitpid (pid=pid@entry=10903, stat_loc=stat_loc@entry=0x7fb58582711c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  2    Thread 0x7fb583fff700 (LWP 604) "SGen worker" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7fb581c2d700 (LWP 605) "Finalizer" 0x00007fb584dfd877 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x97c100 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7fb5812ff700 (LWP 606) "Timer-Scheduler" 0x00007fb58483b826 in __GI___sigsuspend (set=set@entry=0x97c640 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
  5    Thread 0x7fb581454700 (LWP 611) "Threadpool work" 0x00007fb584900b5d in poll () at ../sysdeps/unix/syscall-template.S:84

Thread 5 (Thread 0x7fb581454700 (LWP 611)):
#0  0x00007fb584900b5d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000064c837 in mono_poll (ufds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at mono-poll.c:26
#2  0x00000000005a2b9f in poll_event_wait (callback=0x5a3610 <wait_callback>, user_data=0x7fb5640019b0) at threadpool-ms-io-poll.c:147
#3  0x00000000005a3bb3 in selector_thread (data=<optimized out>) at threadpool-ms-io.c:424
#4  selector_thread (data=data@entry=0x0) at threadpool-ms-io.c:309
#5  0x000000000059c68c in start_wrapper_internal (data=<optimized out>) at threads.c:740
#6  start_wrapper (data=<optimized out>) at threads.c:788
#7  0x0000000000653b4c in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:92
#8  0x00007fb584df570a in start_thread (arg=0x7fb581454700) at pthread_create.c:333
#9  0x00007fb58490c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7fb5812ff700 (LWP 606)):
#0  0x00007fb58483b826 in __GI___sigsuspend (set=set@entry=0x97c640 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000005e71e4 in suspend_thread (info=0x7fb5740008e0, context=context@entry=0x7fb5812fde00) at sgen-os-posix.c:98
#2  0x00000000005e742d in suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7fb5812fde00) at sgen-os-posix.c:125
#3  <signal handler called>
#4  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#5  0x0000000000629f41 in mono_os_cond_wait (mutex=0x21131e0, cond=0x2113208) at ../../mono/utils/mono-os-mutex.h:107
#6  mono_os_cond_timedwait (timeout_ms=4294967295, mutex=0x21131e0, cond=0x2113208) at ../../mono/utils/mono-os-mutex.h:122
#7  _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x409, timeout=timeout@entry=4294967295, alertable=alertable@entry=1, poll=poll@entry=0, alerted=alerted@entry=0x7fb5812fe4cc) at handles.c:1555
#8  0x000000000063fca4 in wapi_WaitForSingleObjectEx (handle=0x409, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at wait.c:181
#9  0x000000000059c3c7 in mono_wait_uninterrupted (numhandles=numhandles@entry=1, handles=handles@entry=0x7fb5812fe588, waitall=waitall@entry=0, ms=ms@entry=-1, error=error@entry=0x7fb5812fe590, thread=<optimized out>) at threads.c:1595
#10 0x000000000059d686 in ves_icall_System_Threading_WaitHandle_WaitOne_internal (handle=0x409, ms=-1) at threads.c:1731
#11 0x0000000041b3c118 in ?? ()
#12 0x0000000002500128 in ?? ()
#13 0xffffffffffffffff in ?? ()
#14 0x7fffffffffffffff in ?? ()
#15 0x00007fb58428cb28 in ?? ()
#16 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fb581c2d700 (LWP 605)):
#0  0x00007fb584dfd877 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x97c100 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x97c100 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007fb584dfd924 in __new_sem_wait_slow (sem=0x97c100 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007fb584dfd9ca in __new_sem_wait (sem=sem@entry=0x97c100 <finalizer_sem>) at sem_wait.c:29
#4  0x00000000005bb8f9 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x97c100 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:166
#5  mono_coop_sem_wait (sem=<optimized out>, flags=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:40
#6  finalizer_thread (unused=unused@entry=0x0) at gc.c:761
#7  0x000000000059c68c in start_wrapper_internal (data=<optimized out>) at threads.c:740
#8  start_wrapper (data=<optimized out>) at threads.c:788
#9  0x0000000000653b4c in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:92
#10 0x00007fb584df570a in start_thread (arg=0x7fb581c2d700) at pthread_create.c:333
#11 0x00007fb58490c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7fb583fff700 (LWP 604)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000623a27 in mono_os_cond_wait (cond=0x98b420 <work_cond>, mutex=<optimized out>) at ../../mono/utils/mono-os-mutex.h:107
#2  thread_func (thread_data=0x0) at sgen-thread-pool.c:110
#3  0x00007fb584df570a in start_thread (arg=0x7fb583fff700) at pthread_create.c:333
#4  0x00007fb58490c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7fb585951740 (LWP 603)):
#0  0x00007fb584dfefcb in __waitpid (pid=pid@entry=10903, stat_loc=stat_loc@entry=0x7fb58582711c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00000000004acd59 in mono_handle_native_sigsegv (signal=signal@entry=11, ctx=ctx@entry=0x7fb585827ac0, info=info@entry=0x7fb585827bf0) at mini-exceptions.c:2427
#2  0x00000000005040fe in mono_arch_handle_altstack_exception (sigctx=sigctx@entry=0x7fb585827ac0, siginfo=siginfo@entry=0x7fb585827bf0, fault_addr=<optimized out>, stack_ovf=stack_ovf@entry=0) at exceptions-amd64.c:795
#3  0x00000000004275d7 in mono_sigsegv_signal_handler (_dummy=11, _info=0x7fb585827bf0, context=0x7fb585827ac0) at mini-runtime.c:2865
#4  <signal handler called>
#5  0x0000000000603d29 in sgen_safe_object_get_size (obj=0x7fb5842cab48) at ../../mono/sgen/sgen-gc.h:716
#6  pin_objects_from_nursery_pin_queue (ctx=..., do_scan_objects=0) at sgen-gc.c:647
#7  pin_objects_in_nursery (do_scan_objects=do_scan_objects@entry=0, ctx=...) at sgen-gc.c:728
#8  0x000000000060789b in collect_nursery (reason=reason@entry=0x702f7d "Nursery full", is_overflow=is_overflow@entry=0, finish_up_concurrent_mark=0, unpin_queue=0x0) at sgen-gc.c:1570
#9  0x0000000000607f5c in major_start_concurrent_collection (reason=<optimized out>) at sgen-gc.c:2121
#10 sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x702f7d "Nursery full", wait_to_finish=0, stw=1) at sgen-gc.c:2277
#11 0x000000000060823c in sgen_ensure_free_space (size=<optimized out>, generation=<optimized out>) at sgen-gc.c:2232
#12 0x00000000005fb80d in sgen_alloc_obj_nolock (vtable=vtable@entry=0x21d0b48, size=size@entry=120) at sgen-alloc.c:262
#13 0x00000000005e8ea1 in mono_gc_alloc_vector (vtable=0x21d0b48, size=120, max_length=22) at sgen-mono.c:1744
#14 0x0000000041aec6de in ?? ()
#15 0x00007fb584138580 in ?? ()
#16 0x000000000211e330 in ?? ()
#17 0x0000000000000078 in ?? ()
#18 0x00000000021d0b48 in ?? ()
#19 0x000000000211de10 in ?? ()
#20 0x0000000002148d80 in ?? ()
#21 0x00007fff9ce80a78 in ?? ()
#22 0x00007fff9ce80b70 in ?? ()
#23 0x00007fff9ce80ac0 in ?? ()
#24 0x0000000041aeb804 in ?? ()
#25 0x00007fff9ce80b90 in ?? ()
#26 0x0000000000000016 in ?? ()
#27 0x00007fb5841385f8 in ?? ()
#28 0x00007fb584138510 in ?? ()
#29 0x00007fb585890c10 in ?? ()
#30 0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
Comment 4 Ben Rush 2016-11-12 01:56:35 UTC
Here is another crash analysis, this time from running it with mono-boehm. 

Native stacktrace:

        mono-boehm() [0x4b40aa]
        mono-boehm() [0x42ea3c]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x113e0) [0x7f34e680f3e0]
        mono-boehm(GC_mark_from+0x3f6) [0x62eee6]
        mono-boehm(GC_do_local_mark+0x2e) [0x62f52e]
        mono-boehm(GC_mark_local+0x148) [0x62f738]
        mono-boehm(GC_help_marker+0x88) [0x62f918]
        mono-boehm(GC_mark_thread+0x37) [0x635d57]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x770a) [0x7f34e680570a]
        /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f34e631c82d]

Debug info from gdb:

[New LWP 19118]
[New LWP 19119]
[New LWP 19120]
[New LWP 19121]
[New LWP 19122]
[New LWP 19123]
[New LWP 19124]
[New LWP 19128]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.
  Id   Target Id         Frame
* 1    Thread 0x7f34e7361740 (LWP 19117) "Main" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  2    Thread 0x7f34e5f3d700 (LWP 19118) "Main" 0x00007f34e680efcb in __waitpid (pid=pid@entry=19913, stat_loc=stat_loc@entry=0x7f34e5f2be4c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  3    Thread 0x7f34e573c700 (LWP 19119) "Main" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7f34e4f3b700 (LWP 19120) "Main" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  5    Thread 0x7f34e473a700 (LWP 19121) "Main" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  6    Thread 0x7f34e3f39700 (LWP 19122) "Main" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7f34e3395700 (LWP 19123) "Finalizer" 0x00007f34e624b826 in __GI___sigsuspend (set=set@entry=0x957f20 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
  8    Thread 0x7f34e29f5700 (LWP 19124) "Timer-Scheduler" 0x00007f34e624b826 in __GI___sigsuspend (set=set@entry=0x957f20 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
  9    Thread 0x7f34e1b45700 (LWP 19128) "Threadpool work" 0x00007f34e624b826 in __GI___sigsuspend (set=set@entry=0x957f20 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30

Thread 9 (Thread 0x7f34e1b45700 (LWP 19128)):
#0  0x00007f34e624b826 in __GI___sigsuspend (set=set@entry=0x957f20 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000006378b1 in _GC_suspend_handler (sig=<optimized out>) at pthread_stop_world.c:182
#2  GC_suspend_handler (sig=<optimized out>) at pthread_stop_world.c:207
#3  <signal handler called>
#4  0x00007f34e6310b5d in poll () at ../sysdeps/unix/syscall-template.S:84
#5  0x000000000061f067 in mono_poll (ufds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at mono-poll.c:26
#6  0x00000000005a9f7f in poll_event_wait (callback=0x5aa9f0 <wait_callback>, user_data=0x7f34e73339b0) at threadpool-ms-io-poll.c:147
#7  0x00000000005aaf93 in selector_thread (data=<optimized out>) at threadpool-ms-io.c:424
#8  selector_thread (data=data@entry=0x0) at threadpool-ms-io.c:309
#9  0x00000000005a3a6c in start_wrapper_internal (data=<optimized out>) at threads.c:740
#10 start_wrapper (data=<optimized out>) at threads.c:788
#11 0x000000000062565c in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:92
#12 0x00000000006374c4 in GC_start_routine (arg=<optimized out>) at pthread_support.c:1508
#13 0x00007f34e680570a in start_thread (arg=0x7f34e1b45700) at pthread_create.c:333
#14 0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 8 (Thread 0x7f34e29f5700 (LWP 19124)):
#0  0x00007f34e624b826 in __GI___sigsuspend (set=set@entry=0x957f20 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000006378b1 in _GC_suspend_handler (sig=<optimized out>) at pthread_stop_world.c:182
#2  GC_suspend_handler (sig=<optimized out>) at pthread_stop_world.c:207
#3  <signal handler called>
#4  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#5  0x00000000005fc771 in mono_os_cond_wait (mutex=0x2315080, cond=0x23150a8) at ../../mono/utils/mono-os-mutex.h:107
#6  mono_os_cond_timedwait (timeout_ms=4294967295, mutex=0x2315080, cond=0x23150a8) at ../../mono/utils/mono-os-mutex.h:122
#7  _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x409, timeout=timeout@entry=4294967295, alertable=alertable@entry=1, poll=poll@entry=0, alerted=alerted@entry=0x7f34e29f432c) at handles.c:1555
#8  0x00000000006124d4 in wapi_WaitForSingleObjectEx (handle=0x409, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at wait.c:181
#9  0x00000000005a37a7 in mono_wait_uninterrupted (numhandles=numhandles@entry=1, handles=handles@entry=0x7f34e29f43e8, waitall=waitall@entry=0, ms=ms@entry=-1, error=error@entry=0x7f34e29f43f0, thread=<optimized out>) at threads.c:1595
#10 0x00000000005a4a66 in ves_icall_System_Threading_WaitHandle_WaitOne_internal (handle=0x409, ms=-1) at threads.c:1731
#11 0x0000000041d323b8 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f34e3395700 (LWP 19123)):
#0  0x00007f34e624b826 in __GI___sigsuspend (set=set@entry=0x957f20 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000006378b1 in _GC_suspend_handler (sig=<optimized out>) at pthread_stop_world.c:182
#2  GC_suspend_handler (sig=<optimized out>) at pthread_stop_world.c:207
#3  <signal handler called>
#4  0x00007f34e680d875 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x955ba0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#5  do_futex_wait (sem=sem@entry=0x955ba0 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#6  0x00007f34e680d924 in __new_sem_wait_slow (sem=0x955ba0 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#7  0x00007f34e680d9ca in __new_sem_wait (sem=sem@entry=0x955ba0 <finalizer_sem>) at sem_wait.c:29
#8  0x00000000005c323c in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x955ba0 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:166
#9  mono_coop_sem_wait (sem=<optimized out>, flags=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:40
#10 finalizer_thread (unused=unused@entry=0x0) at gc.c:761
#11 0x00000000005a3a6c in start_wrapper_internal (data=<optimized out>) at threads.c:740
#12 start_wrapper (data=<optimized out>) at threads.c:788
#13 0x000000000062565c in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:92
#14 0x00000000006374c4 in GC_start_routine (arg=<optimized out>) at pthread_support.c:1508
#15 0x00007f34e680570a in start_thread (arg=0x7f34e3395700) at pthread_create.c:333
#16 0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f34e3f39700 (LWP 19122)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000637773 in GC_wait_marker () at pthread_support.c:1903
#2  0x000000000062f6cc in GC_mark_local (local_mark_stack=0x7f34e3f28ef0, id=<optimized out>) at mark.c:1018
#3  0x000000000062f918 in GC_help_marker (my_mark_no=1016) at mark.c:1127
#4  0x0000000000635d57 in GC_mark_thread (id=<optimized out>) at pthread_support.c:555
#5  0x00007f34e680570a in start_thread (arg=0x7f34e3f39700) at pthread_create.c:333
#6  0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 5 (Thread 0x7f34e473a700 (LWP 19121)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000637773 in GC_wait_marker () at pthread_support.c:1903
#2  0x000000000062f6cc in GC_mark_local (local_mark_stack=0x7f34e4729ef0, id=<optimized out>) at mark.c:1018
#3  0x000000000062f918 in GC_help_marker (my_mark_no=1016) at mark.c:1127
#4  0x0000000000635d57 in GC_mark_thread (id=<optimized out>) at pthread_support.c:555
#5  0x00007f34e680570a in start_thread (arg=0x7f34e473a700) at pthread_create.c:333
#6  0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f34e4f3b700 (LWP 19120)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000637773 in GC_wait_marker () at pthread_support.c:1903
#2  0x000000000062f6cc in GC_mark_local (local_mark_stack=0x7f34e4f2aef0, id=<optimized out>) at mark.c:1018
#3  0x000000000062f918 in GC_help_marker (my_mark_no=1016) at mark.c:1127
#4  0x0000000000635d57 in GC_mark_thread (id=<optimized out>) at pthread_support.c:555
#5  0x00007f34e680570a in start_thread (arg=0x7f34e4f3b700) at pthread_create.c:333
#6  0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7f34e573c700 (LWP 19119)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000637773 in GC_wait_marker () at pthread_support.c:1903
#2  0x000000000062f6cc in GC_mark_local (local_mark_stack=0x7f34e572bef0, id=<optimized out>) at mark.c:1018
#3  0x000000000062f918 in GC_help_marker (my_mark_no=1016) at mark.c:1127
#4  0x0000000000635d57 in GC_mark_thread (id=<optimized out>) at pthread_support.c:555
#5  0x00007f34e680570a in start_thread (arg=0x7f34e573c700) at pthread_create.c:333
#6  0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f34e5f3d700 (LWP 19118)):
#0  0x00007f34e680efcb in __waitpid (pid=pid@entry=19913, stat_loc=stat_loc@entry=0x7f34e5f2be4c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00000000004b4139 in mono_handle_native_sigsegv (signal=signal@entry=11, ctx=ctx@entry=0x7f34e5f2c7c0, info=info@entry=0x7f34e5f2c8f0) at mini-exceptions.c:2427
#2  0x000000000042ea3c in mono_sigsegv_signal_handler (_dummy=11, _info=0x7f34e5f2c8f0, context=0x7f34e5f2c7c0) at mini-runtime.c:2826
#3  <signal handler called>
#4  GC_mark_from (mark_stack_top=0x7f34e5f2cf70, mark_stack_top@entry=0x7f34e5f2cf40, mark_stack=mark_stack@entry=0x7f34e5f2cef0, mark_stack_limit=mark_stack_limit@entry=0x7f34e5f3cef0) at mark.c:724
#5  0x000000000062f52e in GC_do_local_mark (local_mark_stack=0x7f34e5f2cef0, local_top=0x7f34e5f2cf40) at mark.c:925
#6  0x000000000062f738 in GC_mark_local (local_mark_stack=0x7f34e5f2cef0, id=<optimized out>) at mark.c:1056
#7  0x000000000062f918 in GC_help_marker (my_mark_no=1016) at mark.c:1127
#8  0x0000000000635d57 in GC_mark_thread (id=<optimized out>) at pthread_support.c:555
#9  0x00007f34e680570a in start_thread (arg=0x7f34e5f3d700) at pthread_create.c:333
#10 0x00007f34e631c82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f34e7361740 (LWP 19117)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f34e6807e0d in __GI___pthread_mutex_lock (mutex=0x957600 <GC_allocate_ml>) at ../nptl/pthread_mutex_lock.c:80
#2  0x000000000062d415 in GC_malloc_atomic (lb=192) at malloc.c:259
#3  0x00000000005ef078 in mono_gc_alloc_vector (vtable=vtable@entry=0x2923ea8, size=192, max_length=max_length@entry=20) at boehm-gc.c:676
#4  0x00000000005cc362 in mono_array_new_specific_checked (vtable=0x2923ea8, n=20, error=0x7ffcfe427d50) at object.c:5916
#5  0x00000000005d1ce1 in mono_glist_to_array (list=0x2906750, eclass=0x233fc88, error=error@entry=0x7ffcfe427d50) at object.c:8489
#6  0x0000000000424d3a in setup_stack_trace (mono_ex=mono_ex@entry=0x7f34e1658aa0, dynamic_methods=dynamic_methods@entry=0x0, initial_trace_ips=initial_trace_ips@entry=0x0, trace_ips=trace_ips@entry=0x7ffcfe427ed8) at mini-exceptions.c:1316
#7  0x00000000004b4d2f in mono_handle_exception_internal_first_pass (non_exception=0x0, out_prev_ji=<synthetic pointer>, out_ji=<synthetic pointer>, out_filter_idx=<synthetic pointer>, obj=0x7f34e1658aa0, ctx=0x7ffcfe428090) at mini-exceptions.c:1434
#8  mono_handle_exception_internal (ctx=ctx@entry=0x7ffcfe428400, obj=0x7f34e1658aa0, resume=resume@entry=0, out_ji=0x0) at mini-exceptions.c:1721
#9  0x00000000004b5592 in mono_handle_exception (ctx=ctx@entry=0x7ffcfe428400, obj=<optimized out>) at mini-exceptions.c:2027
#10 0x000000000050a869 in altstack_handle_and_restore (ctx=<optimized out>, obj=<optimized out>, stack_ovf=0) at exceptions-amd64.c:776
#11 0x000000000062eee6 in GC_mark_from (mark_stack_top=0x7ffcfe4288d0, mark_stack=0x0, mark_stack_limit=0x27d9d0bf) at mark.c:725
#12 0x000000000062f861 in GC_do_parallel_mark () at mark.c:1100
#13 0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
Comment 5 Vlad Brezae 2016-11-28 19:27:09 UTC
Are you using any unsafe code ? Normally, this looks like some buffer overflow with unsafe usage. Could you check for this by using MONO_GC_DEBUG=nursery-canaries (only with sgen) and let me know if you get any outputs from this ?
Comment 6 Ben Rush 2016-11-28 19:34:57 UTC
No, no unsafe code directly. I'm using the Process class to launch and automate a bunch of external processes. 

I will do what you ask tonight and get back to you. Thanks.
Comment 7 Ben Rush 2016-12-15 02:07:40 UTC
Vlad, 

My apologies for this taking so long. I have been doing a lot for my day job lately in the evenings and the product which has been causing this problem is my side job. At any rate, I have your crash information (as well as the canary information you requested): 


Checking nursery canaries...
CORRUPT CANARY:
addr->0x7ff4cab72930
type->UnixFileInfo
excepted->'koupepia'
found->'P▒▒

tacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_ves_icall_array_new_specific (intptr,int) <0x00057>
  at (wrapper alloc) object.SlowAllocVector (intptr,intptr) <0x00013>
  at System.Collections.Generic.List`1<T_REF>.set_Capacity (int) <0x00061>
  at System.Collections.Generic.List`1<T_REF>.EnsureCapacity (int) <0x00085>
  at System.Collections.Generic.List`1<T_REF>.Add (T_REF) <0x00035>
  at WrfSharp.Helpers.Namelists.NamelistParser.ParseFromString (string) <0x0094e>
  at WrfSharp.Helpers.Namelists.NamelistHelper.UpdatePhysicsParameters (WrfSharp.DataStructures.WrfConfiguration,WrfSharp.DataStructures.PhysicsConfigurationProcessed,erfaces.IFileSystem) <0x00057>
  at WrfSharp.Runner.Program.ComputeStage (WrfSharp.Interfaces.IFileSystem,WrfSharp.Interfaces.ILogger,WrfSharp.Interfaces.IProcessLauncher,WrfSharp.Interfaces.IEnvirorp.Interfaces.IDatabase,WrfSharp.DataStructures.WrfConfiguration,WrfSharp.DataStructures.PhysicsConfigurationProcessed) <0x000e7>
  at WrfSharp.Runner.Program.Main (string[]) <0x00213>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0x000c8>

Native stacktrace:

        mono-sgen() [0x4accca]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7ff4cb89f390]
        /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7ff4cb2db428]
        /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7ff4cb2dd02a]
        mono-sgen() [0x65aae9]
        mono-sgen() [0x65acf7]
        mono-sgen() [0x65ada2]
        mono-sgen() [0x6010a6]
        mono-sgen() [0x6077d3]
        mono-sgen() [0x607c4c]
        mono-sgen() [0x5fb4fd]
        mono-sgen() [0x5e8b91]
        mono-sgen() [0x5c4ac2]
        mono-sgen() [0x5c4c9d]
        [0x41c7be08]

Debug info from gdb:

[New LWP 27949]
[New LWP 27950]
[New LWP 27951]
[New LWP 27955]
Mono support loaded.
Mono support loaded.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007ff4cb89ef7b in __waitpid (pid=pid@entry=29298, stat_loc=stat_loc@entry=0x7fffe9f14a0c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29      ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id         Frame
* 1    Thread 0x7ff4cc3cc740 (LWP 27948) "Main" 0x00007ff4cb89ef7b in __waitpid (pid=pid@entry=29298, stat_loc=stat_loc@entry=0x7fffe9f14a0c, options=options@entry=0) s/unix/sysv/linux/waitpid.c:29
  2    Thread 0x7ff4ca7ff700 (LWP 27949) "SGen worker" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7ff4c87cd700 (LWP 27950) "Finalizer" 0x00007ff4cb89d827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x97b100 <fin at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7ff4c39ff700 (LWP 27951) "Timer-Scheduler" 0x00007ff4cb2db826 in __GI___sigsuspend (set=set@entry=0x97b640 <suspend_signal_mask>) at ../sysdeps/unix/sysuspend.c:30
  5    Thread 0x7ff4c808d700 (LWP 27955) "Threadpool work" 0x00007ff4cb3a0b5d in poll () at ../sysdeps/unix/syscall-template.S:84

Thread 5 (Thread 0x7ff4c808d700 (LWP 27955)):
#0  0x00007ff4cb3a0b5d in poll () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000000064c527 in mono_poll (ufds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at mono-poll.c:26
#2  0x00000000005a2b9f in poll_event_wait (callback=0x5a3610 <wait_callback>, user_data=0x7ff4b00019b0) at threadpool-ms-io-poll.c:147
#3  0x00000000005a3bb3 in selector_thread (data=<optimized out>) at threadpool-ms-io.c:424
#4  selector_thread (data=data@entry=0x0) at threadpool-ms-io.c:309
#5  0x000000000059c68c in start_wrapper_internal (data=<optimized out>) at threads.c:740
#6  start_wrapper (data=<optimized out>) at threads.c:788
#7  0x000000000065381c in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:92
#8  0x00007ff4cb8956ba in start_thread (arg=0x7ff4c808d700) at pthread_create.c:333
#9  0x00007ff4cb3ac82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7ff4c39ff700 (LWP 27951)):
#0  0x00007ff4cb2db826 in __GI___sigsuspend (set=set@entry=0x97b640 <suspend_signal_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:30
#1  0x00000000005e6ed4 in suspend_thread (info=0x7ff4bc0008e0, context=context@entry=0x7ff4c39fde00) at sgen-os-posix.c:98
#2  0x00000000005e711d in suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7ff4c39fde00) at sgen-os-posix.c:125
#3  <signal handler called>
#4  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#5  0x0000000000629c31 in mono_os_cond_wait (mutex=0x1c95f20, cond=0x1c95f48) at ../../mono/utils/mono-os-mutex.h:107
#6  mono_os_cond_timedwait (timeout_ms=4294967295, mutex=0x1c95f20, cond=0x1c95f48) at ../../mono/utils/mono-os-mutex.h:122
#7  _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x407, timeout=timeout@entry=4294967295, alertable=alertable@entry=1, poll=poll@entry=0, alerted=alerted@c39fe4cc) at handles.c:1555
#8  0x000000000063f994 in wapi_WaitForSingleObjectEx (handle=0x407, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at wait.c:181
#9  0x000000000059c3c7 in mono_wait_uninterrupted (numhandles=numhandles@entry=1, handles=handles@entry=0x7ff4c39fe588, waitall=waitall@entry=0, ms=ms@entry=-1, error=0x7ff4c39fe590, thread=<optimized out>) at threads.c:1595
#10 0x000000000059d686 in ves_icall_System_Threading_WaitHandle_WaitOne_internal (handle=0x407, ms=-1) at threads.c:1731
#11 0x0000000041cfa068 in ?? ()
#12 0x00000000020e9d10 in ?? ()
#13 0xffffffffffffffff in ?? ()
#14 0x7fffffffffffffff in ?? ()
#15 0x00007ff4cab0eb60 in ?? ()
#16 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7ff4c87cd700 (LWP 27950)):
#0  0x00007ff4cb89d827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x97b100 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-05
#1  do_futex_wait (sem=sem@entry=0x97b100 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007ff4cb89d8d4 in __new_sem_wait_slow (sem=0x97b100 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007ff4cb89d97a in __new_sem_wait (sem=sem@entry=0x97b100 <finalizer_sem>) at sem_wait.c:29
#4  0x00000000005bb8d9 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x97b100 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:166
#5  mono_coop_sem_wait (sem=<optimized out>, flags=<optimized out>) at ../../mono/utils/mono-coop-semaphore.h:40
#6  finalizer_thread (unused=unused@entry=0x0) at gc.c:761
#7  0x000000000059c68c in start_wrapper_internal (data=<optimized out>) at threads.c:740
#8  start_wrapper (data=<optimized out>) at threads.c:788
#9  0x000000000065381c in inner_start_thread (arg=<optimized out>) at mono-threads-posix.c:92
#10 0x00007ff4cb8956ba in start_thread (arg=0x7ff4c87cd700) at pthread_create.c:333
#11 0x00007ff4cb3ac82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7ff4ca7ff700 (LWP 27949)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000623717 in mono_os_cond_wait (cond=0x98a420 <work_cond>, mutex=<optimized out>) at ../../mono/utils/mono-os-mutex.h:107
#2  thread_func (thread_data=0x0) at sgen-thread-pool.c:110
#3  0x00007ff4cb8956ba in start_thread (arg=0x7ff4ca7ff700) at pthread_create.c:333
#4  0x00007ff4cb3ac82d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7ff4cc3cc740 (LWP 27948)):
#0  0x00007ff4cb89ef7b in __waitpid (pid=pid@entry=29298, stat_loc=stat_loc@entry=0x7fffe9f14a0c, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00000000004acd59 in mono_handle_native_sigsegv (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2427
#2  <signal handler called>
#3  0x00007ff4cb2db428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#4  0x00007ff4cb2dd02a in __GI_abort () at abort.c:89
#5  0x000000000065aae9 in monoeg_log_default_handler (log_domain=<optimized out>, log_level=<optimized out>, message=<optimized out>, unused_data=<optimized out>) at g
#6  monoeg_log_default_handler (log_domain=<optimized out>, log_level=G_LOG_LEVEL_ERROR, message=<optimized out>, unused_data=<optimized out>) at goutput.c:221
#7  0x000000000065acf7 in monoeg_g_logv (log_domain=0x0, log_level=G_LOG_LEVEL_ERROR, format=<optimized out>, args=args@entry=0x7fffe9f15988) at goutput.c:113
#8  0x000000000065ada2 in monoeg_g_log (log_domain=log_domain@entry=0x0, log_level=log_level@entry=G_LOG_LEVEL_ERROR, format=format@entry=0x6fe7e8 "CORRUPT CANARY:\nad->%s\nexcepted->'%s'\nfound->'%s'\n") at goutput.c:123
#9  0x00000000006010a6 in sgen_debug_verify_nursery (do_dump_nursery_content=0) at sgen-debug.c:666
#10 0x00000000006077d3 in collect_nursery (reason=reason@entry=0x702c3d "Nursery full", is_overflow=is_overflow@entry=0, finish_up_concurrent_mark=0, unpin_queue=0x0) :1513
#11 0x0000000000607c4c in major_start_concurrent_collection (reason=<optimized out>) at sgen-gc.c:2121
#12 sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x702c3d "Nursery full", wait_to_finish=0, stw=1) at sgen-gc.c:2277
#13 0x0000000000607f2c in sgen_ensure_free_space (size=<optimized out>, generation=<optimized out>) at sgen-gc.c:2232
#14 0x00000000005fb4fd in sgen_alloc_obj_nolock (vtable=vtable@entry=0x223dfb8, size=104, size@entry=96) at sgen-alloc.c:262
#15 0x00000000005e8b91 in mono_gc_alloc_vector (vtable=vtable@entry=vtable("WrfSharp.DataStructures.NamelistItem[]"), size=96, max_length=max_length@entry=8) at sgen-m
#16 0x00000000005c4ac2 in mono_array_new_specific_checked (vtable=vtable("WrfSharp.DataStructures.NamelistItem[]"), n=8, error=0x7fffe9f15c70) at object.c:5776
#17 0x00000000005c4c9d in ves_icall_array_new_specific (vtable=<optimized out>, n=<optimized out>) at object.c:5790
#18 0x0000000041c7be08 in ?? ()
#19 0x00007ff4cab0d070 in ?? ()
#20 0x00007ff4cab0d0a0 in ?? ()
#21 0x0000000000000008 in ?? ()
#22 0x0000000000000008 in ?? ()
#23 0x0000000000000008 in ?? ()
#24 0x0000000001cc9b00 in ?? ()
#25 0x0000000000000000 in ?? ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Aborted (core dumped)
Comment 8 Vlad Brezae 2016-12-15 18:10:23 UTC
The canary corruption is another strong indicator of some unsafe bad behavior. Maybe one of the third parties are doing something wrong. I'm afraid there is not much that can be done from the stacktraces.

How easy would it be to setup the application and crash it ? Can it be done in an automated way ?
Comment 9 Ben Rush 2016-12-15 19:32:19 UTC
Vlad, 

Unfortunately it's a very complex application with a lot of parts that need to be sublaunched. All the program itself does is launch these applications, it doesn't actually do anything more than parse configuration files, update some configuration files, launch a process and then wait. 

Is it possible the Mono.Posix library has a bug in it? That's really the only third party library I'm using.
Comment 10 Vlad Brezae 2016-12-15 21:48:10 UTC
Does the canary corruption always occur on an UnixFileInfo ? Would it be possible to get access to the machine to repro this ? Not much can be done without repro + gdb + source recompilation.
Comment 11 Ben Rush 2016-12-15 21:56:21 UTC
Let me see what I can do to reduce this to a more minimal, more quickly reproducible state. As it stands right now, it is strangely becoming more infrequent (though as you can see, it's still happening). 

Let me get back to you in a few days.
Comment 12 Ben Rush 2016-12-29 05:36:47 UTC
So I think I may have inadvertently solved this problem. I'm about 80% sure it has to do with the Mono.Posix library I was using from within Visual Studio. I noticed there are two Mono.Posix libraries available through NuGet: "Mono.Posix" and "Mono.Posix-4.5". I was using "Mono.Posix" originally, but since moving over to Mono.Posix-4.5 this issue no longer occurs. 

There's still a slight possibility I might reproduce it in the future, but for now it has not appeared once over many, many runs. 

So...I will post here if I can duplicate it, but for now it appears resolved for me.
Comment 13 Vlad Brezae 2017-01-06 18:23:03 UTC
This seems to be a known scenario where mono loads assemblies from the current folder instead of the GAC directories. The nuget library that you were previously compiling against was probably incompatible with the used mono. You should be able to avoid the copying of the assembly in the executable folder by setting "CopyLocal=false" for it.

Marking as resolved for now
Comment 14 Ben Rush 2017-01-06 18:51:31 UTC
Thanks for your time, Vlad. It has now gone a week without crashing. I'm 99% sure it's resolved.
Comment 15 Roman 2018-05-05 07:46:05 UTC
I have same problem but I don't use Mono.Posix.dll

I have C library for working with this pos_terminal https://www.verifone.com/en/us/devices/countertops-pin-pads/vx-520

I use 

```[DllImport("libpos.so.1.24.0", EntryPoint = "pos_open", CallingConvention = CallingConvention.Cdecl)]
        private extern static Boolean Open(ref IntPtr handle_p, string name, string log);

        [DllImport("libpos.so.1.24.0", EntryPoint = "pos_set", CallingConvention = CallingConvention.Cdecl)]
        private extern static Boolean Set(ref IntPtr handle_p, string param, string val);
```

And my application crashes after second method calling 

myName@BABYLON-5:/media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug$ sudo mono --debug ConsoleApplication.exe 
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) PosTerminal.CommandWrapper.Set (intptr&,string,string) [0x0001f] in <9b3bfb8fdc44474d831a941f1e5698ec>:0
  at PosTerminal.CommandWrapper.SetParam (string,string) [0x00004] in /media/myName/Documents/Projects/NewBasket/PosTerminal/CommandWrapper.cs:168
  at ConsoleApplication.MainClass.Main (string[]) [0x00022] in /media/myName/Documents/Projects/NewBasket/ConsoleApplication/Program.cs:15
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) [0x00051] in <f3c7ded2422f4a6e8752f69fae0a14d1>:0
/proc/self/maps:
416f1000-41701000 rwxp 00000000 00:00 0 
4190b000-4191b000 rwxp 00000000 00:00 0 
56458a7ba000-56458abdb000 r-xp 00000000 08:18 2492164                    /usr/bin/mono-sgen
56458addb000-56458ade2000 r--p 00421000 08:18 2492164                    /usr/bin/mono-sgen
56458ade2000-56458ade6000 rw-p 00428000 08:18 2492164                    /usr/bin/mono-sgen
56458ade6000-56458adfd000 rw-p 00000000 00:00 0 
56458c985000-56458ca41000 rw-p 00000000 00:00 0                          [heap]
7f7284000000-7f7284021000 rw-p 00000000 00:00 0 
7f7284021000-7f7288000000 ---p 00000000 00:00 0 
7f7288cfa000-7f7288e6c000 r-xp 00000000 08:18 2893317                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f7288e6c000-7f728906c000 ---p 00172000 08:18 2893317                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f728906c000-7f7289076000 r--p 00172000 08:18 2893317                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f7289076000-7f7289078000 rw-p 0017c000 08:18 2893317                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f7289078000-7f728907c000 rw-p 00000000 00:00 0 
7f728907c000-7f72890a2000 r-xp 00000000 08:15 379958                     /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/libpos.so.1.24.0
7f72890a2000-7f72892a2000 ---p 00026000 08:15 379958                     /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/libpos.so.1.24.0
7f72892a2000-7f72892a4000 rw-p 00026000 08:15 379958                     /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/libpos.so.1.24.0
7f72892a4000-7f72892ad000 ---p 00000000 00:00 0 
7f72892ad000-7f72894a5000 rw-p 00000000 00:00 0 
7f72894a5000-7f7289a0d000 r-xp 00000000 08:18 2892802                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f7289a0d000-7f7289c0c000 ---p 00568000 08:18 2892802                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f7289c0c000-7f7289c0d000 r--p 00567000 08:18 2892802                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f7289c0d000-7f7289c0e000 rw-p 00568000 08:18 2892802                    /usr/lib/mono/aot-cache/amd64/mscorlib.dll.so
7f7289c0e000-7f7289c34000 rw-p 00000000 00:00 0 
7f7289c34000-7f7289fff000 r--p 00000000 08:18 2891934                    /usr/lib/mono/4.5/mscorlib.dll
7f7289fff000-7f728afff000 rw-p 00000000 00:00 0 
7f728afff000-7f728b000000 ---p 00000000 00:00 0 
7f728b000000-7f728bc00000 rw-p 00000000 00:00 0 
7f728bc26000-7f728bcd8000 rw-p 00000000 00:00 0 
7f728bcd9000-7f728bd0b000 rw-p 00000000 00:00 0 
7f728bd0b000-7f728bd3c000 r--p 00000000 08:18 3952192                    /usr/share/locale/uk/LC_MESSAGES/libc.mo
7f728bd3c000-7f728bd3d000 rw-p 00000000 00:00 0 
7f728bd3d000-7f728bd9c000 ---p 00000000 00:00 0 
7f728bd9c000-7f728bf31000 r-xp 00000000 08:18 4588406                    /lib/x86_64-linux-gnu/libc-2.24.so
7f728bf31000-7f728c131000 ---p 00195000 08:18 4588406                    /lib/x86_64-linux-gnu/libc-2.24.so
7f728c131000-7f728c135000 r--p 00195000 08:18 4588406                    /lib/x86_64-linux-gnu/libc-2.24.so
7f728c135000-7f728c137000 rw-p 00199000 08:18 4588406                    /lib/x86_64-linux-gnu/libc-2.24.so
7f728c137000-7f728c13b000 rw-p 00000000 00:00 0 
7f728c13b000-7f728c151000 r-xp 00000000 08:18 4594524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f728c151000-7f728c350000 ---p 00016000 08:18 4594524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f728c350000-7f728c351000 r--p 00015000 08:18 4594524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f728c351000-7f728c352000 rw-p 00016000 08:18 4594524                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f728c352000-7f728c36a000 r-xp 00000000 08:18 4588421                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f728c36a000-7f728c569000 ---p 00018000 08:18 4588421                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f728c569000-7f728c56a000 r--p 00017000 08:18 4588421                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f728c56a000-7f728c56b000 rw-p 00018000 08:18 4588421                    /lib/x86_64-linux-gnu/libpthread-2.24.so
7f728c56b000-7f728c56f000 rw-p 00000000 00:00 0 
7f728c56f000-7f728c572000 r-xp 00000000 08:18 4588409                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f728c572000-7f728c771000 ---p 00003000 08:18 4588409                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f728c771000-7f728c772000 r--p 00002000 08:18 4588409                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f728c772000-7f728c773000 rw-p 00003000 08:18 4588409                    /lib/x86_64-linux-gnu/libdl-2.24.so
7f728c773000-7f728c77a000 r-xp 00000000 08:18 4588423                    /lib/x86_64-linux-gnu/librt-2.24.so
7f728c77a000-7f728c979000 ---p 00007000 08:18 4588423                    /lib/x86_64-linux-gnu/librt-2.24.so
7f728c979000-7f728c97a000 r--p 00006000 08:18 4588423                    /lib/x86_64-linux-gnu/librt-2.24.so
7f728c97a000-7f728c97b000 rw-p 00007000 08:18 4588423                    /lib/x86_64-linux-gnu/librt-2.24.so
7f728c97b000-7f728ca7e000 r-xp 00000000 08:18 4588410                    /lib/x86_64-linux-gnu/libm-2.24.so
7f728ca7e000-7f728cc7d000 ---p 00103000 08:18 4588410                    /lib/x86_64-linux-gnu/libm-2.24.so
7f728cc7d000-7f728cc7e000 r--p 00102000 08:18 4588410                    /lib/x86_64-linux-gnu/libm-2.24.so
7f728cc7e000-7f728cc7f000 rw-p 00103000 08:18 4588410                    /lib/x86_64-linux-gnu/libm-2.24.so
7f728cc7f000-7f728cca2000 r-xp 00000000 08:18 4588365                    /lib/x86_64-linux-gnu/ld-2.24.so
7f728cca3000-7f728cce4000 rw-p 00000000 00:00 0 
7f728cce4000-7f728ce76000 r--p 00000000 08:18 2758121                    /usr/lib/locale/locale-archive
7f728ce76000-7f728ce7b000 rw-p 00000000 00:00 0 
7f728ce7c000-7f728ce7d000 r--p 00000000 08:15 866                        /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/PosTerminal.pdb
7f728ce7d000-7f728ce80000 r--p 00000000 08:15 865                        /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/PosTerminal.dll
7f728ce80000-7f728ce81000 r--p 00000000 08:15 869                        /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/ConsoleApplication.pdb
7f728ce81000-7f728ce89000 rw-p 00000000 00:00 0 
7f728ce89000-7f728ce90000 r--s 00000000 08:18 2894135                    /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f728ce90000-7f728ce95000 rw-p 00000000 00:00 0 
7f728ce95000-7f728ce97000 r--p 00000000 08:15 868                        /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/ConsoleApplication.exe
7f728ce97000-7f728ce9e000 rw-p 00000000 00:00 0 
7f728ce9e000-7f728ce9f000 rw-s 00000000 00:13 300913                     /dev/shm/mono.4723
7f728ce9f000-7f728cea2000 rw-p 00000000 00:00 0 
7f728cea2000-7f728cea3000 r--p 00023000 08:18 4588365                    /lib/x86_64-linux-gnu/ld-2.24.so
7f728cea3000-7f728cea4000 rw-p 00024000 08:18 4588365                    /lib/x86_64-linux-gnu/ld-2.24.so
7f728cea4000-7f728cea5000 rw-p 00000000 00:00 0 
7ffe07899000-7ffe078a1000 ---p 00000000 00:00 0 
7ffe08077000-7ffe08098000 rw-p 00000000 00:00 0                          [stack]
7ffe080e0000-7ffe080e2000 r--p 00000000 00:00 0                          [vvar]
7ffe080e2000-7ffe080e4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Native stacktrace:

	mono(+0xcb3ac) [0x56458a8853ac]
	mono(+0x160a10) [0x56458a91aa10]
	/media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/libpos.so.1.24.0(_ZN2cp7storage3setEPKcS2_b+0x50) [0x7f7289091990]

Debug info from gdb:

[New LWP 4724]
[New LWP 4725]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f728c362b3a in __waitpid (pid=pid@entry=4727, stat_loc=stat_loc@entry=0x7ffe080947fc, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29	../sysdeps/unix/sysv/linux/waitpid.c: Немає такого файла або каталогу.
  Id   Target Id         Frame 
* 1    Thread 0x7f728ce76740 (LWP 4723) "mono" 0x00007f728c362b3a in __waitpid (pid=pid@entry=4727, stat_loc=stat_loc@entry=0x7ffe080947fc, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  2    Thread 0x7f728b7ff700 (LWP 4724) "SGen worker" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7f72894a4700 (LWP 4725) "Finalizer" 0x00007f728c361536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x56458adeb9a0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205

Thread 3 (Thread 0x7f72894a4700 (LWP 4725)):
#0  0x00007f728c361536 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x56458adeb9a0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x56458adeb9a0 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x00007f728c3615e4 in __new_sem_wait_slow (sem=0x56458adeb9a0 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x00007f728c361679 in __new_sem_wait (sem=sem@entry=0x56458adeb9a0 <finalizer_sem>) at sem_wait.c:29
#4  0x000056458aa35932 in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x56458adeb9a0 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:209
#5  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x56458adeb9a0 <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:43
#6  finalizer_thread (unused=unused@entry=0x0) at gc.c:852
#7  0x000056458a9f451c in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:1012
#8  start_wrapper (data=0x56458c9e8440) at threads.c:1072
#9  0x00007f728c359494 in start_thread (arg=0x7f72894a4700) at pthread_create.c:333
#10 0x00007f728be84acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7f728b7ff700 (LWP 4724)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000056458aa92c4b in mono_os_cond_wait (mutex=0x56458adfa860 <lock>, cond=0x56458adfa820 <work_cond>) at ../../mono/utils/mono-os-mutex.h:173
#2  get_work (job=<synthetic pointer>, do_idle=<synthetic pointer>, work_context=<synthetic pointer>, worker_index=0) at sgen-thread-pool.c:165
#3  thread_func (data=<optimized out>) at sgen-thread-pool.c:196
#4  0x00007f728c359494 in start_thread (arg=0x7f728b7ff700) at pthread_create.c:333
#5  0x00007f728be84acf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7f728ce76740 (LWP 4723)):
#0  0x00007f728c362b3a in __waitpid (pid=pid@entry=4727, stat_loc=stat_loc@entry=0x7ffe080947fc, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x000056458a885490 in mono_handle_native_crash (signal=signal@entry=0x56458aabd824 "SIGSEGV", ctx=ctx@entry=0x0, info=info@entry=0x0) at mini-exceptions.c:2830
#2  0x000056458a91aa10 in altstack_handle_and_restore (ctx=0x7ffe08095330, obj=0x0, stack_ovf=0) at exceptions-amd64.c:849
#3  0x00007f7289091990 in cp::storage::set(char const*, char const*, bool) () from /media/myName/Documents/Projects/NewBasket/ConsoleApplication/bin/Debug/libpos.so.1.24.0
#4  0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

Перервано
Comment 16 Roman 2018-05-05 07:47:52 UTC
This happens always. Mono 5.10, OS Debian 9 Stretch