Notice (2018-05-24): bugzilla.xamarin.com is now in
Please join us on
Visual Studio Developer Community and in the
Mono organizations on
GitHub to continue tracking issues. Bugzilla will remain
available for reference in read-only mode. We will continue to work
on open Bugzilla bugs, copy them to the new locations
as needed for follow-up, and add the new items under Related
Our sincere thanks to everyone who has contributed on this bug
tracker over the years. Thanks also for your understanding as we
make these adjustments and improvements for the future.
Please create a new report on
GitHub or Developer Community with
your current version information, steps to reproduce, and relevant error
messages or log files if you are hitting an issue that looks similar to
this resolved bug and you do not yet see a matching new report.
A mono application crash randomly in production. A crash doesn't correlate to a [memory/cpu/thread count].
This happens on most of the 8 servers in production. On some servers it happens 2-3 times per day, on others once per 3 days.
Other mono applications doesn't crash on the same servers and the same mono runtime.
We don't know how to reproduce it.
The big change that might be related in the last release of our application was, that it uses a lot of short threads. These threads are created for each message handler. The thread can take a few miliseconds to complete.
1) Can you advice how to troubleshoot in this case?
2) Does a Native stacktrace tells you what the runtime is doing at the moment of crash? I'd like to know the context, it could help me to do some workaround in my code to avoid this Mono path.
**The line where Mono is crashing:**
Mono JIT compiler version 4.6.2 (Stable 22.214.171.124/08fd525 Tue Nov 29 13:50:06 UTC 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
LLVM: supported, not enabled.
**The output on crash is:**
* Assertion at ../../mono/utils/mono-os-semaphore.h:228, condition `errno != EINVAL' not met
Debug info from gdb:
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
Please use the latest stable mono version which is 4.8.
Thanks for an answer. But that's the last thing we would want to try, because updating the Mono would require a full regression and as we can't reproduce that in our QA, we wouldn't know if this helped or not until we deploy to production.
We've been in a similar situation when we tried to upgrade Mono to solve a problem, but the newer Mono version had a socket defect that caused a deadlock. It was a big pain to figure out an issue.
So I'd like to find the root cause of this problem, if it's a known issue, than I'd better include a targeted Mono patch that solves the problem. Or change our code to avoid this Mono path.
Unfortunately, there is no meaningful stack trace and the assertion location is low level code which is called from a lot of places, so its not possible to determine what the exact issue is, it might be fixed in a later mono version, or it might not be.
Could you install debug symbols and get a backtrace with full symbols on it?
Created attachment 21943 [details]
Mono crashed in our performance environment yesterday. There's a GDB installed and the output has a bit more info. See an attachment [crash_report.txt].
1) Is it normal that 5 thread stacks where printed only? There's 45 threads total.
2) Does it help to identify the problem? Is there anything else I could do to help identify the problem?
Created attachment 21944 [details]
Not sure if it's related, but we're seeing the memory leaking in most mono processes. There's a lot of [Thread, InternalThread, GenericPrincipal] objects uncollected. See the [US3_mprof_heapshot.txt ] attached.
The backtrace doesn't seem to contain the crashing thread, so it doesn't help much. It should print out all the threads, not sure why its not happening.
Yes, that's another problem we're facing. If I try to Debug with GDB and run a macro "mono_backtrace" then it always causes SIGSEGV for a process. mono_pmip works on 10% of threads. On other threads it causes SIGSEGV.
I never saw it printing a full stacktrace in any of our environments.
This happened on Mono v4.4 as well.
Created attachment 21960 [details]
crash report 2
Here's another crash report from a different service.
do you think this glibc bug can be related?
in our environment:
# rpm -q -a | grep glibc
It could be related, or not, hard to say, EINVAL is a general error code. I'd strongly suggest trying to update to mono 4.8, it _might_ fix the problem, there are packages available for most distros here:
If you can still reproduce with latest mono version, please feel free to reopen the bug. Thank you.