Bug 10003 - Mono crashes with SIGSEGV
Summary: Mono crashes with SIGSEGV
Status: RESOLVED NORESPONSE
Alias: None
Product: Runtime
Classification: Mono
Component: GC ()
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2013-02-03 10:36 UTC by Sergey Zhukov
Modified: 2017-07-07 19:06 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
stack traces (41.53 KB, text/x-log)
2013-02-03 10:36 UTC, Sergey Zhukov
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NORESPONSE

Description Sergey Zhukov 2013-02-03 10:36:48 UTC
Created attachment 3326 [details]
stack traces

I often get a mono crash in my async socket server in production environment. The crashes perform randomly generally when high load (more then 500 socket clients connected and ~95% memory used), and as I think they are related to operations of allocation or collection of objects in GC. I don't use unsafe or unmanaged code or interops, all code is 100% managed. Unfortunately, such randomness does not allow me to create small reproducible test-case and what can I do at this moment it's just to attach stack traces of faults. I hope, these stack traces help someone, who will meet such errors and run into the issue. If I can to recreate reproducible test-case I will attach it.

Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-1ubuntu2.2)
OS: Ubuntu 12.04
command-line: mono-sgen ./myserver.exe
Comment 1 Sergey Zhukov 2013-02-03 10:39:18 UTC
There are three different stack traces in the file.
Comment 2 Mark Probst 2013-02-04 18:29:39 UTC
These look like three separate issues, though they might have the same ultimate cause.

The first one is an OOM crash, which you've filed as #10005.

The second one looks like an exception handling problem, related to stack walking.

The third one is an SGen crash which doesn't seem caused by OOM.

We won't be able to do much without reproducible test cases.  Also, could you please try Mono 3.0.3?
Comment 3 Sergey Zhukov 2013-02-05 05:21:04 UTC
It's not easy to transfer production server to mono 3.0.3, because this operation requires additional hardware and retesting whole application. I am going to do it, but can't do it right now. By the way, I ran the sample provided in the #10005 in mono 3.0.4, and added the comment.
Comment 4 Mark Probst 2013-02-05 16:09:52 UTC
What if you run on 2.10 with Boehm instead of SGen?
Comment 5 Sergey Zhukov 2013-02-05 16:55:54 UTC
When I run my server with Boehm it starts to use swap memory, perfomance drops alot and my clients get lags and choose to disconnect from the server. So I cannot reach critical number of clients to check what's going on. I don't know how to tell mono with Boehm not to use swap while allocating memory. Also it seems that sgen behaviour in mono 3.0.4 similar to Boehm in 2.10.8 (gc tries to allocate memory in swap, when sgen gc in 2.10.8 does not allocate memory in swap)
Comment 6 Mark Probst 2013-02-05 16:58:40 UTC
Neither SGen nor Boehm allocate memory in swap.  They might allocate too much memory, which would then end up in swap.  That would point to a memory leak somewhere.  Your first backtrace also indicates that, though, and as far as I understood that was 2.10 with SGen?
Comment 7 Sergey Zhukov 2013-02-05 18:45:53 UTC
Stacktraces in this issue were for 2.10.8 with sgen gc.

In bug #10005 I provided test case, which tries to allocate whole memory by 100K chunks. The results were:
mono 2.10.8.1 Boehm - at first all free memory was allocated, than started to allocate memory from swap. After long time of swapping and awaiting the final message (couple of hours), I killed the program. Will try it again later on virtual machine with small swap.
mono 2.10.8.1 Sgen - program allocated all free memory and crashed 
mono 3.0.4 Sgen - at first program allocated all free memory, than all swap memory and than throwed OutOfMemoryException
Comment 8 Mark Probst 2013-02-05 22:11:10 UTC
Yes, but that's not the point.

The first backtrace here indicates that you're running into OOM on your server application, with Mono 2.10.8 SGen.  That and the fact that Boehm starts requiring swap as well, suggests that your application has a memory leak.

How does your application perform on Microsoft's CLR?

Mark
Comment 9 Sergey Zhukov 2013-02-06 19:00:51 UTC
I think Boehm could start to swap due to high memory fragmenting. I created small test application (client and server) and got "Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS" one time in client with Boehm. Client which sent random blocks of bytes with size of 100-300K started to use swap, but not too much before chashed with this message. 
Also I try to produce memory leaks in the server, but currently I could not manage this. Test application server consumes stable amount of memory and don't want to grow up. 

I don't have server with Microsoft OS to move production from linux to windows.
Comment 10 Mark Probst 2013-02-12 15:08:10 UTC
How do you know your application is not leaking memory?
Comment 11 Ludovic Henry 2017-07-07 19:06:59 UTC
If you can still reproduce with latest mono version, please feel free to reopen the bug. Thank you.