Notice (2018-05-24): bugzilla.xamarin.com is now in
Please join us on
Visual Studio Developer Community and in the
Mono organizations on
GitHub to continue tracking issues. Bugzilla will remain
available for reference in read-only mode. We will continue to work
on open Bugzilla bugs, copy them to the new locations
as needed for follow-up, and add the new items under Related
Our sincere thanks to everyone who has contributed on this bug
tracker over the years. Thanks also for your understanding as we
make these adjustments and improvements for the future.
Please create a new report on
GitHub or Developer Community with
your current version information, steps to reproduce, and relevant error
messages or log files if you are hitting an issue that looks similar to
this resolved bug and you do not yet see a matching new report.
Created attachment 3326 [details]
I often get a mono crash in my async socket server in production environment. The crashes perform randomly generally when high load (more then 500 socket clients connected and ~95% memory used), and as I think they are related to operations of allocation or collection of objects in GC. I don't use unsafe or unmanaged code or interops, all code is 100% managed. Unfortunately, such randomness does not allow me to create small reproducible test-case and what can I do at this moment it's just to attach stack traces of faults. I hope, these stack traces help someone, who will meet such errors and run into the issue. If I can to recreate reproducible test-case I will attach it.
Mono JIT compiler version 184.108.40.206 (Debian 220.127.116.11-1ubuntu2.2)
OS: Ubuntu 12.04
command-line: mono-sgen ./myserver.exe
There are three different stack traces in the file.
These look like three separate issues, though they might have the same ultimate cause.
The first one is an OOM crash, which you've filed as #10005.
The second one looks like an exception handling problem, related to stack walking.
The third one is an SGen crash which doesn't seem caused by OOM.
We won't be able to do much without reproducible test cases. Also, could you please try Mono 3.0.3?
It's not easy to transfer production server to mono 3.0.3, because this operation requires additional hardware and retesting whole application. I am going to do it, but can't do it right now. By the way, I ran the sample provided in the #10005 in mono 3.0.4, and added the comment.
What if you run on 2.10 with Boehm instead of SGen?
When I run my server with Boehm it starts to use swap memory, perfomance drops alot and my clients get lags and choose to disconnect from the server. So I cannot reach critical number of clients to check what's going on. I don't know how to tell mono with Boehm not to use swap while allocating memory. Also it seems that sgen behaviour in mono 3.0.4 similar to Boehm in 2.10.8 (gc tries to allocate memory in swap, when sgen gc in 2.10.8 does not allocate memory in swap)
Neither SGen nor Boehm allocate memory in swap. They might allocate too much memory, which would then end up in swap. That would point to a memory leak somewhere. Your first backtrace also indicates that, though, and as far as I understood that was 2.10 with SGen?
Stacktraces in this issue were for 2.10.8 with sgen gc.
In bug #10005 I provided test case, which tries to allocate whole memory by 100K chunks. The results were:
mono 18.104.22.168 Boehm - at first all free memory was allocated, than started to allocate memory from swap. After long time of swapping and awaiting the final message (couple of hours), I killed the program. Will try it again later on virtual machine with small swap.
mono 22.214.171.124 Sgen - program allocated all free memory and crashed
mono 3.0.4 Sgen - at first program allocated all free memory, than all swap memory and than throwed OutOfMemoryException
Yes, but that's not the point.
The first backtrace here indicates that you're running into OOM on your server application, with Mono 2.10.8 SGen. That and the fact that Boehm starts requiring swap as well, suggests that your application has a memory leak.
How does your application perform on Microsoft's CLR?
I think Boehm could start to swap due to high memory fragmenting. I created small test application (client and server) and got "Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS" one time in client with Boehm. Client which sent random blocks of bytes with size of 100-300K started to use swap, but not too much before chashed with this message.
Also I try to produce memory leaks in the server, but currently I could not manage this. Test application server consumes stable amount of memory and don't want to grow up.
I don't have server with Microsoft OS to move production from linux to windows.
How do you know your application is not leaking memory?
If you can still reproduce with latest mono version, please feel free to reopen the bug. Thank you.