Bug 16735 - 10x slowdown between boehm -> sgen on this profile
Summary: 10x slowdown between boehm -> sgen on this profile
Status: RESOLVED FEATURE
Alias: None
Product: Runtime
Classification: Mono
Component: GC ()
Version: 3.2.x
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2013-12-11 11:09 UTC by Jonathan Shore
Modified: 2014-04-20 20:57 UTC (History)
6 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
the simplified application with the 10x disparity (1.79 MB, application/x-bzip2)
2013-12-11 11:09 UTC, Jonathan Shore
Details
The new test, faster and corrects excessive allocation (1.79 MB, application/x-bzip2)
2014-01-03 17:48 UTC, Jonathan Shore
Details
sgen run with mono --stats (6.53 KB, application/octet-stream)
2014-01-03 19:50 UTC, Jonathan Shore
Details
boehm run with mono --stats (5.19 KB, application/octet-stream)
2014-01-03 19:51 UTC, Jonathan Shore
Details
Crashes Mono/sgen when running in MonoDevelop (2.35 KB, application/x-gzip)
2014-04-16 06:10 UTC, Craig Minihan
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FEATURE

Description Jonathan Shore 2013-12-11 11:09:43 UTC
Created attachment 5637 [details]
the simplified application with the 10x disparity

As posted on the mono-revel-list, have an application that is run on 10 threads which uses 9+ cores with the boehm collector and approximately 2 cores with seen.  The total runtime has a nearly 10x difference. I am running this on a dual xeon E2630 with 12 real cores and 24 virtual on ubuntu 12.04 with a fairly recent build of mono (approximately 3.2.5).

I would like to be able to use sgen as run into the 32 bit barrier with boehm (which core dumps when i hit it).  With a 10x degradation on something that already takes 12 hours to run, cannot use seen currently.  Really appreciate an analyze on the working set of this app to see whether can determine a better profile for sgen or perhaps some tweak to the GC alg itself.

Running this on a smaller env on osx (8 threads) I saw that sgen did better than boehm (though only utilized 3 cores and boehm 1+ core).  The behavior on linux on 10 thread 2 cpus had the nice 10x profile for boehm and 2x for sgen.

I've included a scaled down application, where have faked the data sources, as an attachment.  There are 2 scripts:

- run-sgen
- run-boehm

these just run the application from the local dir with either sgen or boehm respectively.
Comment 1 Jonathan Shore 2013-12-11 11:11:25 UTC
sorry my osx auto-correct mangled the above test.  1st paragraph should read:

As posted on the mono-devel-list, have an application that is run on 10 threads
making use of 9+ cores with the boehm collector and only 2 cores with
sgen.  The total runtime has a nearly 10x difference. I am running this on a
dual xeon E2630 with 12 real cores and 24 virtual on ubuntu 12.04 with a fairly
recent build of mono (approximately 3.2.5).
Comment 2 Rodrigo Kumpera 2013-12-16 14:50:09 UTC
Your problem is that sgen is using the serial major collector and boehm on linux defaults to the parallel one.

You should try to enable it by setting MONO_GC_PARAMS=marksweep-par and see if it makes any difference. As from properly measuring it locally that's the only difference I see between the two.
Comment 3 Mark Probst 2013-12-18 19:51:00 UTC
How long are these supposed to run?
Comment 4 Jonathan Shore 2013-12-30 14:19:44 UTC
@Mark, with the included configuration, under boehm runs for 5377 secs and sgen 52912 secs.  I also tried MONO_GC_PARAMS='major=marksweep-par', with some improvement, but not much.

Boehm (default settings)
Performance counter stats for '/opt/mono-3.0/bin/mono-boehm --llvm /home/jonathan/Dev/hf/lib/Debug/FeatureGeneratorCSVFile.exe -info -config etc/samples/orderbook-2013-CX-V11.xml -out features-2013-CX.csv':

   48579862.522506 task-clock                #    9.034 CPUs utilized          
       188,866,824 context-switches          #    0.004 M/sec                  
            46,500 CPU-migrations            #    0.000 M/sec                  
         1,475,427 page-faults               #    0.000 M/sec                  
140,468,865,368,193 cycles                    #    2.892 GHz                    
   <not supported> stalled-cycles-frontend 
   <not supported> stalled-cycles-backend  
80,012,982,451,027 instructions              #    0.57  insns per cycle        
16,967,686,291,478 branches                  #  349.274 M/sec                  
    95,315,728,420 branch-misses             #    0.56% of all branches        

    5377.495775794 seconds time elapsed

SGen (default settings)
Performance counter stats for '/opt/mono-3.0/bin/mono-sgen --llvm /home/jonathan/Dev/hf/lib/Debug/FeatureGeneratorCSVFile.exe -info -config etc/samples/orderbook-2013-CX-V11.xml -out features-2013-CX.csv':

  108414200.651113 task-clock                #    2.049 CPUs utilized          
        65,792,604 context-switches          #    0.001 M/sec                  
            30,536 CPU-migrations            #    0.000 M/sec                  
       309,928,477 page-faults               #    0.003 M/sec                  
263,506,866,481,917 cycles                   #    2.431 GHz                    
   <not supported> stalled-cycles-frontend 
   <not supported> stalled-cycles-backend  
130,560,004,191,686 instructions             #    0.50  insns per cycle        
27,570,367,199,486 branches                  #  254.306 M/sec                  
   382,673,241,515 branch-misses             #    1.39% of all branches        

   52912.358974732 seconds time elapsed
Comment 5 Mark Probst 2014-01-02 13:17:05 UTC
The first thing I notice here is that the SGen run triggers around 200 times as many page faults, which is likely an indication that it's running out of RAM and using swap, which would obviously slows everything down.  That might explain a significant part of the difference in run time.
Comment 6 Jonathan Shore 2014-01-02 14:51:27 UTC
@Mark, actually this machine has 128GB of memory and this process is using < 2GB (on boehm) and probably no more than 2.9GB with sgen.   So for sure we are not going to swap here.   Lots of free memory.  The boehm version cannot even use more than 2^31 (or is it less).
Comment 7 Mark Probst 2014-01-02 18:12:37 UTC
Running on Boehm on 64 bits, after 2h 40m, I get

schani@ubuntu:~/Work/mono/bugs/bug16735/test$ time ./run-boehm
Mono Warning: --llvm not enabled in this runtime.
INFO  [2014-01-02 11:33:19.865]: loading config
INFO  [2014-01-02 11:33:19.930]: generating & writing features to: /dev/null
INFO  [2014-01-02 11:33:20.976]: generating features for: 2013-3-7
INFO  [2014-01-02 11:33:20.976]: generating features for: 2013-3-29
INFO  [2014-01-02 11:33:20.979]: generating features for: 2013-3-14
INFO  [2014-01-02 11:33:20.979]: generating features for: 2013-3-20
Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS

...
Comment 8 Jonathan Shore 2014-01-02 18:27:18 UTC
I have an artificial heap alloc in the test to simulate the data buffers used in against a massive time series DB we have.  I can remove these allocs and see whether helps.   Can also reduce the dates in the config to the # of threads to reduce the runtime. 

I'll provide a new test without the allocs and also run a 4 thread test on a smaller config and provide a comparison.
Comment 9 Jonathan Shore 2014-01-03 17:47:16 UTC
Ok, I adjusted the test program. I had a bug in the scaled down version which did more allocation then it should have.   Now the test (with 4 dates on 4 cores runs in ~10mins with boehm), however, it takes 2x as long with sgen.

I modified the test config to include more dates and run on 8 cores.  sgen was only able to use 2-3 cores on average and boehm 7-8 cores.   I changed this back to 4 cores, since you seem to be testing on a system with fewer cores.

I will upload a new test which runs property and should duplicate the results below.  The tar is called:  test-revised.tar.bz2

Here are the perf stats:

-- sgen --
    3521837.766373 task-clock                #    2.323 CPUs utilized          
         1,132,129 context-switches          #    0.000 M/sec                  
             1,175 CPU-migrations            #    0.000 M/sec                  
         6,474,002 page-faults               #    0.002 M/sec                  
 9,611,027,836,737 cycles                    #    2.729 GHz                    
   <not supported> stalled-cycles-frontend 
   <not supported> stalled-cycles-backend  
 6,692,832,113,098 instructions              #    0.70  insns per cycle        
 1,368,218,488,590 branches                  #  388.496 M/sec                  
    14,441,003,383 branch-misses             #    1.06% of all branches        

    1516.316481703 seconds time elapsed

-- boehm --
    3181953.581850 task-clock                #    4.004 CPUs utilized          
           885,443 context-switches          #    0.000 M/sec                  
            11,125 CPU-migrations            #    0.000 M/sec                  
           365,322 page-faults               #    0.000 M/sec                  
 9,239,060,598,468 cycles                    #    2.904 GHz                    
   <not supported> stalled-cycles-frontend 
   <not supported> stalled-cycles-backend  
 5,205,787,599,872 instructions              #    0.56  insns per cycle        
 1,076,025,922,754 branches                  #  338.165 M/sec                  
     5,494,832,381 branch-misses             #    0.51% of all branches        

     794.708046875 seconds time elapsed
Comment 10 Jonathan Shore 2014-01-03 17:48:58 UTC
Created attachment 5753 [details]
The new test, faster and corrects excessive allocation
Comment 11 Jonathan Shore 2014-01-03 17:50:44 UTC
Meant to add to the above test, for 4 cores, the program is ~2x slower with sgen, for 8 cores, approximately 4x slower.
Comment 12 Rodrigo Kumpera 2014-01-03 18:14:58 UTC
Jonathan, can you run mono with --stats? This will give us some key metrics between the two GCs that can easily help us discard some possibilities.
Comment 13 Jonathan Shore 2014-01-03 18:57:25 UTC
Sure. will give it a spin.
Comment 14 Jonathan Shore 2014-01-03 19:50:58 UTC
Created attachment 5754 [details]
sgen run with mono --stats
Comment 15 Jonathan Shore 2014-01-03 19:51:38 UTC
Created attachment 5755 [details]
boehm run with mono --stats
Comment 16 Jonathan Shore 2014-01-03 19:52:19 UTC
I loaded the logs of the mono --stats for sgen and boehm respectively
Comment 17 Rodrigo Kumpera 2014-01-04 00:29:40 UTC
The log shows exactly what I feared, your workload is not very generational.

The amount of time spent scanning the remembered set dominates minor collection time.

Maybe we're missing some opportunity onto how to better organize the major heap to reduce this
problem, but other than that, there's nothing much that can be done.
Comment 18 Jonathan Shore 2014-01-04 09:44:05 UTC
By remembered set, I assume these are the collection of objects from a previous generation which have been moved to the main heap?   I assume then that the references in the graph representing the remembered set have to be scanned to determine which objects in the nursery need to be kept?

With regard to "coloring", this seems analogous to what boehm does in marking objects as reachable.  I assume that Boehm also explores the graph from known roots.

Why is this scanning with sgen more expensive than with boehm?    Speculating here, I am going to guess this is because the nurseries are small.   Perhaps with one heap space, Boehm decides to GC less frequently, depending on overall heap growth?

Could we configure the nursery to be much bigger (say 32MB) and reduce the # of GC invocations?  I vaguely recall that you have some ratio of nursery to overall initial heap size, which might be affected.

Finally, if such a reconfiguration of sgen is impossible or not likely to be successful, does the 64bit heap option of Boehm work?   I had tried the large-heap option in the build some versions ago and found that it was not stable.
Comment 19 Jonathan Shore 2014-01-04 09:49:01 UTC
I meant to say: Speculating here, I am guessing that because the nurseries are small we do more "object reachable" scans with sgen than with boehm for this application.  It seems in this environment, if the # of live objects in the main heap is large, a frequent nursery cleanup is going to be very expensive.   Doing less frequent GC would be a win in this scenario.
Comment 20 Rodrigo Kumpera 2014-01-08 21:00:28 UTC
The remembered set is the set of objects in the old generation that points to objects in the young generation.

When performing a minor collection, all of those old objects must be scanned to have their pointers updated to the new location of the young objects they point.

As the size of this remembered set grows, the less profitable generational collections are. From your longs, it's clear that this is the case since the majority of the collection time is spend scanning them.

One solution for this is to increase the nursery to be large enough so the relative size of the remembered set doesn't overburden it. Another one is to restructure the code to be more amendable to generational collections.

Besides this, sgen minor collections are not parallel, this is what explain why wallclock time is so bigger but cycle count difference is not that big.

My suggestion is that you increase the nursery size until it no longer improves your performance/footprint.
This and enabling the parallel major collection should be enough to produce a reasonable improvement.

Boehm has a very scalable parallel collector, so it might just be the case that sgen's one won't be able to beat it for some time.

Besides that, writing a parallel nursery collector could possibly help here too. We have no current plans for such a thing at Xamarin.
Comment 21 Jonathan Shore 2014-01-09 14:03:04 UTC
Fair enough.  This is not all that relevant to the mobile business.   I will play with nursery size and see how that helps.   Otherwise have to see whether the large heap boehm is workable.
Comment 22 Craig Minihan 2014-04-11 19:17:47 UTC
I know this is closed but: I see something similar to Jonathan's behaviour in my app.

Generally sgen doesn't survive a test pass where Boehm is flawless: much faster and uses less memory.

I'm aiming to run a Mono based app on a 20+ core 256GB+ RAM server. I'm currently using 20GB+ RAM in testing which Boehm is very happy with. It would be nice to have an sgen impl capable of taking this kind of load.

I've run some passes on MS .NET 4.5 and can see their GC is much more aggressive than either Boehm or sgen. Memory usage is lower by about 10-15% as well. On a prod platform with 100GB+ RAM that equates to a lot of RAM.

It this becomes critical to me I'll run some analysis on sgen directly at the C level to see what is going on.
Comment 23 Jonathan Shore 2014-04-11 19:41:35 UTC
@Craig.  I never did get a good outcome with any configuration of sgen.   I would hope that could get a version of boehm that can use a larger heap.  As the boehm heap in mono is restricted in size.
Comment 24 Craig Minihan 2014-04-11 19:49:06 UTC
What size restriction do you see? I assume you built with --with-large-heap=yes?
Comment 25 Jonathan Shore 2014-04-11 19:51:56 UTC
I found --with-large-heap=yes to be unstable in the past, so have not been using it.   Perhaps it works properly now.  Do you use it?   

If i recalls, when a boehm heap tries to exceed 2.xg, a boehm based VM will show a VM error and die (this on regular heap).
Comment 26 Craig Minihan 2014-04-11 20:18:58 UTC
I've not seen any stability issues with Boehm with large-heap enabled. I've run the same test on the following platforms:

1) CentOS 6.4 - Mono 64bit - sgen - core dumps
2) CentOS 6.4 - Mono 64bit - boehm - completes
3) Windows - .NET 4.5 64bit - completes
4) OSX 10.8 - Mono 64bit - sgen - completes, but is very slow (could be comms or old cpu here)

I had Boehm running up to 40GB a while back since I have a large source data set - it just takes a while to get to this size so it isn't something I run every day.

I've tested on 3.2.3, 3.2.8 and a cherry picked 3.2.8 branch with all sgen fixes submitted after 3.2.8. They all show the same issue.

I've played with sgen settings via the env variable and can cause it to crash in a couple of seconds of app startup depending on the params I supply. Using defaults it can take anywhere from 2mins - 10mins to crash.
Comment 27 Jonathan Shore 2014-04-11 20:23:42 UTC
good to know re: large heap.  I will give it a shot.   You may want to submit your test and see whether it interests the mono folks towards making sgen improvements.   Do you have a large # of (small) objects that are held long term in the main heap?   That is the situation I have.   

The cost of the GC scan on hundreds of thousands of objects is substantial.   I've written java code with similar characteristics, but not had that sort of problem with the GC.  Perhaps needs a 3rd tier in the GC for long-lived objects.
Comment 28 Rodrigo Kumpera 2014-04-14 08:57:09 UTC
Jonathan, Sgen has a very similar setup to HotSpot default collectors. We even have an optional aging nursery.

Craig, unless you provide us with backtraces and test case, there's absolutely nothing we can do to fix your crash. GC crashes can look very similar to the untrained eye.
Comment 29 Craig Minihan 2014-04-14 10:03:00 UTC
Rodrigo, I've been living with Boehm for the moment - but this isn't something I'm keen on in Prod. The good news: I can crash Mono/Sgen pretty much at will, I get lots of stack traces (from the many threads) always ending with this:

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

Running the same code with the Boehm GC doesn't show the issue. Neither does it show on .NET (running the same IL).

I'd have to scrub the trace and core dumps before I can make them available - I've just not had time for this yet.

The most likely cause of the crashes are:
1) An app level issue which only manifests itself when sgen is active
2) A native lib issue
3) A genuine sgen issue

The reason I ended up commenting on this issue is because the sgen/boehm behaviour I'm seeing is strikingly similar to what Jonathan reported (performance, threads, etc).

I'm running CentOS 6.4 x64 so I can rule out native libs by running on Mono/Windows, Mono/OSX and OpenSUSE which you guys supply the binaries for - these are all on my list. I'll report back with more detail soon.
Comment 30 Mark Probst 2014-04-14 14:26:24 UTC
I got this on OSX with 64 bit:

./run-boehm  7636.94s user 3158.23s system 267% cpu 1:07:14.22 total
./run-sgen  3115.04s user 69.26s system 238% cpu 22:13.73 total

In other words, Boehm is three times slower than SGen.
Comment 31 Craig Minihan 2014-04-14 14:42:49 UTC
Mark, I can't see these scripts in mono/mono in github. If you send them to me I can run them thru on my side.

TBH Boehm isn't performing badly at all, so more speed will make me a very happy bunny!
Comment 32 Mark Probst 2014-04-14 15:24:36 UTC
These scripts just run Jonathan's test case with Boehm and SGen, respectively, no special options at all.

run-boehm:

$HOME/Work/mono/mono/mono/mini/mono-boehm --stats lib/FeatureGeneratorCSVFile.exe -info -config test.xml -out /dev/null

run-sgen:

$HOME/Work/mono/mono/mono/mini/mono-sgen --stats lib/FeatureGeneratorCSVFile.exe -info -config test.xml -out /dev/null
Comment 33 Rodrigo Kumpera 2014-04-14 15:39:31 UTC
The performance issue here is quite straighforward.

Boehm on linux uses a very well tuned parallel collector. 
Sgen doesn't have a parallel nursery collector and the old gen collector parallel mode is not enabled by default.

As Jonathan mentioned above, using the parallel collector did help a bit, but given our implementation is not as optimized as Boehm's it's to be expected to be slower.
Comment 34 Mark Probst 2014-04-14 17:14:34 UTC
As far as I can tell the issue here is quite different.  Here's what I get when running this on Linux/AMD64 (on VMWare):

boehm

Minor GC collections                : 0
Major GC collections                : 1895
Minor GC time                       : 0.00 ms
Major GC time                       : 97706.61 ms

real	15m17.131s
user	56m11.088s
sys	0m53.332s

sgen

Minor GC collections                : 50659
Major GC collections                : 474
Minor GC time                       : 635462.12 ms
Major GC time                       : 93245.80 ms

real	24m21.718s
user	51m24.356s
sys	5m22.612s

SGen uses pretty much the same amount of CPU time as Boehm, but since the workload is very parallel, the many minor GC interruptions, which only utilize one CPU, slow us down a lot.  Using a large nursery, 64MB in this case, ameliorates the problem:

Minor GC collections                : 3085
Major GC collections                : 104
Minor GC time                       : 146279.40 ms
Major GC time                       : 30087.86 ms

real	14m50.709s
user	48m46.860s
sys	0m53.496s
Comment 35 Rodrigo Kumpera 2014-04-14 17:19:22 UTC
So it's a matter of lack of user tuning? Awesome.

We should look into adding GC ergonomics for the 64bits VM so it can automatically scale up the nursery size when in performance mode.

Still there's the issue of crashes.
Comment 36 Craig Minihan 2014-04-14 17:33:12 UTC
My results:
* Boehm: 561.470464126 seconds time elapsed
* Sgen: 904.595372987 seconds time elapsed
* Sgen/64m nursery: 561.322264625 seconds time elapsed

So inline with Mark's. I'll run against my crashing test case to see what results we get.
Comment 37 Craig Minihan 2014-04-14 18:57:44 UTC
I'm not seeing a SIGSEGV right now - so I'll revert out some changes to see if I can repl that. I'm seeing this tho:

With sgen defaults:
Error: Garbage collector could not allocate 16384 bytes of memory for major heap section.

The system has 32GB of RAM, at the point this message appeared process memory was <<< 10GB.

And then with MONO_GC_PARAMS=nursery-size=64m, app completes, pressing ctrl-c to terminate gives:
* Assertion at domain.c:1044, condition `chunk' not met
Stacktrace:
* Assertion at mini-exceptions.c:824, condition `unwind_options == MONO_UNWIND_NONE' not met
Aborted (core dumped)

Strangely the amount of memory used by the process under sgen was much higher than Boehm, 9GB to 14GB respectively.

I know this isn't much to go on at the moment - the good news was the app got to these conditions much quicker with a larger nursery size so it will be easier to narrow down over multiple passes.
Comment 38 Craig Minihan 2014-04-15 19:51:22 UTC
Ok, so here is a stack trace - caveats at the bottom:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

 Stacktrace:


Native stacktrace:

	/usr/local/bin/mono() [0x4a2a35]
	/usr/local/bin/mono() [0x4fe30b]
	/usr/local/bin/mono() [0x4154f9]
	/lib64/libpthread.so.0() [0x37e060f500]
	/usr/local/bin/mono() [0x5d445b]
	/usr/local/bin/mono() [0x5d4a54]
	/usr/local/bin/mono() [0x5cae5f]
	/usr/local/bin/mono() [0x5cb090]
	/usr/local/bin/mono() [0x5ce5ef]
	/usr/local/bin/mono() [0x5d1424]
	/usr/local/bin/mono() [0x5d15f8]
	/usr/local/bin/mono() [0x5e0e53]
	/usr/local/bin/mono() [0x5e9313]
	/usr/local/bin/mono() [0x5e987c]
	[0x41767f0c]

Debug info from gdb:

Mono support loaded.
[New LWP 18790]
[New LWP 18789]
[New LWP 18788]
[New LWP 18786]
[New LWP 18785]
[New LWP 18784]
[New LWP 18783]
[New LWP 18782]
[New LWP 18781]
[New LWP 18780]
[New LWP 18779]
[New LWP 18778]
[New LWP 18777]
[New LWP 18776]
[New LWP 18775]
[New LWP 18774]
[New LWP 18773]
[New LWP 18772]
[New LWP 18771]
[New LWP 18770]
[New LWP 18760]
[New LWP 18759]
[Thread debugging using libthread_db enabled]
0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  23 Thread 0x7fd2090a3700 (LWP 18759)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  22 Thread 0x7fd208ea2700 (LWP 18760)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  21 Thread 0x7fd208ca1700 (LWP 18770)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  20 Thread 0x7fd208aa0700 (LWP 18771)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  19 Thread 0x7fd20889f700 (LWP 18772)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  18 Thread 0x7fd2085ea700 (LWP 18773)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  17 Thread 0x7fd20839f700 (LWP 18774)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  16 Thread 0x7fd1ebe52700 (LWP 18775)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  15 Thread 0x7fd1ebc51700 (LWP 18776)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  14 Thread 0x7fd1eba50700 (LWP 18777)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  13 Thread 0x7fd1eb84f700 (LWP 18778)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  12 Thread 0x7fd1eb64e700 (LWP 18779)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  11 Thread 0x7fd1eb44d700 (LWP 18780)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  10 Thread 0x7fd1eb24c700 (LWP 18781)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  9 Thread 0x7fd1eb04b700 (LWP 18782)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  8 Thread 0x7fd1ea963700 (LWP 18783)  0x00000037e060f09d in waitpid () from /lib64/libpthread.so.0
  7 Thread 0x7fd1e92ff700 (LWP 18784)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  6 Thread 0x7fd1c3bff700 (LWP 18785)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  5 Thread 0x7fd1c2fff700 (LWP 18786)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  4 Thread 0x7fd1c2dfe700 (LWP 18788)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  3 Thread 0x7fd1c23ff700 (LWP 18789)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
  2 Thread 0x7fd1c21fe700 (LWP 18790)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
* 1 Thread 0x7fd210a8d780 (LWP 18757)  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6

Thread 23 (Thread 0x7fd2090a3700 (LWP 18759)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd2090a2780) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd2090a2780) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060d71e in sem_wait () from /lib64/libpthread.so.0
#5  0x0000000000623ed8 in mono_sem_wait (sem=0x96b360, alertable=1) at mono-semaphore.c:119
#6  0x00000000005a1d85 in finalizer_thread (unused=<value optimized out>) at gc.c:1073
#7  0x000000000057fd77 in start_wrapper_internal (data=0x2276cc0) at threads.c:643
#8  start_wrapper (data=0x2276cc0) at threads.c:688
#9  0x0000000000617e73 in thread_start_routine (args=0x21f1318) at wthreads.c:294
#10 0x0000000000628dd0 in inner_start_thread (arg=0x2276dd0) at mono-threads-posix.c:49
#11 0x00000037e0607851 in start_thread () from /lib64/libpthread.so.0
#12 0x00000037dfee890d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fd208ea2700 (LWP 18760)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd208ea1580) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd208ea1580) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e94a in recv () from /lib64/libpthread.so.0
#5  0x00000000004d02f9 in socket_transport_recv (buf=0x7fd208ea1d80, len=11) at debugger-agent.c:1085
#6  0x00000000004d3fda in transport_recv (arg=<value optimized out>) at debugger-agent.c:1475
#7  debugger_thread (arg=<value optimized out>) at debugger-agent.c:9139
#8  0x0000000000617e73 in thread_start_routine (args=0x21f13e0) at wthreads.c:294
#9  0x0000000000628dd0 in inner_start_thread (arg=0x2278740) at mono-threads-posix.c:49
#10 0x00000037e0607851 in start_thread () from /lib64/libpthread.so.0
#11 0x00000037dfee890d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fd208ca1700 (LWP 18770)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd208ca0340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd208ca0340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=16512, max_length=2060) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd200002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7fd208aa0700 (LWP 18771)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd208a9f280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd208a9f280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Int32[]"), size=8904, max_length=2217) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1f4002540 in ?? ()
#11 0x00007fd208aa06e0 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7fd20889f700 (LWP 18772)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd20889e300) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd20889e300) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Object[]"), size=14688, max_length=1832) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1f8002540 in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7fd2085ea700 (LWP 18773)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd2085e9300) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd2085e9300) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Object[]"), size=12936, max_length=1613) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1ec002540 in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fd20839f700 (LWP 18774)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd20839e200) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd20839e200) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Object[]"), size=9928, max_length=1237) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1f0002540 in ?? ()
#11 0x000000000000001f in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fd1ebe52700 (LWP 18775)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1ebe51340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1ebe51340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=14280, max_length=1781) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1e4002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fd1ebc51700 (LWP 18776)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1ebc50280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1ebc50280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Collections.Generic.Link[]"), size=14288, max_length=1782) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1dc002540 in ?? ()
#11 0x00007fd1ebc50a60 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fd1eba50700 (LWP 18777)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eba4f340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eba4f340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=13968, max_length=1742) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1e0002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fd1eb84f700 (LWP 18778)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb84e340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb84e340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=13232, max_length=1650) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1d4002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fd1eb64e700 (LWP 18779)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb64d340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb64d340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=13896, max_length=1733) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1d8002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fd1eb44d700 (LWP 18780)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb44c280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb44c280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Collections.Generic.Link[]"), size=16680, max_length=2081) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1cc002540 in ?? ()
#11 0x00007fd1eb44ca60 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fd1eb24c700 (LWP 18781)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb24b340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb24b340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=12376, max_length=1543) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1d0002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fd1eb04b700 (LWP 18782)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb04a280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1eb04a280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Collections.Generic.Link[]"), size=14040, max_length=1751) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1c4002540 in ?? ()
#11 0x00007fd1eb04aa60 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fd1ea963700 (LWP 18783)):
#0  0x00000037e060f09d in waitpid () from /lib64/libpthread.so.0
#1  0x00000000004a2ac8 in mono_handle_native_sigsegv (signal=<value optimized out>, ctx=<value optimized out>) at mini-exceptions.c:2299
#2  0x00000000004fe30b in mono_arch_handle_altstack_exception (sigctx=0x7fd208057ac0, fault_addr=<value optimized out>, stack_ovf=0) at exceptions-amd64.c:908
#3  0x00000000004154f9 in mono_sigsegv_signal_handler (_dummy=11, info=0x7fd208057bf0, context=0x7fd208057ac0) at mini.c:6769
#4  <signal handler called>
#5  sgen_par_object_get_size (ptr=0x7fd2090b9208, obj=0x7fd1c287f018, queue=0x96c260) at ../../mono/metadata/sgen-gc.h:745
#6  sgen_safe_object_get_size (ptr=0x7fd2090b9208, obj=0x7fd1c287f018, queue=0x96c260) at ../../mono/metadata/sgen-gc.h:777
#7  major_copy_or_mark_object (ptr=0x7fd2090b9208, obj=0x7fd1c287f018, queue=0x96c260) at sgen-marksweep.c:1398
#8  0x00000000005d4a54 in major_scan_object (start=<value optimized out>, queue=0x96c260) at sgen-scan-object.h:64
#9  0x00000000005cae5f in sgen_drain_gray_stack (max_objs=-1, ctx=...) at sgen-gc.c:1194
#10 0x00000000005cb090 in precisely_scan_objects_from (worker_data=<value optimized out>, job_data_untyped=0x7fd20ab8cfb0) at sgen-gc.c:1601
#11 scan_from_registered_roots (worker_data=<value optimized out>, job_data_untyped=0x7fd20ab8cfb0) at sgen-gc.c:2055
#12 job_scan_from_registered_roots (worker_data=<value optimized out>, job_data_untyped=0x7fd20ab8cfb0) at sgen-gc.c:2345
#13 0x00000000005ce5ef in major_copy_or_mark_from_roots (old_next_pin_slot=0x7fd1ea96271c, finish_up_concurrent_mark=0, scan_mod_union=0) at sgen-gc.c:3000
#14 0x00000000005d1424 in major_do_collection (reason=0x6fd32e "LOS overflow") at sgen-gc.c:3304
#15 0x00000000005d15f8 in sgen_perform_collection (requested_size=8352, generation_to_collect=1, reason=0x6fd32e "LOS overflow", wait_to_finish=0) at sgen-gc.c:3499
#16 0x00000000005e0e53 in sgen_los_alloc_large_inner (vtable=vtable("System.Int32[]"), size=8352) at sgen-los.c:346
#17 0x00000000005e9313 in mono_gc_alloc_obj_nolock (vtable=vtable("System.Int32[]"), size=<value optimized out>) at sgen-alloc.c:204
#18 0x00000000005e987c in mono_gc_alloc_vector (vtable=vtable("System.Int32[]"), size=8352, max_length=2079) at sgen-alloc.c:491
#19 0x0000000041767f0c in ?? ()
#20 0x0000000040803000 in ?? ()
#21 0x00007fd1c8002540 in ?? ()
#22 0x00007fd1ea962a60 in ?? ()
#23 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fd1e92ff700 (LWP 18784)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1e92fe280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1e92fe280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Object[]"), size=11288, max_length=1407) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1bc002540 in ?? ()
#11 0x00007fd1e92fea60 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fd1c3bff700 (LWP 18785)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c3bfe340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c3bfe340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=9736, max_length=1213) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1b8002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fd1c2fff700 (LWP 18786)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c2ffe300) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c2ffe300) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Object[]"), size=8904, max_length=1109) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1b0002540 in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fd1c2dfe700 (LWP 18788)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c2dfd340) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c2dfd340) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Double[]"), size=8128, max_length=1012) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1b4002540 in ?? ()
#11 0x0000000040803000 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fd1c23ff700 (LWP 18789)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c23fe280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c23fe280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Object[]"), size=8872, max_length=1105) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1a8002540 in ?? ()
#11 0x000000000097e560 in sgen_nursery_start ()
#12 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fd1c21fe700 (LWP 18790)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c21fd280) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fd1c21fd280) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005e9871 in mono_gc_alloc_vector (vtable=vtable("System.Collections.Generic.Link[]"), size=8616, max_length=1073) at sgen-alloc.c:489
#8  0x0000000041767f0c in ?? ()
#9  0x0000000040803000 in ?? ()
#10 0x00007fd1ac002540 in ?? ()
#11 0x000000000097e560 in sgen_nursery_start ()
#12 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fd210a8d780 (LWP 18757)):
#0  0x00000037dfe32c54 in sigsuspend () from /lib64/libc.so.6
#1  0x00000000005c76f4 in suspend_thread (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fff306b6a40) at sgen-os-posix.c:113
#2  suspend_handler (sig=<value optimized out>, siginfo=<value optimized out>, context=0x7fff306b6a40) at sgen-os-posix.c:131
#3  <signal handler called>
#4  0x00000037e060e052 in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00000037e0609388 in _L_lock_854 () from /lib64/libpthread.so.0
#6  0x00000037e0609257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005c9bb9 in mono_gc_register_root_inner (start=0x244e780 "", size=24, descr=0x0, root_type=1) at sgen-gc.c:3924
#8  0x00000000005e8e1c in mono_gc_alloc_fixed (size=24, descr=0x0) at sgen-alloc.c:627
#9  0x00000000005a4273 in new_slot (hash=0x2276d50, key=0x7fd20a441030, value=0x7fd20a592c38, replace=0) at mono-hash.c:85
#10 mono_g_hash_table_insert_replace (hash=0x2276d50, key=0x7fd20a441030, value=0x7fd20a592c38, replace=0) at mono-hash.c:451
#11 0x000000000057de62 in create_thread (thread=0x7fd20a441030, internal=0x7fd20911e950, start_info=0x2457020, threadpool_thread=0, no_detach=0, stack_size=0, throw_on_failure=0) at threads.c:721
#12 0x000000000057f4c0 in ves_icall_System_Threading_Thread_Thread_internal (this=0x7fd20a441030, start=<value optimized out>) at threads.c:1171
#13 0x0000000040f2b964 in ?? ()
#14 0x0000000040803000 in ?? ()
#15 0x000000000226d460 in ?? ()
#16 0x00007fd20a441030 in ?? ()
#17 0x000000000227d290 in ?? ()
#18 0x00007fff306b7350 in ?? ()
#19 0x00007fff306b71c0 in ?? ()
#20 0x00000000021fd920 in ?? ()
#21 0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

This is 100% reproducible with: Mono 3.2.8 (tarball), MonoDevelop 4.2.2 running the process in release mode under the debugger. Every single thread shows sgen in its stack; I guess thread #8 looks the most suspect. The crash happens about 1-2s into the process.

Rodrigo/Mark, this is test code so I'm happy to post here - let me know.
Comment 39 Miguel de Icaza [MSFT] 2014-04-15 21:55:22 UTC
Craig,

If you have a reliable way of crashing with a test code, please post it here.
Comment 40 Craig Minihan 2014-04-16 06:10:18 UTC
Created attachment 6590 [details]
Crashes Mono/sgen when running in MonoDevelop

This code should cause a crash in Mono when run in the debugger under MonoDevelop.

Platform:
* CentOS 6.4 x64
* Mono 3.2.8 (tarball)
* MonoDevelop 4.2.2
* CPU: i7, 8GB RAM

Build it, run it (in the debugger), crash. There is a strong relationship between the number of threads started and the SIGSEGV, too few and all is well. Setting above ProcessorCount * 3 causes the exception in my env at least.
Comment 41 Miguel de Icaza [MSFT] 2014-04-20 20:57:44 UTC
I opened a new bug to track the sgen/debugger crash:

#19181