Bug 39563 - Error: Garbage collector could not allocate 16384 bytes of memory for major heap section ; even though 50 GB of memory are free
Summary: Error: Garbage collector could not allocate 16384 bytes of memory for major h...
Status: RESOLVED ANSWERED
Alias: None
Product: Runtime
Classification: Mono
Component: GC ()
Version: 4.2.0 (C6)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-03-11 22:05 UTC by Christian Hüning
Modified: 2016-10-06 15:24 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Reproduction code (982 bytes, application/octet-stream)
2016-03-11 22:05 UTC, Christian Hüning
Details
strace log file (5.07 MB, text/plain)
2016-03-11 23:16 UTC, Christian Hüning
Details
strace -e mmap logifle with complete error (9.89 KB, text/plain)
2016-03-11 23:29 UTC, Christian Hüning
Details
Couldnt create Thread exception after mmap errors (9.77 KB, text/plain)
2016-03-12 07:44 UTC, Christian Hüning
Details
strace mmap file with exact errors as in headline (66.08 KB, text/plain)
2016-03-12 08:04 UTC, Christian Hüning
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED ANSWERED

Description Christian Hüning 2016-03-11 22:05:36 UTC
Created attachment 15381 [details]
Reproduction code

The attached repo code fills up my memory up to 30-35 GB, then Mono reports the following error and dies: 
Error: Garbage collector could not allocate 16384 bytes of memory for major heap section.

Hardware: CPU 24 Cores, RAM 96 GB
Executed via: Docker 1.10.2
OS: Linux Ubuntu 15.10
Kernel Version: 4.2.0-30-generic
Mono: Mono JIT compiler version 4.5.0 (Nightly 4.5.0.262/2ed8402 Thu Mar 10 19:38:40 UTC 2016)
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

also tried with: Mono 4.2.3 and Mono 4.3.2.467

Tried with default MONO_GC_PARAMS and with MONO_GC_PARAMS="nursery-size=2g" . Outcome is the same.
Comment 1 Christian Hüning 2016-03-11 23:16:01 UTC
Created attachment 15382 [details]
strace log file
Comment 2 Christian Hüning 2016-03-11 23:29:34 UTC
Created attachment 15383 [details]
strace -e mmap logifle with complete error
Comment 3 Christian Hüning 2016-03-12 07:44:09 UTC
Created attachment 15384 [details]
Couldnt create Thread exception after mmap errors
Comment 4 Christian Hüning 2016-03-12 08:04:27 UTC
Created attachment 15385 [details]
strace mmap file with exact errors as in headline
Comment 5 Christian Hüning 2016-03-16 17:51:38 UTC
It actually turned out that it has been a problem with the NUMA based architecture of the computer I ran the application on. The application ran inside of a docker container, which was running on top of a VM, which was scheduled to use 96 GB and 24 cores of a NUMA based compute node. Turned out that Linux KVM as my Hypervisor uses a "strict mode" for NUMA nodes, that limits the usable amount of memory per node to the node's own memory. This means the VM with 24 cores can only use 48GB of memory and silently fails with not being able to allocate more memory.

However the Mono Test codes fails with the same error, when being run on the native NUMA machine. So could there be a possible problem with NUMA architecture and Mono?
Comment 6 Christian Hüning 2016-03-18 11:46:18 UTC
I will open another issue for the NUMA stuff
Comment 7 Christian Hüning 2016-03-18 11:58:13 UTC
The NUMA error is further tracked here: https://bugzilla.xamarin.com/show_bug.cgi?id=39759
Comment 8 Sylvain 2016-10-06 15:24:14 UTC
I see the same error running mono v4.0.4, v4.4.2, v4.6.1 on ARMv7 linux.

System with 1Gb RAM. 

After allocating about 271M, mono dies with Garbage collector could not allocate 16384 bytes of memory for major heap section.

In my opinion this issue is not only related to NUMA architecture and is still present for "regular" ARM linux systems.