Bug 39759 - Mono fails to allocate memory on NUMA architecture based systems
Summary: Mono fails to allocate memory on NUMA architecture based systems
Status: RESOLVED INVALID
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: 4.2.0 (C6)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Christian Hüning
URL:
Depends on:
Blocks:
 
Reported: 2016-03-18 11:57 UTC by Christian Hüning
Modified: 2016-08-29 09:20 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED INVALID

Description Christian Hüning 2016-03-18 11:57:41 UTC
I have a machine with a Dual Xeon CPU and 96 GB Ram that is bound together via NUMA. This results in a division of CPU and memory ressources into nodes. When Mono is run inside a virtual machine using Linux KVM, Mono fails to allocate more than 48 GB of memory since the NUMA based architecture in conjunction with KVM prevents the usage of the 2nd CPU's memory (read here for details: http://www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers/(offset)/3 ; and this issue for reference: https://bugzilla.xamarin.com/show_bug.cgi?id=39563)

However when run on the native hardware with Ubuntu 14.04 the app fails with the same error, of not being able to allocate more than 48 GB of memory, even though there are 96 GB available. I guess that Mono somehow sees 96 GB of available memory, but due to the NUMA architecture specialties is not able to allocate more than 48 GB.

Cheers,
Christian
Comment 1 Andi McClure 2016-04-04 16:12:33 UTC
Assigning this back to Christian for more information.

Can you please explain in more detail:

- What is the failure you are seeing? (What error message, etc)
- What is your expected behavior?
- Why do you expect that behavior? Isn't limiting the mono process to its local bank desirable…?

Thanks
Comment 2 Christian Hüning 2016-04-04 18:56:59 UTC
Hi,

the failure I see is that Mono can't allocate any more memory. Like "Failed to allocate memory: 16000(something)".

My expected behaviour would be that the mono process uses all available memory. 

I expect this, because I think it should be possible to use more than 48 GB of memory from a NUMA based system. According to documentation this is even possible with Linux KVM as a hypervisor. Since that's the case I thought it should be possible without a virtualization layer in between. Or am I wrong here?
Comment 3 Andi McClure 2016-04-04 21:15:32 UTC
Christian--

Our team does not have any knowledge about NUMA. However, when mono allocates memory we do so through ordinary memory allocation mechanisms, typically mmap+MAP_ANONYMOUS. Therefore we expect mono would behave exactly the same way as any other Linux application. So my question would be what the standard Linux behavior is.

Skimming Google indicates the Linux kernel supports multiple NUMA "policies", that different processes may be subscribed to different policies on a single machine, and only some policies allow a process to allocate memory from more than one node. I also turn up this standard command-line utility:

http://linux.die.net/man/8/numactl

It offers a sample invocation which launches a program with a policy would allow memory to allocate from multiple nodes:

    numactl --cpubind=0 --membind=0,1 -- mono etc etc etc

There is also a --preferred option, so you could run --cpubind=0 --membind=0,1 --preferred=0 and then only start to touch the second node's slower memory when you go above 48GB memory usage.

Several documents indicate the default policy out of the box is something like this (pick a random cpu to bind to and then "prefer" its memory), for example:

https://www.kernel.org/doc/Documentation/vm/numa

*However* if you have a system administrator on the machine you are using they may have configured a different default.

Are you familiar with any of this? Have you tried running with numactl? What do numactl --show and numactl --hardware say? Also, this document:

https://codywu2010.wordpress.com/2015/10/01/how-shared-library-gets-mmaped-on-numa-system/

Says there is a numa_maps in /proc that indicates memory usage across NUMA nodes. If you check numa_maps, do you see any indication that mono (or any process) is using memory from a node above N0?

Basically unless you have reason to believe otherwise, my suspicion would be that your problem is due to a local system configuration issue.
Comment 4 Alex Rønne Petersen 2016-08-29 09:20:03 UTC
Since there hasn't been any activity on this bug for several months, I'm going to close it. If you still have problems with Mono's memory allocation that aren't addressed by Andi's comment, feel free to reopen. Thanks!

(Closing as INVALID as this does not appear to actually be a bug in Mono, but just how all apps behave on a NUMA system.)