Notice (2018-05-24): bugzilla.xamarin.com is now in
Please join us on
Visual Studio Developer Community and in the
Mono organizations on
GitHub to continue tracking issues. Bugzilla will remain
available for reference in read-only mode. We will continue to work
on open Bugzilla bugs, copy them to the new locations
as needed for follow-up, and add the new items under Related
Our sincere thanks to everyone who has contributed on this bug
tracker over the years. Thanks also for your understanding as we
make these adjustments and improvements for the future.
Please create a new report on
GitHub or Developer Community with
your current version information, steps to reproduce, and relevant error
messages or log files if you are hitting an issue that looks similar to
this resolved bug and you do not yet see a matching new report.
I have a machine with a Dual Xeon CPU and 96 GB Ram that is bound together via NUMA. This results in a division of CPU and memory ressources into nodes. When Mono is run inside a virtual machine using Linux KVM, Mono fails to allocate more than 48 GB of memory since the NUMA based architecture in conjunction with KVM prevents the usage of the 2nd CPU's memory (read here for details: http://www.admin-magazine.com/Archive/2014/20/Best-practices-for-KVM-on-NUMA-servers/(offset)/3 ; and this issue for reference: https://bugzilla.xamarin.com/show_bug.cgi?id=39563)
However when run on the native hardware with Ubuntu 14.04 the app fails with the same error, of not being able to allocate more than 48 GB of memory, even though there are 96 GB available. I guess that Mono somehow sees 96 GB of available memory, but due to the NUMA architecture specialties is not able to allocate more than 48 GB.
Assigning this back to Christian for more information.
Can you please explain in more detail:
- What is the failure you are seeing? (What error message, etc)
- What is your expected behavior?
- Why do you expect that behavior? Isn't limiting the mono process to its local bank desirable…?
the failure I see is that Mono can't allocate any more memory. Like "Failed to allocate memory: 16000(something)".
My expected behaviour would be that the mono process uses all available memory.
I expect this, because I think it should be possible to use more than 48 GB of memory from a NUMA based system. According to documentation this is even possible with Linux KVM as a hypervisor. Since that's the case I thought it should be possible without a virtualization layer in between. Or am I wrong here?
Our team does not have any knowledge about NUMA. However, when mono allocates memory we do so through ordinary memory allocation mechanisms, typically mmap+MAP_ANONYMOUS. Therefore we expect mono would behave exactly the same way as any other Linux application. So my question would be what the standard Linux behavior is.
Skimming Google indicates the Linux kernel supports multiple NUMA "policies", that different processes may be subscribed to different policies on a single machine, and only some policies allow a process to allocate memory from more than one node. I also turn up this standard command-line utility:
It offers a sample invocation which launches a program with a policy would allow memory to allocate from multiple nodes:
numactl --cpubind=0 --membind=0,1 -- mono etc etc etc
There is also a --preferred option, so you could run --cpubind=0 --membind=0,1 --preferred=0 and then only start to touch the second node's slower memory when you go above 48GB memory usage.
Several documents indicate the default policy out of the box is something like this (pick a random cpu to bind to and then "prefer" its memory), for example:
*However* if you have a system administrator on the machine you are using they may have configured a different default.
Are you familiar with any of this? Have you tried running with numactl? What do numactl --show and numactl --hardware say? Also, this document:
Says there is a numa_maps in /proc that indicates memory usage across NUMA nodes. If you check numa_maps, do you see any indication that mono (or any process) is using memory from a node above N0?
Basically unless you have reason to believe otherwise, my suspicion would be that your problem is due to a local system configuration issue.
Since there hasn't been any activity on this bug for several months, I'm going to close it. If you still have problems with Mono's memory allocation that aren't addressed by Andi's comment, feel free to reopen. Thanks!
(Closing as INVALID as this does not appear to actually be a bug in Mono, but just how all apps behave on a NUMA system.)