Bug 6216 - mmap(...PROT_NONE...) failed
Summary: mmap(...PROT_NONE...) failed
Status: RESOLVED ANSWERED
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: unspecified
Hardware: Other Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
: 678 11026 ()
Depends on:
Blocks:
 
Reported: 2012-07-20 15:25 UTC by vc
Modified: 2017-07-11 23:35 UTC (History)
14 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
workaround (251 bytes, patch)
2012-08-16 04:17 UTC, Zoltan Varga
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED ANSWERED

Description vc 2012-07-20 15:25:19 UTC
Hi,
We are getting hard crash of mono at some instances. This is multi threaded application. Code failure points to following code below

if (null == (Object)src) return null;
            Byte[] newArray = new Byte[src.stringLength];
            if((null != src.characters) || (0 != src.defaultByteValue))
                if (0 == src.defaultByteValue) for (UInt32 i = 0; i < src.characters.Length; ++i) newArray[i] = src.characters[i];
                else for (UInt32 i = 0; i < src.stringLength; ++i) newArray[i] = src[i];
            return newArray;

failure is pointing to line where new Byte[] is being created. This is seen in production. internal testing has not shown this behavior so far. Please let me know what else i can capture or modify the code to fix it.


mmap(...PROT_NONE...) failed
Stacktrace:

  at (wrapper managed-to-native) object.__icall_wrapper_mono_array_new_specific (intptr,int) <0xffffffff>
  at A.A.SV.SS.op_Implicit (A.A.SV.SS) [0x00013] in D:\xyz\Dev\A.A.SV\SS.cs:225
  at A.A.SV.SS.ReplaceBytes (uint,uint,A.A.SV.SS,uint,uint) [0x00001] in D:\xyz\Dev\A.A.SV\SS.cs:676
  at (wrapper remoting-invoke-with-check) A.A.SV.SS.ReplaceBytes (uint,uint,A.A.SV.SS,uint,uint) <IL 0x0003f, 0xffffffff>
  at A.A.SR.EI.VRBCopy (A.A.SV.SS,A.A.SV.SS) [0x00001] in D:\xyz\Dev\A.A.SR\EI.cs:335
  at A.A.SR.SyncInstructions.VRead (A.A.SR.InternalVars,string,bool,object&) [0x00304] in D:\xyz\Dev\A.A.SR\SyncInstructions.cs:219  
  at A.C.DynamicAdapters.Generated.DeserializingAdapter_ISerializedResource_Context_0.Invoke (uint,object[]) <IL 0x048f1, 0x02622>
  at (wrapper xdomain-dispatch) A.C.Interfaces.ISerializedResource.Invoke (object,byte[]&,byte[]&,uint) <IL 0x00065, 0xffffffff>
  at (wrapper xdomain-invoke) A.C.Interfaces.ISerializedResource.Invoke (uint,object[]) <IL 0x000b6, 0xffffffff>
  at A.C.DynamicAdapters.Generated.SerializingAdapter_ISE_ISerializedResource_10.__Exec (string) <0x000a2>
  at A.C.DynamicAdapters.Generated.IndirectionAdapter_ISE_ISE_8.__Exec (string) <0x0002b>
  at A.A.EE.RM.Enter (A.A.EE.PInstance,string) <0x00170>
  at A.A.EE.RM.ProcessThreadStart (object) <0x00613>
  at System.Threading.Thread.StartInternal () [0x0002b] in /root/packages/BUILD/mono-2.10.8/mcs/class/corlib/System.Threading/Thread.cs:705
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <IL 0x0004e, 0xffffffff>

Native stacktrace:

        mono [0x80e156b]
        [0x87e440]
        /lib/libc.so.6(abort+0x101) [0x499701]
        mono [0x82255e7]
        mono [0x82263d7]
        mono [0x82406ed]
        mono [0x822d850]
        mono [0x822cf48]
        mono [0x822de55]
        mono [0x822678d]
        mono [0x8226b00]
        mono [0x8226c9b]
        mono [0x82275ef]
        mono(mono_array_new_specific+0xa2) [0x81ca332]
 	[0x111368]
        [0x7bfb3b0]
        [0x64849bc]
        [0x648495c]
        [0x64848a8]
        [0x7bfc800]
        [0x55aafa4]
        [0x36d380c]
        [0x647e018]
        [0x8d023b3]
        [0x1007623]
        [0x7699f0d]
        [0x26c827]
        [0x281183]
        [0x2810b4]
        [0x27f731]
        [0xb435e4]
        [0xc6ba56]
        [0xd81b21]
        mono [0x80623e8]
        mono(mono_runtime_invoke+0x40) [0x81c3e80]
        mono(mono_runtime_delegate_invoke+0x36) [0x81c40d6]
        mono [0x819b282]
        mono [0x81fda25]
        mono [0x8228699]
        /lib/libpthread.so.0 [0x5eb5ab]
        /lib/libc.so.6(clone+0x5e) [0x540cfe]

Debug info from gdb:


Unfortunately I do not have exact reproducible steps now.
Comment 1 vc 2012-07-30 18:40:31 UTC
we had another incident reported. It was at another method but at a place where new byte array was getting initialised.
There is enough memmory available in system. Our application was using around 8% of total memory. No other process was utilising the memory or CPU at the time issue was reported.
Comment 2 Zoltan Varga 2012-08-16 04:15:52 UTC
Try the attached patch as a workaround.
Comment 3 Zoltan Varga 2012-08-16 04:17:12 UTC
Created attachment 2359 [details]
workaround
Comment 4 vc 2012-08-24 19:18:51 UTC
Thanks Zoltan,
Would apply the workaround and check how it goes. Meanwhile I was looking into our code to check for any memory leaks. we disposed some objects after completion of thread processing. I had collected memory statistics. Now it takes little longer to meet above error. Strange thing to notice is we see RSS memory well within rangebound, but VSS memory keeps on increasing slowly & steadily. we do expect memory to go up and come down as part of application. I can see that happening with RSS, but not with VSS. crash of mono happens at a time where VSS memory reaches around 2.2 GB.( RSS at this point of time is ~ 1.5 GB).Could this be case of memory leak, If so could you please guide me how to check where is the leak?
Ours is 32 bit Linux (RHEL 5.4) machine and mono 2.10.8. 
Errors point to different menthods in our code but top line points to icall_wrapper_mono_array_new_specific
Comment 5 Zoltan Varga 2012-08-25 07:56:33 UTC
We have a profiler, see man mprof-report for more info. Its possible that this problem happens when the applications memory usage reaches a certain limit.
Comment 6 Miguel de Icaza [MSFT] 2012-09-02 13:00:01 UTC
Perhaps you are exhausting the virtual address space (VSS), even if not all of that memory is in use (RSS).
Comment 8 Zoltan Varga 2012-09-19 20:45:51 UTC
Never saw anything like that. Older versions of mono allowed background threads to die silently when they threw an unhandled exception, but newer versions abort the app with an error message.
Comment 9 Ben 2012-11-05 08:26:22 UTC
I would like to add my weight to this problem.  I have a new server running mono 3.0.0 with mod_mono 2.10 and xsp 2.10 on SUSE 12.1 on Linux 3.1.10 x86_64.

After approx. 500,000 hits to XSP, this will fail.  The stack-trace occurs in different places, but always about memory.  For instance:

mmap(...PROT_NONE...) failed
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_string_alloc (intptr) <0xffffffff>
  at (wrapper alloc) object.Alloc (intptr,int) <0xffffffff>
  at string.Concat (string,string) <0x00077>
  at string.Concat (object,object) <0x00053>
  at [My code.]


This comes from a line of code:

String s;
...
s += "foo";

The server was running approx. 50 threads at time of failure.

This is excellently predictable.  I will try the patch and let you know.
Comment 10 Ben 2012-11-05 11:08:56 UTC
Post "dissable_munmap" from patch.

Sample of my configure.in:

--------------------------------------------------------
case "$host" in
        powerpc*-*-linux*)
                # https://bugzilla.novell.com/show_bug.cgi?id=504411
                disable_munmap=yes
        ;;
esac

disable_munmap=yes
host_win32=no
target_win32=no
platform_android=no
--------------------------------------------------------



However same failure experienced:

--------------------------------------------------------
mmap(...PROT_NONE...) failed
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_string_alloc (intptr) <0xffffffff>
  at (wrapper alloc) object.Alloc (intptr,int) <0xffffffff>
  at string.Concat (string,string) <0x00077>
  at string.Concat (object,object) <0x00053>
  at ...
--------------------------------------------------------

Memory at time of failure (within 60 seconds):

VmPeak:  6762544 kB
VmSize:  6697664 kB
VmLck:         0 kB
VmHWM:   4429688 kB
VmRSS:   4329604 kB
VmData:  6644464 kB
VmStk:       136 kB
VmExe:      3224 kB
VmLib:      2740 kB
VmPTE:      9168 kB
VmSwap:     4912 kB
Threads:        41

Only thing I noted is that the GC never seemed to run.  The memory allocation grew linearly from 0 to ~6.6GB.  At no time was this reclaimed.  (The server has 12GB)

My GC is Boehm:

Mono JIT compiler version 3.0.0 (tarball Tue Oct 23 17:00:36 BST 2012)
Copyright (C) 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            Included Boehm (with typed GC and Parallel Mark)
Comment 11 Zoltan Varga 2012-11-05 13:43:46 UTC
You can see whenever a GC was done by running mono with:
MONO_LOG_LEVEL=debug MONO_LOG_MASK=gc mono <app>

You can also try changing this line in configure.in:

        *-*-linux*)
                host_win32=no
                CPPFLAGS="$CPPFLAGS -DGC_LINUX_THREADS -D_GNU_SOURCE -D_REENTRANT -DUSE_MMAP"

to:

        *-*-linux*)
                host_win32=no
                CPPFLAGS="$CPPFLAGS -DGC_LINUX_THREADS -D_GNU_SOURCE -D_REENTRANT"

i.e. remove -DUSE_MMAP.
Comment 12 Ben 2012-11-06 08:55:13 UTC
Thanks for the suggestion.  I recompiled mono without this option and tried again.  

Sorry, same exception resulted:

mmap(...PROT_NONE...) failed
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_string_alloc (intptr) <0xffffffff>
  at (wrapper alloc) object.Alloc (intptr,int) <0xffffffff>
  at string.Concat (string,string) <0x0007e>
  at string.Concat (object,object) <0x0005b>
  at ...

I could not use the mono debug you suggested, as mod_mono does not capture the stdin/stderr (as far as I know, and nothing in /var/log/apache2/error.log).

However I added a small log of my own.  This recorded the Boehm GC made 2164 'Collect()' calls during the life of the code: about one every three seconds.

However the memory allocation at termination was still massive:

VmPeak:  3576212 kB
VmSize:  3576212 kB
VmLck:         0 kB
VmHWM:   2393788 kB
VmRSS:   2359216 kB
VmData:  3520652 kB
VmStk:       136 kB
VmExe:      3224 kB
VmLib:      7076 kB
VmPTE:      4980 kB
VmSwap:        0 kB
Threads:        34

Something is sticking very firmly in memory...
Comment 13 Zoltan Varga 2012-11-06 09:24:47 UTC
Something is not right here, that error can only happen if USE_MUNMAP is defined. Could you check whenever -DUSE_MUNMAP is present in libgc/Makefile or not ?
Comment 14 Ben 2012-11-06 14:40:08 UTC
Thanks for checking, you are correct:

# grep "DUSE_MUNMAP" Makefile
CPPFLAGS = -DGC_LINUX_THREADS -D_GNU_SOURCE -D_REENTRANT -DUSE_MMAP -DUSE_MUNMAP -D__default_codegen__ -DUSE_COMPILER_TLS

I have deleted the mono-3.0.0 code, extracted from tar file again, changed configuration to your request.  However this does NOT remove the line above.

I note you asked me to remove USE_MMAP from configure.in, and then check for USE_MUNMAP in the Makefile?

I have now removed all -DUSE_MMAP and added the disable_munmap=yes declaration from the original patch.

After configure, The CPPFLAGS in the Makefile in libgc still read:

CPPFLAGS = -DGC_LINUX_THREADS -D_GNU_SOURCE -D_REENTRANT -DUSE_MMAP -DUSE_MUNMAP -D__default_codegen__ -DUSE_COMPILER_TLS

Is this a bug in configure?
Comment 15 Zoltan Varga 2012-11-06 14:42:28 UTC
You have to run autogen.sh to regenerate configure from configure.in.
Comment 16 Ben 2012-11-07 11:59:14 UTC
Apologies for my lack of experience with GNU Automake and associated tools.

I ran my code with your changes, and the exception does change:

Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_string_alloc (intptr) <0xffffffff>
  at (wrapper alloc) object.Alloc (intptr,int) <0xffffffff>
  at string.Concat (string,string) <0x00079>
  at string.Concat (object,object) <0x0005b>
  at ...

The memory at this time was:

VmPeak:  8031028 kB
VmSize:  7965556 kB
VmLck:         0 kB
VmHWM:   6480228 kB
VmRSS:   6480228 kB
VmData:  5192272 kB
VmStk:       136 kB
VmExe:      3220 kB
VmLib:      7056 kB
VmPTE:     10552 kB
VmSwap:       44 kB
Threads:        35

This is considerably more than without the change suggested, and the memory load at failure is far closer to the physical memory available on the server.

Point of interest: it always fails in 'string.Concat', although from many places in my code.

I still have to work out what is leaking memory in mod_mono, but this at least extends the time between failure.
Comment 17 Zoltan Varga 2012-11-07 13:18:33 UTC
This looks like a normal out of memory situation now, which mono is not very good at handling gracefully.
Comment 18 Miguel de Icaza [MSFT] 2012-11-07 15:27:25 UTC
This is running into an internal limit in Boehm for the memory that your process allocates.

Rebuild your Mono like this:

./configure --with-large-heap=yes

http://stackoverflow.com/questions/7248980/mono-too-many-heap-sections-increase-maxhincr-or-max-heap-sects-when-app-takes
Comment 19 Ben 2012-11-09 04:37:49 UTC
Makes a big difference. Now mod-mono-server was killed by OS at 18G:

VmPeak: 18760644 kB
VmSize: 18760644 kB
VmLck:         0 kB
VmHWM:  11049600 kB
VmRSS:  10991872 kB
VmData:  2276020 kB
VmStk:       136 kB
VmExe:      3220 kB
VmLib:      6740 kB
VmPTE:     27904 kB
VmSwap:  6507880 kB

We still have the memory leak in the above program.  I have submitted that as a separate bug:  https://bugzilla.xamarin.com/show_bug.cgi?id=8284

Thanks for your help.
Comment 20 laneser 2012-12-06 20:10:57 UTC
I compile mono with configure --with-large-heap=yes.

Try to solve the problem with remove DUSE_MUNMAP at configure but not work.
Then try to use mono --gc=sgen , looks no problem.
Comment 21 Zoltan Varga 2013-04-21 15:35:20 UTC
*** Bug 11026 has been marked as a duplicate of this bug. ***
Comment 22 Zoltan Varga 2013-04-21 15:39:17 UTC
*** Bug 678 has been marked as a duplicate of this bug. ***
Comment 23 Zoltan Varga 2013-04-22 16:32:35 UTC
This is still open.
Comment 24 evolvedmicrobe 2013-05-07 20:05:27 UTC
Having the same problem with Mono 3.0.2, sporadic failure location, always with this error though.  Really not sure how or where the mmap command is being called.

I just switched to sgen GC and am hoping this avoids the issue entirely.  If it does, is there any downside to this?  Any reason it isn't the default?
Comment 25 Randy Martinez 2014-04-03 13:30:58 UTC
This is still going on, both on Mono 3.2.5 and 3.0.7.
Comment 26 David Evans 2014-06-12 19:42:50 UTC
In case it helps anyone else, we encountered this error as we were migrating from mono 2.6 to 2.10 awhile back. We worked around it by increasing our linux process maximum mmap setting. Something different about the garbage collector was causing more unmappings than before so we were blowing past the default application limit, which on our version of ubuntu was set to 65535. We increased that 10 fold and haven't seen the problem since.

So now we have:
vm.max_map_count = 655360

Maybe a different bug because we were hitting this crash mostly during GC's, can't tell from the original description here. But I was surprised that the linux application limit for blocks would have that particular limit and it took awhile to figure out that was our issue. We have a BIG app which could hit 10 gigs of ram in use, which related probably. We were and are still using Boehm GC, now on 2.10.9
Comment 27 Rodrigo Kumpera 2014-09-08 11:21:29 UTC
The problem is that both of mono's GC use munmap in the middle of VM regions, causing them to split and leading to the abort.

This is not easily fixable.
Comment 28 Craig Minihan 2014-11-18 18:40:20 UTC
I had 8 servers running load tests on Mono 3.8/Boehm today. Five of the boxes crashed out with this error. In two of those cases other processes died too - unfortunately there is nothing useful in dmesg or messages.

It is almost certainly due to low memory in each case - so my fault but:

* Can Mono/Boehm use swap if available? It should have soldiered on since there is 16GB swap available, but that didn't happen.
* It doesn't seem possible to detect low system memory on Mono/Linux since there is no way to get system memory usage info and no way to know that the application has little memory overhead to play with.

The only info I have is from System.Diagnostics.Process.GetCurrentProcess() (PagedMemorySize64 is always 0) and the "Mono Memory" performance counter. It would be *great* to have more process, perf counter or maybe even WMI (CIM_OperatingSystem!) to keep highly utilized processes from bad places.
Comment 29 Rodrigo Kumpera 2017-07-11 23:35:15 UTC
Please switch to sgen as the boehm gc is deprecated.