Bug 39181 - Memory leak with Compiled lamdba expressions
Summary: Memory leak with Compiled lamdba expressions
Status: RESOLVED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: JIT ()
Version: 4.2.0 (C6)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Rodrigo Kumpera
URL:
Depends on:
Blocks:
 
Reported: 2016-02-27 19:44 UTC by Artyom Antyipin
Modified: 2016-05-05 20:48 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Artyom Antyipin 2016-02-27 19:44:35 UTC
I've noticed that when using Expression<TDelegate>.Compile the memory usage is monotonically increasing on mono. This does not happen when using Microsoft .NET.
I've written simple test:

public static void Main(string[] args)
{
	Expression<Func<int, bool>> expr = x => x < 2;
	for (var i = 0; i < int.MaxValue; ++i)
	{
		for (var j = 0; j < int.MaxValue; ++j)
		{
			expr.Compile();
		}
		GC.Collect();
		GC.WaitForPendingFinalizers();
		GC.Collect();
		GC.WaitForPendingFinalizers();
		Console.WriteLine(GC.GetTotalMemory(true));
	}
}
On windows (Microsoft .NET) actually used memory remains the same after some time. On Linux (mono) the actually used memory keeps growing despite GetTotalMemory returning the same amount all the time.

I've run profiler (mono --profile=log:nocalls) on this test. It shows the following infrmation:
JIT summary
	Compiled methods: 65827
	Generated code size: 2023852
	JIT helpers: 66501
	JIT helpers code size: 1858187

GC summary
	GC resizes: 0
	Max heap size: 0
	Object moves: 9390
	Gen1 collections: 766, max time: 3518us, total time: 302741us, average: 395us
	GC handles weak: created: 1, destroyed: 0, max: 1
	GC handles normal: created: 2, destroyed: 0, max: 2

As far as I can understand it -- Delegate objects are collected "properly", yet JIT-ted code remains in memory -- which explains increasing memory usage.

It seems to be a critical bug for applications that executes dynamic code based on Expressions generated at run time.
Comment 1 Marek Safar 2016-02-29 14:35:23 UTC
I cannot reproduce the issue. The memory allocation goes up but it goes down too when I run it with j < 100000.
Comment 2 Artyom Antyipin 2016-03-01 20:07:15 UTC
I've modified the code as you suggested (j < 100000):
public static void Main(string[] args)
{
	Expression<Func<int, bool>> expr = x => x < 2;
	for (var i = 0; i < int.MaxValue; ++i)
	{
		for (var j = 0; j < 100000; ++j)
		{
			expr.Compile();
		}
		GC.Collect();
		GC.WaitForPendingFinalizers();
		GC.Collect();
		GC.WaitForPendingFinalizers();
		Console.WriteLine(GC.GetTotalMemory(true));
	}
}
and ran program for 10 minutes. I'm using top to measure memory used by the process. After running for ten minutes I've got the following output for 'top -c -p $(pgrep -d, mono)':
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                  
22921 artyom    20   0 1167856 1,005g  13628 R  99,9  6,5  10:24.98 mono LambdaCompile.exe 

As you can see 1+ Gb of resident memory is used. And continuously growing...

So the issue is completely reproducible on my machine.
OS: Ubuntu 15.10 (4.2.0-30-generic)
mono (installed from "deb http://download.mono-project.com/repo/debian wheezy main"):
Mono JIT compiler version 4.2.2 (Stable 4.2.2.30/996df3c Mon Feb 15 17:30:30 UTC 2016)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

Program compiled using: xbuild /p:Configuraton=Release

What further infromation can I provide?
Comment 3 Marek Safar 2016-03-04 14:35:19 UTC
I can repro this now and it looks like large leak in unmanaged memory I ran it long enough that it crashes with (over 2GB of memory allocated in over 1h)

mono(89162,0xa2ded000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
Could not allocate 56 bytes
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Delegate.CreateDelegate_internal (System.Type,object,System.Reflection.MethodInfo,bool) <0x00012>
  at System.Delegate.CreateDelegate (System.Type,object,System.Reflection.MethodInfo,bool,bool) <0x00645>
  at System.Delegate.CreateDelegate (System.Type,object,System.Reflection.MethodInfo) <0x0003b>
  at System.Reflection.Emit.DynamicMethod.CreateDelegate (System.Type,object) <0x00045>
  at System.Linq.Expressions.Compiler.LambdaCompiler.CreateDelegate () <0x0007f>
  at System.Linq.Expressions.Compiler.LambdaCompiler.Compile (System.Linq.Expressions.LambdaExpression,System.Runtime.CompilerServices.DebugInfoGenerator) <0x00097>
  at System.Linq.Expressions.Expression`1<TDelegate_REF>.Compile () <0x0001b>
  at XX.Main (string[]) <0x000db>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0x000a0>

Native stacktrace:
Comment 4 Rodrigo Kumpera 2016-03-07 05:31:10 UTC
I cannot reproduce that with master. 

Only with 4.2.1 and 4.3.2.

I'll see what's happening in 4.3.2 and try to backport.
Comment 5 Rodrigo Kumpera 2016-03-07 06:47:56 UTC
I believe this to be fixed by 8cb221daeb0f9dd8246fad076056413401b0b26a

If you disable AOT "-O=-aot", the leak goes away.
Comment 6 Rodrigo Kumpera 2016-04-14 06:29:22 UTC
This will be fixed on 4.6
Comment 7 Matt Z 2016-05-05 06:54:36 UTC
Rodrigo, for Mono 4.2/4.4 is the -O=-aot workaround okay for production use?
Comment 8 Rodrigo Kumpera 2016-05-05 20:29:12 UTC
Yes, it will disable the usage of any pre compiled images.

The side effect is that you might experience a startup time penalty.
Comment 9 Matt Z 2016-05-05 20:48:56 UTC
We have found that there are two related leaks:

1) Compiling an Expression (or DynamicMethod) into a delegate
2) Invoking the delegate the first time

#1 is fixed by disabling AOT, but #2 is not