Bug 9679 - Performance regression from Mono 2.10.9 to 2.10.10/3.0.x
Summary: Performance regression from Mono 2.10.9 to 2.10.10/3.0.x
Status: RESOLVED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: unspecified
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2013-01-19 18:41 UTC by Charles Esterbrook
Modified: 2013-05-23 15:26 UTC (History)
6 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
profile of benchmark on Mono 2.10.9 (194.38 KB, text/plain)
2013-01-19 20:47 UTC, Charles Esterbrook
Details
profile of benchmark on Mono 2.10.10 (203.16 KB, text/plain)
2013-01-19 20:47 UTC, Charles Esterbrook
Details
New small program recreates the problem. (1.29 KB, application/octet-stream)
2013-01-19 22:05 UTC, Charles Esterbrook
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Charles Esterbrook 2013-01-19 18:41:51 UTC
The regression is between Mono 2.10.9 and 2.10.10 on Mac OS X and slows performance by 40%. Mono 3.0.3 shows no improvement over 2.10.10. I have not checked other operating systems.

Start with Mono 2.10.9 from package MonoFramework-MRE-2.10.9_10.macos10.xamarin.x86.dmg obtained from http://download.mono-project.com/archive/2.10.9/download/. This is the fast version of Mono.

Since this is a benchmark, shut down web browsers, cloud services, irc client, file syncers, auto backer uppers, etc. Check tasks for anything consuming CPU.

The commands below are entered at a bash prompt on Mac OS X 10.6.8.

mono --version
  outputs: Mono JIT compiler version 2.10.9 (tarball Mon May  7 20:25:51 EDT 2012)

Get the Cobra compiler which is the program in question that slows down in Mono 2.10.10. Five commands:

svn co -r 2900 http://cobra-language.com/svn/cobra/trunk/ cobra-workspace
cd cobra-workspace/Source
bin/build
./cobra -bsl
bin/build-compiler-lib -turbo

Now it's time to benchmark. I like to repeat a few times:

mono --version
  outputs: Mono JIT compiler version 2.10.9 (tarball Mon May  7 20:25:51 EDT 2012)

cobra benchmark-compiler.cobra
  Total time (ms): 4609
cobra benchmark-compiler.cobra
  Total time (ms): 4617
cobra benchmark-compiler.cobra
  Total time (ms): 4627

Install Mono 2.10.10:

mono --version
  outputs: Mono JIT compiler version 2.10.10 (mono-2-10/4d9ada6 Wed Nov 28 22:38:28 EST 2012)

cobra benchmark-compiler.cobra
  Total time (ms): 6566
cobra benchmark-compiler.cobra
  Total time (ms): 6485
cobra benchmark-compiler.cobra
  Total time (ms): 6507

Mono Framework MRE 3.0.3 shows no real improvment:

cobra benchmark-compiler.cobra
  Total time (ms): 6477
cobra benchmark-compiler.cobra
  Total time (ms): 6453
cobra benchmark-compiler.cobra
  Total time (ms): 6505


Notes:

The slowdown going from 2.10.9 (~4617.33 ms) to 2.10.10 (~6519.33 ms) is 41%.

Rebuilding the programs under 2.10.10 does not improve anything.

Cobra is a compiler so it does a variety of things including loading assemblies, reflection, file i/o, string processing, data structures, etc. I have not yet determined if some parts are slowing down more than others.

By default, Cobra targets .NET 2.0. However, targeting .NET 4.0 under Mono 2.10.10 did not improve the benchmark.

Running with --gc=sgen on either Mono version results in about 15% performance improvement. But the sgen runs are still 40% slower vs. Mono 2.10.9 with sgen.

The 2.10.9 package (MonoFramework-MRE-2.10.9_10.macos10.xamarin.x86.dmg) is 85MB while the 2.10.10 package (MonoFramework-MRE-2.10.10.macos10.xamarin.x86.dmg) grows 20% to 102MB. Not sure what all was added in 2.10.10 over 2.10.9.

Discussion at http://lists.ximian.com/pipermail/mono-list/2013-January/049451.html
Comment 1 Charles Esterbrook 2013-01-19 20:45:24 UTC
I ran the mono profiler with:
mono --profile=log:report,calldepth=200 benchmark-compiler.exe

Nearly all methods become inflated including Mono's own methods:

# 2.10.9
Total(ms) Self(ms)      Calls Method name
   12349      512     333200 System.MonoCustomAttrs:GetCustomAttributes (System.Reflection.ICustomAttributeProvider,System.Type,bool)

# 2.10.10
Total(ms) Self(ms)      Calls Method name
   17718      751     333070 System.MonoCustomAttrs:GetCustomAttributes (System.Reflection.ICustomAttributeProvider,System.Type,bool)

# = 43-46% slower

# 2.10.9
    1263       18      77650 System.Globalization.CompareInfo:internal_compare_managed (string,int,int,string,int,int,System.Globalization.CompareOptions)
# 2.10.10
    1967       28      77650 System.Globalization.CompareInfo:internal_compare_managed (string,int,int,string,int,int,System.Globalization.CompareOptions)
# = 55% slower

GC became slower and fatter:

# .9

GC summary
	GC resizes: 19
	Max heap size: 28307456
	Object moves: 0
	Gen0 collections: 51, max time: 20102us, total time: 555057us, average: 10883us

# .10

GC summary
	GC resizes: 21
	Max heap size: 44642304
	Object moves: 0
	Gen0 collections: 38, max time: 59475us, total time: 1213134us, average: 31924us

HTH. I'll attach the profile reports.
Comment 2 Charles Esterbrook 2013-01-19 20:47:31 UTC
Created attachment 3242 [details]
profile of benchmark on Mono 2.10.9
Comment 3 Charles Esterbrook 2013-01-19 20:47:58 UTC
Created attachment 3243 [details]
profile of benchmark on Mono 2.10.10
Comment 4 Charles Esterbrook 2013-01-19 22:04:21 UTC
I got this down from about 50,000 lines to 62.  :-)

The following standalone C# program runs 38.9% slower on Mono 2.10.10 than on Mono 2.10.9 on my Mac OS X 10.6.8 box.

The program is nonsensical. It just does some reflection and builds up lists, sets and dictionaries.

$ mono --version
Mono JIT compiler version 2.10.9 (tarball Mon May  7 20:25:51 EDT 2012)
Copyright (C) 2002-2011 Novell, Inc, Xamarin, Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       normal
	Notification:  kqueue
	Architecture:  x86
	Disabled:      none
	Misc:          debugger softdebug 
	LLVM:          yes(2.9svn-mono)
	GC:            Included Boehm (with typed GC)

$  mono benchmark.exe
elapsed: 2045

$  mono benchmark.exe
elapsed: 2043

$  mono benchmark.exe
elapsed: 2046

$  mono --version
Mono JIT compiler version 2.10.10 (mono-2-10/4d9ada6 Wed Nov 28 22:38:28 EST 2012)
Copyright (C) 2002-2012 Novell, Inc, Xamarin, Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       normal
	Notification:  kqueue
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug 
	LLVM:          yes(2.9svn-mono)
	GC:            Included Boehm (with typed GC)

$  mono benchmark.exe
elapsed: 2838

$  mono benchmark.exe
elapsed: 2868

$  mono benchmark.exe
elapsed: 2840

Taking the middle values, the slow down is 2045 -> 2840 or 38.9%.

I'll attach the program as benchmark.cs.
Comment 5 Charles Esterbrook 2013-01-19 22:05:09 UTC
Created attachment 3244 [details]
New small program recreates the problem.
Comment 6 Zoltan Varga 2013-01-22 17:31:05 UTC
Are you using the official binary packages, or are you compiling from source ?
Comment 7 Zoltan Varga 2013-01-22 18:09:57 UTC
I can repro this with just:

		var a = Assembly.GetExecutingAssembly();
		for (int i = 0; i < Reps; i++) {
			var v = a.GetName ();
		}
Comment 8 Charles Esterbrook 2013-01-22 22:03:20 UTC
Official binary packages.

And yes, my C# program may not be the minimum. But it's much smaller than the original 50,000 line program and a bit smaller than the original 100 lines of C# I came up with.

It will be interesting to see if getting your even smaller program performing better will resolve the slowdown with mine.

Thanks for looking into this.
Comment 9 Marek Safar 2013-02-02 10:21:05 UTC
This is quite interesting. I was looking into it too but could not find the exact reason for sure but noticed few things.

mono_string_new seems to be slower in 2.10.10/master than in 2.10.9 that's probably due to memmove/memcpy change where we now have custom mono_gc_memmove implementation.

Another related issue is that MemberwiseClone/mono_object_clone which is used internally for fill assembly name is ~30% slower which may be related to memcpy change too.
Comment 10 Rodrigo Kumpera 2013-05-18 23:24:52 UTC
This has been fixed.
Comment 11 Charles Esterbrook 2013-05-19 02:37:40 UTC
What was the fix?
Comment 12 Rodrigo Kumpera 2013-05-23 11:23:27 UTC
The performance regression in mono_string_new is due to us handling null strings.
Comment 13 Charles Esterbrook 2013-05-23 15:26:11 UTC
Thanks for the fix and the explanation.