Bug 56341 - AOT process is too slow
Summary: AOT process is too slow
Status: RESOLVED NORESPONSE
Alias: None
Product: iOS
Classification: Xamarin
Component: Mono runtime / AOT compiler ()
Version: XI 10.10 (d15-2)
Hardware: PC Windows
: --- normal
Target Milestone: Untriaged
Assignee: Zoltan Varga
URL:
Depends on:
Blocks:
 
Reported: 2017-05-13 10:20 UTC by Jahmai
Modified: 2017-06-19 14:24 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NORESPONSE

Description Jahmai 2017-05-13 10:20:35 UTC
AOT'ing our applications has always taken a long time. However it seems to get slower with every release.

In the latest release 15.2, it takes 6 minutes(!) to AOT our application. Looking at the resource usage on the mac build host, it appears that only one mono-sgen64 process is executed during AOT, and it only uses 1 logical core. This is a terrible waste of performance on multi-core machines, especially when you have 12 cores.

------

Microsoft Visual Studio Professional 2017 
Version 15.2 (26430.6) Release
VisualStudio.15.Release/15.2.0+26430.6
Microsoft .NET Framework
Version 4.7.02046

Installed Version: Professional

Visual Basic 2017   00369-60000-00001-AA196
Microsoft Visual Basic 2017

Visual C# 2017   00369-60000-00001-AA196
Microsoft Visual C# 2017

Visual C++ 2017   00369-60000-00001-AA196
Microsoft Visual C++ 2017

Visual F# 4.1   00369-60000-00001-AA196
Microsoft Visual F# 4.1

Application Insights Tools for Visual Studio Package   8.6.00404.2
Application Insights Tools for Visual Studio

ASP.NET and Web Tools 2017   15.0.30503.0
ASP.NET and Web Tools 2017

ASP.NET Web Frameworks and Tools 2017   5.2.50303.0
For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0   15.0.30209.0
Azure App Service Tools v3.0.0

Azure Data Lake Node   1.0
This package contains the Data Lake integration nodes for Server Explorer.

Azure Data Lake Tools for Visual Studio   2.2.5000.0
Microsoft Azure Data Lake Tools for Visual Studio

Common Azure Tools   1.9
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

Fabric.DiagnosticEvents   1.0
Fabric Diagnostic Events

JavaScript Language Service   2.0
JavaScript Language Service

JavaScript Project System   2.0
JavaScript Project System

JavaScript UWP Project System   2.0
JavaScript UWP Project System

JetBrains ReSharper Ultimate 2017.1.2    Build 108.0.20170428.75743
JetBrains ReSharper Ultimate package for Microsoft Visual Studio. For more information about ReSharper Ultimate, visit http://www.jetbrains.com/resharper. Copyright © 2017 JetBrains, Inc.

KofePackagePackage Extension   1.0
KofePackagePackage Visual Studio Extension Detailed Info

Merq   1.1.17-rc (cba4571)
Command Bus, Event Stream and Async Manager for Visual Studio extensions.

Microsoft Azure Hive Query Language Service   2.2.5000.0
Language service for Hive query

Microsoft Azure Tools   2.9
Microsoft Azure Tools for Microsoft Visual Studio 2017 - v2.9.50131.1

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual Studio VC Package   1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio   Mono.Debugging.VisualStudio
Support for debugging Mono processes with Visual Studio.

Node.js Tools   1.3.50417.1
Adds support for developing and debugging Node.js apps in Visual Studio

NuGet Package Manager   4.2.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

PostSharp Tools for Visual Studio   5.0.24.0
PostSharp is the #1 pattern-aware extension to C# and VB.

SQL Server Data Tools   15.1.61702.140
Microsoft SQL Server Data Tools

ToolWindowHostedEditor   1.0
Hosting json editor into a tool window

TypeScript   2.2.2.0
TypeScript tools for Visual Studio

Visual Studio Tools for Universal Windows Apps   15.0.26430.06
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.

WiX Toolset Visual Studio Extension   0.9.17.13693
WiX Toolset Visual Studio Extension version 0.9.17.13693
Copyright (c) .NET Foundation and contributors. All rights reserved.

Xamarin   4.5.0.443 (c871575)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android SDK   7.3.0.13 (448f54f)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK   10.10.0.30 (30b6e87)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.

--------


=== Visual Studio Professional 2017 for Mac ===

Version 7.0 (build 3146)
Installation UUID: 5782cfba-0b27-4bcc-8621-d13ae60b90a2
Runtime:
	Mono 5.0.0.100 (2017-02/9667aa6) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 500000100

=== NuGet ===

Version: 4.0.0.2323

=== .NET Core ===

Runtime: /usr/local/share/dotnet/dotnet
SDK: /usr/local/share/dotnet/sdk/1.0.1/Sdks
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/5.0.0/lib/mono/msbuild/15.0/bin/Sdks

=== Xamarin.Profiler ===

Version: 1.5.4
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Xamarin.Android ===

Not Installed

=== Xamarin Inspector ===

Not Installed

=== Apple Developer Tools ===

Xcode 8.3.2 (12175)
Build 8E2002

=== Xamarin.iOS ===

Version: 10.10.0.33 (Visual Studio Professional)
Hash: 3e5ac5ff
Branch: d15-2
Build date: 2017-05-05 18:11:38-0400

=== Xamarin.Mac ===

Version: 3.4.0.33 (Visual Studio Professional)

=== Build Information ===

Release ID: 700003146
Git revision: 7553fda2e327d15807298d444007896e5b975dba
Build date: 2017-05-09 12:12:36-04
Xamarin addins: c3d580752be79fc902422f1d55d1cdc2d8b98799
Build lane: monodevelop-lion-d15-2

=== Operating System ===

Mac OS X 10.12.4
Darwin 16.5.0 Darwin Kernel Version 16.5.0
    Fri Mar  3 16:52:33 PST 2017
    root:xnu-3789.51.2~3/RELEASE_X86_64 x86_64
Comment 1 Zoltan Varga 2017-05-13 13:43:20 UTC
AOT is parallelized with one aot process per assembly. It looks like on of the assemblies in the app exposes a bug in the aot compiler so it takes a long time to run, this is why only one aot compiler process is left.
Comment 2 Jahmai 2017-05-13 13:47:01 UTC
Yeah we have 7 assemblies of our own, plus SDK and libraries, so we're definitely not seeing that level of paralellism.

Any suggestions as to how to figure out what the culprit is?
Comment 3 Zoltan Varga 2017-05-13 17:50:53 UTC
Would it be possible to zip up the 'obj/iPhone' directory and attach it to the bug or upload it to dropbox etc. ? You can make the attachment private. Also, can you attach the build log as well ?
Comment 4 Sebastien Pouliot 2017-05-15 18:59:24 UTC
A full build log with `-v -v -v -v -time -time -time -time` (in the Additional mtouch arguments) would also be helpful to get a better idea of where the time is spent (in case something is, than the AOT compiler, is taking too much time)
Comment 5 Michael Zimdars 2017-05-17 13:14:37 UTC
I have the same issue and I saw, that the command line of mono-sgen64 is veeery long and repeating the same assemblies over and over again.
Is it ensured that each assembly is only processed once, or could it be that they are processed multiple times even though they are the same?
I try to attach the command line as a (private) attachment
Comment 6 Michael Zimdars 2017-05-17 13:17:39 UTC
couldn't upload as private because I'm not allowed, but it contains company specific info, so I don't want to share publicly. How can I give the info privately?
Comment 7 Sebastien Pouliot 2017-05-17 13:19:29 UTC
@Michael, Each assembly is processed once _per architecture_

So if you build for armv7, armv7s and arm64 then they will be AOT'ed 3 times. Caching is done (if there's no change then it won't be re-AOT'ed, but if they are changes then all architectures needs to be re-compiled).

If you're doing debug builds you can easily to "device specific builds" or select a single architecture (in your project options) to cut down on the build time.

> couldn't upload as private because I'm not allowed,

File your own bug report and make it private. The attach the logs to the bug report. This will allow us to see the logs and also comment on them (in private, not in public).
Comment 8 Alex Soto [MSFT] 2017-06-19 14:24:49 UTC
We have not received the requested information. If you are still experiencing this issue please provide all the requested information and re-open the bug report. Thanks!