Bug 40726 - Very slow build times due to PackLibraryResources
Summary: Very slow build times due to PackLibraryResources
Status: VERIFIED FIXED
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: iOS ()
Version: 4.0.0 (C6)
Hardware: PC Windows
: High critical
Target Milestone: 4.1.0 (C7)
Assignee: Bugzilla
URL:
: 40799 40833 ()
Depends on:
Blocks:
 
Reported: 2016-04-27 08:16 UTC by Fabien Molinet
Modified: 2016-05-19 05:24 UTC (History)
9 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Mac configuration (1.13 KB, text/plain)
2016-04-27 08:20 UTC, Fabien Molinet
Details
PC configuration (33.31 KB, image/png)
2016-04-27 08:20 UTC, Fabien Molinet
Details
VS build logs (4.88 MB, application/x-zip-compressed)
2016-05-03 09:21 UTC, Fabien Molinet
Details
Build logs with older architecture (2.21 MB, application/zip)
2016-05-04 08:21 UTC, Fabien Molinet
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Fabien Molinet 2016-04-27 08:16:01 UTC
We recently changed the architecture of a solution, basically we have split some code logic and we have now 5 additional projects. 2 are iOS library, 1 is an Android library, 2 are PCL projects.

After this change the build times, for Xamarin iOS, on Visual Studio suddenly became extremely long.
It used to take at maximum one minute to build the iOS project, on Visual Studio. Now it takes 5 minutes.

With Xamarin Studio before/after does not really matter. In both cases it builds quickly.

We recorded a video that shows how slow it is.
Could you please provide us a solution? It is dragging down our team's productivity.
Comment 1 Fabien Molinet 2016-04-27 08:20:24 UTC
Created attachment 15854 [details]
Mac configuration
Comment 2 Fabien Molinet 2016-04-27 08:20:43 UTC
Created attachment 15855 [details]
PC configuration
Comment 3 Fabien Molinet 2016-04-27 08:25:28 UTC
Screen record video is available on Dropbox:
https://www.dropbox.com/s/jukawyccpx8scrd/Very%20Slow%20build%20with%20Visual%20Studio.mov?dl=0
Comment 4 Richard Hopkins 2016-04-29 08:16:56 UTC
The customer has asked to escalate this issue as it is causing severe productivity issues and affecting their development capability.  Have raised the importance level accordingly.
Comment 5 Jose Gallardo 2016-05-02 14:23:57 UTC
@Fabien,

We need to identify where is the build process being so slow.
In order to do that we need a detailed output for a full Rebuild, and a second build after that.

Please first change this dropdown:

Tools - Options - Projects & Solutions - Build and Run - MSBuild Output Verbosity: Diagnostic.

And then run a full Rebuild, and copy the Build Output from the Output Window / Build Pane.

Please share that content.

Then try a simple "Build" (not a rebuild) which should be fast, as there are no changes, and share that build output as well.

With that information we should be able to identify potential issues and get a diagnostic about what's causing such a slow build time.

Thanks!
Comment 6 Fabien Molinet 2016-05-03 09:21:52 UTC
Created attachment 15882 [details]
VS build logs
Comment 7 Fabien Molinet 2016-05-03 09:48:11 UTC
Hi Jose,

We uploaded VS build logs. In the archive you will find:

1. a full rebuild (5 minutes)
2. a build without any change (very fast)
3. a build with a simple change in a library iOS project -> this one takes 5 minute
4. a build with a simple change in the app iOS project -> this one takes 1 minute

Many thanks for your help

Cheers,
Fabien
Comment 8 Jose Gallardo 2016-05-03 14:07:36 UTC
Hi Fabien,

Thanks for sharing the build output files.
At a first look, at the end of each file you can see something like: 

11>      342 ms  MSBuild                                   13 calls
11>     1165 ms  Csc                                        1 calls
11>     1330 ms  RemoveDir                                  2 calls
11>     5208 ms  OptimizeImage                              1 calls
11>    250814 ms  PackLibraryResources                       1 calls
11>
11>Build succeeded.
11>
11>Time Elapsed 00:04:20.16
========== Rebuild All: 11 succeeded, 0 failed, 0 skipped ==========


The issue is clearly around PackLibraryResources, which is taking ~4 mins of your build. In cases (2) and (4) it's not executed. But in cases (1) and (3) it does take most of the build time.

PackLibraryResources requires to copy all those png images from Windows to the Mac over the network, that's why it impacts the build time from Windows and not from the Mac.

You mentioned this was a side effect of an architecture change. It might be helpful if you can share the same cases output for the previous architecture build, in particular for the Rebuild scenario (I guess those png files weren't on a library before?).

We'll take a deeper look and investigate why it's taking so long, and see some potential fixes on either your side or ours.

I'll keep you updated.

Thanks again!
Comment 9 Ben Beckley 2016-05-03 21:16:16 UTC
*** Bug 40833 has been marked as a duplicate of this bug. ***
Comment 10 Brendan Zagaeski (Xamarin Team, assistant) 2016-05-04 02:13:17 UTC
## For the engineering team

In case it's helpful, the private duplicate Bug 40833 includes a sample project and steps to replicate.
Comment 11 Fabien Molinet 2016-05-04 08:21:11 UTC
Created attachment 15901 [details]
Build logs with older architecture
Comment 12 Fabien Molinet 2016-05-04 08:29:22 UTC
Hi Jose,

Please find new build logs.
We used the old architecture which has fewer projects.

Cheers,
Fabien
Comment 13 Ben Beckley 2016-05-04 14:55:15 UTC
*** Bug 40799 has been marked as a duplicate of this bug. ***
Comment 14 Jose Gallardo 2016-05-04 15:28:49 UTC
Thanks Fabien,

We got other reports about issues with PackLibraryResources, and we're investigating it trying to get a patch for the issue. There is something definitely wrong on that task, or in some target making the task to process files that are not Embedded Resources.

We already have some repro solutions where we can confirm unexpected behaviors, and that's being used in our investigation. 

I'll keep you updated.

Thanks again!
Comment 15 Jose Gallardo 2016-05-06 00:40:59 UTC
Hi Fabien,

We've been investigating deeply the issues around PackLibraryResources. There are basically two main issues about it. The first one is that for making the resources to be bundled in the assembly after being potentially processed on the Mac side, we need to bring them back from the Mac, but we're doing that even if the file hasn't changed.
That's something to improve, which affects particularly Builds after changing something on the library project unrelated to the bundled resources / content files, like in your case 3.
The second issue we've found is that in the case we need to actually get the files from the Mac side, we're always using SCP to get the files, paying an overhead of creating a new SSH connection for each file, which can be useful for files bigger than 1 MB, but not for smaller ones, making the build slower.
We've implemented fixes for both issues, which should improve the build performance, particularly for the case 3, but maybe not that noticeable for case 1.
It would be great if you can give it a try downloading the following internal build:

https://s3.amazonaws.com/xvs.xamarin.com/patcheddrops/Xamarin.VisualStudio_4.1.0.465.msi

It's based on the Beta channel build, which means that you will need to switch to the beta channel on the Mac side.

We've also identified some additional great (but riskier) improvements we can implement in the way this task is used. But that will mostly be implemented in our next development cycle.

I would really appreciate your feedback about this internal build.

Thanks!
Comment 16 Fabien Molinet 2016-05-09 11:14:49 UTC
Many thanks Jose.
We installed the new addin and we now get much better performances !

Rebuild: around 50 seconds
Build with a change in the iOS library project: around 15 seconds
Comment 17 Jose Gallardo 2016-05-09 14:56:12 UTC
That's great!

Marking the bug as resolved.

Thanks Fabien!
Comment 18 Shruti 2016-05-19 05:24:54 UTC
As per comment(16) and (17), I am closing this issue.

Thanks!!