Bug 18692 - Including large numbers of Resources result in really slow build times
Summary: Including large numbers of Resources result in really slow build times
Status: VERIFIED FIXED
Alias: None
Product: Android
Classification: Xamarin
Component: MSBuild ()
Version: 4.12.0
Hardware: PC Mac OS
: High normal
Target Milestone: ---
Assignee: Jonathan Pryor
URL:
Depends on:
Blocks:
 
Reported: 2014-04-01 07:39 UTC by dean.ellis
Modified: 2014-09-23 11:57 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Test Case (22 bytes, text/plain)
2014-04-01 07:43 UTC, dean.ellis
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description dean.ellis 2014-04-01 07:39:10 UTC
From a customer case. They have a project with a large number of assets, the build process as a result takes up to 4 minutes. Attaching a test case which demonstrates the issue. It also includes an Eclipse project for comparison
Comment 1 dean.ellis 2014-04-01 07:43:23 UTC
Created attachment 6443 [details]
Test Case
Comment 2 dean.ellis 2014-04-01 07:44:00 UTC
The test case does not have a 4 minute build time but it does show the difference between Eclipse and Xamarin.Android
Comment 4 Jonathan Pryor 2014-04-01 16:40:10 UTC
Why do they have so many assets? Are they embedding HTML resources and using WebBrowser with file:///android_asset?

Could they instead use @(EmbeddedResource)? Unless you _absolutely_ require Assets or Resources, I _strongly_ suggest that @(EmbeddedResource) be used, because:

1. It's portable to all .NET platforms (Android, iOS, Desktop .NET, Metro, etc.)
2. It's faster. Obtaining and reading a stream via Assembly.GetManifestResourceStream() is much faster than using e.g. Assets.Open().
http://msdn.microsoft.com/en-us/library/ms950960.aspx
http://msdn.microsoft.com/en-us/library/vstudio/xc4235zt(v=vs.100).aspx

That aside, I'm confused by the test case: there are _four_ projects of interest, with different build times:

1. ImageView/ImageView_Xamarin/AssetImageView
real	0m4.058s [0]

2. ImageView/ImageView_Xamarin/ResourceImageView
real	0m12.855s [0]

3. ImageView/ImageView_Eclipse/AssetImageView
real	0m0.817s [1]

4. ImageView/ImageView_Eclipse/ResourceImageView
real	0m0.792s [3]

Unsurprisingly, the Eclipse projects build faster.

What confuses me is that the bug is stating that "large numbers of _assets_ result in really slow build times" (emphasis mine), whereas from the above timings we see that large numbers of _resources_ result in really slow build times. AssetImageView is ~1/3 the time of ResourceImageView.

The Xamarin projects will always take longer to build, as they _do_ more. For example, the @(AndroidAsset) files are copied into obj/Debug/assets, so that assets may be obtained from referenced assemblies and so that <Link/>ed files are properly copied. This extra copy -- along with all the other things normally done as part of the build -- limit how fast a Xamarin.Android build can be when compared to Eclipse.

Resources are "worse": not only must @(AndroidResource) files be copied, as is done with @(AndroidAssets), but they must also be "fixed up", so that e.g. a Layout .axml file can refer to C# names (as is done in ImageView/ImageView_Xamarin/AssetImageView/Resources/Layout/Main.axml!). This "fixup" in turn has overhead when done from a clean build.

Both the @(AndroidAsset) and @(AndroidResource) overheads _should_ be "one-time" affairs, only performed if the copies/fixups haven't already been done, and they shouldn't be re-done unless/until more assets/resources are added to the project. (Presumably you're not adding new assets/resources every time you build. Presumably you're building e.g. after every method you write or every file you update.)

With that in mind, on my machines, rebuilds _are_ faster: AssetImageView rebuilds in 1.3s (compared to ~4s), while ResourceImageView rebuilds in 1.2s (compared to ~12s).

The design expectation is that rebuilds will be far more frequent than "clean" builds. As such, it is "acceptable" for initial builds to be slow(er), as long as rebuilds times are commensurate with the amount of work involved.)

[0]: time xbuild /v:diag > b.txt
[1]: android update project -p . -t 11 --subprojects
    time ant debug
Comment 6 Jonathan Pryor 2014-04-09 16:10:28 UTC
Further investigation suggests that while things suck, we should still be able to do better.

Part of the problem is that when we create a .apk, we "create it from scratch," ignoring a significant portion of the work previously done. (We may be able to skip some things such as extracting resources/assets from referenced assemblies, fixing up some resources, etc.,)

We still generate the .apk structure via a "fresh" aapt invocation, while aapt allows updating an already existing .apk structure, allowing it to be much faster when updating packages.

For example, I have a Java project which contains 5M in resources (extracted from another project), and no code of consequence. On my speedy machine, building the project takes 11s. Editing a resource and rebuilding the project only takes 4s, implying that the android toolchain is able to optimize updating an existing .apk.

We should investigate doing likewise.
Comment 7 dean.ellis 2014-07-04 08:38:41 UTC
I've spent a good deal of time looking at this issue. I created a test project which had a ton of image resources in it (500) and did the build under ant. The first time around it took 6 minutes on my machine to build but only a  few seconds on subsequent builds.  Looking at the verbose ant output this is because the android toolchain pre-crunched the .png files before adding them to the apk (aapt is run with the --no-crunch argument) . It seems that its the crunching of the images that is causing much of the build slowdown. 

In our build system we currently do not pre-crunch the images, instead this is done when the images are actually added to the intermediate apk. This can be seen by running the aapt tool with the same command line arguments but adding the -v on the end. In the test project almost all of the time was spent crunching the images, adding the layout files and processing the resources is very quick.

Another issue is that when we copy the layout resources to the temp directory we are currently always copying them because we do some post-processing on them after the copy, as a result msbuild/xbuild always thinks they are modified. 

I'm working on a new Task which will do the Copy and Resource Update at the same time and only update the files that have actually been changed. This will work not only for layout files but for any other files. After this is done the Task will output a list of the files that have been changed. This is so we can crunch the .png files at that point (and only crunch the ones that have changed). 

The net result will be we can then run the aapt tool to build the apk with the --no-crunch argument just as the android tooling does, this should produce a significant speed increase on Rebuilds for projects will a large number of images. The first build from clean will still take some time on these projects because the images need to be processed, but after that as long as the intermediate files are there we will be able to only process what had changed.

NOTE the -u parameter for aapt is a bit confusing, if you use it to update resources, it seems to completely replace the internal resource file in the apk, also its worth noting that not even the android tool chain seems to use this option instead they use the -f to rebuild the base apk each time.
Comment 8 dean.ellis 2014-07-31 12:02:49 UTC
Fixed in monodroid/master/fa94e32

The changes which pre-crunch the images has been merged, unfortunately not in time to make it into the 4.16 release, but it will be 4.18 release.
Comment 9 Shruti 2014-09-23 08:50:38 UTC
I have checked this issue with attached project 'Test Case' on stable build Xamarin.Android Version: 4.16.0.17 and latest build Xamarin.Android Version: 4.18.0.32. I  have observed that on both builds it is taking time of around 15 seconds to build the project. 

Here is the screencast regarding same : http://screencast.com/t/27lxjR3z9m

But I am not sure, it is correct way to verify this issue because I am not able to reproduce this issue on stable builds. 


Please let me know What I have missed to check this issue.

Environment Info :
=== Xamarin Studio ===

Version 5.5 (build 203)
Installation UUID: 67eaf3b4-f8a7-4ab4-a9df-3197350ca5dc
Runtime:
	Mono 3.10.0 ((detached/ac51002)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 310000016

=== Apple Developer Tools ===

Xcode 6.0 (6256.8)
Build 6A279r

=== Xamarin.Mac ===

Version: 1.10.0.7 (Starter Edition)

=== Xamarin.Android ===

Version: 4.18.0.32 (Trial Edition)
Android SDK: /Users/ixamarin78/Desktop/android-sdk-macosx
	Supported Android versions:
		2.1    (API level 7)
		2.2    (API level 8)
		2.3    (API level 10)
		3.1    (API level 12)
		3.2    (API level 13)
		4.0    (API level 14)
		4.0.3  (API level 15)
		4.1    (API level 16)
		4.2    (API level 17)
		4.3    (API level 18)
		4.4    (API level 19)
		4.4.87 (API level 20)
		4.5    (API level 21)
Java SDK: /usr
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

=== Xamarin.iOS ===

Version: 8.2.0.177 (Trial Edition)
Hash: 3b73f2a
Branch: 
Build date: 2014-09-19 00:21:20-0400

=== Build Information ===

Release ID: 505000203
Git revision: 0d80212b712c0d42ecb20cf007b2192d715377a0
Build date: 2014-09-22 14:06:52-04
Xamarin addins: c31fea2f959659225466adef1c0793a8d7eebcef

=== Operating System ===

Mac OS X 10.9.4
Darwin XamnewiMac.local 13.3.0 Darwin Kernel Version 13.3.0
    Tue Jun  3 21:27:35 PDT 2014
    root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64
Comment 10 dean.ellis 2014-09-23 08:58:22 UTC
@Shruti  it looked like you are building from a clean state. In this case all the resources will need to be crunched/processed. The speed up is seen after you have done a full build and modify just one Resource file. It should not take 15 seconds but only a few as it should only be processing a small number of files (i.e only the ones that have changed). 

So to test you need to do a full build, them modify a single image or layout in resources and then just build (not rebuild) you should see quicker build times.
Comment 11 Mohit Kheterpal 2014-09-23 11:57:45 UTC
Thanks dean.

I have checked this issue and follow steps mentioned in comment 10. 

When I first full build the project then it takes 15-17 seconds and then I add button in Main.axml file then it takes only 4-5 seconds to build again. It takes less time to build again as it build only the changed file.

Hence closing this issue.

Environment info :
=== Xamarin Studio ===

Version 5.5 (build 204)
Installation UUID: 45555ffd-f3f0-4f29-9543-e369d711ab7d
Runtime:
	Mono 3.10.0 ((detached/ac51002)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 310000016

=== Apple Developer Tools ===

Xcode 6.0.1 (6528)
Build 6A317

=== Xamarin.iOS ===

Version: 8.2.0.177 (Enterprise Edition)
Hash: 3b73f2a
Branch: 
Build date: 2014-09-19 00:21:20-0400

=== Xamarin.Android ===

Version: 4.18.0.31 (Enterprise Edition)
Android SDK: /Users/xamarin77/Desktop/android-sdk-macosx
	Supported Android versions:
		2.1    (API level 7)
		2.2    (API level 8)
		2.3    (API level 10)
		3.1    (API level 12)
		3.2    (API level 13)
		4.0    (API level 14)
		4.0.3  (API level 15)
		4.1    (API level 16)
		4.2    (API level 17)
		4.3    (API level 18)
		4.4    (API level 19)
		4.4.87 (API level 20)
		4.5    (API level 21)
Java SDK: /usr
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

=== Xamarin.Mac ===

Version: 1.11.0.1 (Enterprise Edition)

=== Build Information ===

Release ID: 505000204
Git revision: 2affa3acc16487d4f8fb5cb33c49e2423469091e
Build date: 2014-09-23 07:14:56-04
Xamarin addins: c31fea2f959659225466adef1c0793a8d7eebcef

=== Operating System ===

Mac OS X 10.9.5
Darwin Xamarin77s-Mac-mini.local 13.4.0 Darwin Kernel Version 13.4.0
    Sun Aug 17 19:50:11 PDT 2014
    root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64