Bug 54303 - Provide a diagnostic logging mechanism for what the linker marks and sweeps.
Summary: Provide a diagnostic logging mechanism for what the linker marks and sweeps.
Status: CONFIRMED
Alias: None
Product: Android
Classification: Xamarin
Component: Linker ()
Version: 7.1 (C9)
Hardware: PC Windows
: --- enhancement
Target Milestone: ---
Assignee: Radek Doulik
URL:
Depends on:
Blocks:
 
Reported: 2017-03-30 18:27 UTC by Jon Douglas [MSFT]
Modified: 2017-09-22 17:10 UTC (History)
2 users (show)

Tags: bb
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report for Bug 54303 on Developer Community or GitHub if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: Developer Community HTML or GitHub Markdown
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:
Status:
CONFIRMED

Description Jon Douglas [MSFT] 2017-03-30 18:27:10 UTC
*Description:

The Mono Linker has traditionally stumped many Xamarin developers in the past with errors that aren't straightforward of what the issue could be. This feature request hopes to outline why an optional diagnostic linker log would help developers figure out these issues.

1. The Linker runs before dx.

Seeing the list of DependsOnTargets:

_CompileDex
  _CompileToDalvikWithDx
    _GeneratePackageManagerJava
      _GetAddOnPlatformLibraries
        _GenerateJavaStubs
          _PrepareAssemblies
            _LinkAssemblies

Because the Linker runs before dx, we can run into traditional dex issues such as an item not making the dex list. Here is an example of this:

java.lang.RuntimeException: Unable to instantiate activity ComponentInfo{com.microsoft.azure.mobile.xamarin.puppet/com.microsoft.azure.mobile.distribute.DeepLinkActivity}: java.lang.ClassNotFoundException: Didn't find class "com.microsoft.azure.mobile.distribute.DeepLinkActivity" on path: DexPathList[[zip file "/data/app/com.microsoft.azure.mobile.xamarin.puppet-2/base.apk"],nativeLibraryDirectories=[/data/app/com.microsoft.azure.mobile.xamarin.puppet-2/lib/arm, /system/fake-libs, /data/app/com.microsoft.azure.mobile.xamarin.puppet-2/base.apk!/lib/armeabi-v7a, /system/lib, /vendor/lib]]

Yet this error message from the Android runtime is letting the developer know that their class did not make the dex list. So then the developer needs to figure out why that class didn't make the dex list. This can lead the developer down the wrong path of something like multidex/proguard/etc and eventually the linker.

2. Not many developers actually know what the Linker does.

We have some extensive documentation on the Linker that can be found here: https://developer.xamarin.com/guides/android/advanced_topics/linking/

I continuously see developers who notice their packages are too bloated so they go through this document and eventually set the linker to "Link All Assemblies (SDK and User Assemblies)" which will of course give them the best result; a small package. 

However enabling this almost always ends up with a developer running into an issue regarding their code being stripped because of no static references/using reflection/etc.

*Proposal:

Although this is only 2 initial reasons, I propose a diagnostic log that can help developers solve these issues.

Here are two ideas of how to achieve this. Please note that these are simply ideas and there is probably a more elegant solution available:
==========================================

1. The Mono Linker seems to already support an `output` path that emits the linked assemblies:

https://github.com/mono/linker/tree/master/linker

The output directory

By default, the linker will create an `output' directory in the current directory where it will emit the linked files, to avoid erasing source assemblies. You can specify the output directory with the option:

-o output_directory

One could compare the same source and linked assembly by running through ildasm/monodis and then do a general diff of the text output of the two assemblies.

2. For each of the major steps in the Linker such as:

https://github.com/mono/linker/blob/1bdcf6b7bfbe3b03fdaa76f6124d0d7374f08615/linker/Mono.Linker/Driver.cs#L328-L335

- ResolveFromAssembly / ResolveFromXml
- LoadReferences
- Blacklist
- Mark
- Sweep
- Clean
- Output

It seems that adding logging mechanisms around the "Mark" and "Sweep" areas could help provide an entire list of what is being marked and swept.

Mark:

> So the linker will annotate assemblies, types, methods and fields to know
> what should be linked or not, and what have been processed, and how it should
> process them.

> This is really useful as we don't have to recreate a full hierarchy of classes
> to encapsulate the different Cecil types to add the few informations we want.

Sweep:

> This simple step will walk through all the elements of an assembly, and based
> on their annotations, remove them or keep them.

This would probably be the more ideal method of logging rather than #1 as you'd already have a list of what is marked and what the sweeper actually removes.

*MSBuild Task Source:

https://github.com/xamarin/xamarin-android/blob/4e30546fbc2caced3c2faee5f024a572bce12f21/src/Xamarin.Android.Build.Tasks/Tasks/LinkAssemblies.cs

*Output:

The output should simply be a text file or xml file that lets the developer know what was all stripped during the LinkAssemblies MSBuild task.
Comment 1 Jon Douglas [MSFT] 2017-09-22 17:10:06 UTC
There now exists some great documentation and tooling by Radek!

https://github.com/mono/mono/blob/master/mcs/tools/linker-analyzer/README.md