Bug 18137 - Release builds do not include file name/line number in exception stack traces
Summary: Release builds do not include file name/line number in exception stack traces
Status: RESOLVED FIXED
Alias: None
Product: Android
Classification: Xamarin
Component: General ()
Version: 4.10.0.x
Hardware: Macintosh Windows
: Normal normal
Target Milestone: ---
Assignee: Jonathan Pryor
URL:
Depends on:
Blocks:
 
Reported: 2014-03-03 17:02 UTC by leankitryan
Modified: 2017-09-12 14:45 UTC (History)
14 users (show)

Tags: bb
Is this bug a regression?: ---
Last known good build:


Attachments
Sample solution (18.10 KB, application/x-zip-compressed)
2014-03-03 17:02 UTC, leankitryan
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description leankitryan 2014-03-03 17:02:34 UTC
Created attachment 6205 [details]
Sample solution

We are sending exceptions and crashes through Google Analytics. I've noticed the exception stack traces do not include any line numbers. Example

System.NullReferenceException: Object reference not set to an instance of an object at MyApp.MyMethod1 () [0x00000] in <filename unknown>:0 at MyApp.MyMethod2 () [0x00000] in <filename unknown>:0

If I run the same application in debug mode, the exception looks fine with line numbers in the stack trace.

See attached sample solution with a Xamarin Android project which demonstrates the issue. Line numbers are printed (to the Console) when ran in debug mode. Line numbers are omitted when ran in release mode. A console project is also included which includes line numbers regardless of whether the build is release or debug.

Does Xamarin Android not support this? Normal .NET applications support line numbers in stack traces even in release mode if the proper .pdb files are included. Without line numbers in stack traces troubleshooting exceptions is very difficult.
Comment 1 Jonathan Pryor 2014-03-03 17:59:05 UTC
> Normal .NET applications support line in stack traces even in
> release mode if the proper .pdb files are included

Mono is slightly different, in that by default stack trace information is not computed:

http://docs.go-mono.com/?link=man%3amono(1)
> --debug, --debug=OPTIONS
> Turns on the debugging mode in the runtime. If an assembly was compiled with
> debugging information, it will produce line number information for stack traces.

I forget why this isn't enabled by default; I believe it's because it increases memory use and may impact JIT times.

Regardless, there are three blockers:

1. --debug would need to be enabled by default.
2. .mdb files would need to be included into the .apk
3. It needs to be made to work ;-)

(1) You could enable --debug by setting the debug.mono.debug system property to 1:

http://docs.xamarin.com/guides/android/advanced_topics/environment/
# Add a new file with Build action=AndroidEnvironment, contents:
debug.mono.debug=1

(2) You also need the debug symbols to be present (the .mdb files), the .mdb files are not present within the .apk. No .mdb files, no line numbers, and by default .mdb files are not included (to reduce .apk size).

To include .mdb files, edit your Project Options and set the Debug Information setting to Full. (Not PdbOnly.) (XS: Project Options > Build / Compiler > Debug information.)

The result is that the $(DebugSymbols) MSBuild property should be True, and the $(DebugType) MSBuild property should be Full.

(3) Even with (1) and (2) done, it still doesn't work, probably because of bugs in libmonodroid.so. :-(
Comment 2 Mikayla Hutchinson [MSFT] 2014-03-03 18:13:47 UTC
Technically you do not need the MDB files in the APK. If you enable --debug and the MDB files are not present, then you will have IL offsets in traces, and you could write a tool to use the MDB files to compute line numbers from the IL offsets.

But enabling --debug will increase memory use and prevent the JIT from using certain optimizations.
Comment 3 leankitryan 2014-03-04 09:17:00 UTC
Thanks for taking the time to reply. The response is rather unfortunate, however. We are nearing release our application and are in the final stages of bug fixes. Some of the bugs aren't reproducible on our end. Without line numbers, troubleshooting those exceptions is going to be much more difficult. I wish I had known this sooner. It's a standard convenience in the regular .NET world that I took for granted - lesson learned.
Comment 4 Michael Carter 2014-05-05 19:33:15 UTC
I also would like to see this implemented. I think it's pretty critical to supporting released apps.

Is this something that is planned? I haven't seen any activity on this bug which was reported 2 months ago.

I'm surprised that this isn't supported by default.
Comment 5 Bernie Habermeier 2014-08-26 19:10:51 UTC
Wow -- what a bummer.  Running into the same issue on iOS.
Comment 6 Justin Toth 2015-03-06 11:35:34 UTC
I'd also like to see this capability, it's been painful to track down crashes from HockeyApp without the line numbers. What we've had to do is add null checks and throw custom exceptions, or break down functions into smaller functions, so that we can narrow down where the crashes are occurring.

I hope someone will come up with a creative way to make this work to ease our pain.
Comment 7 David Barrett 2015-03-06 11:51:32 UTC
Please, please, please consider making this work.  I understand the need to minimize memory and performance impact of symbolication, but production .Net apps do this all day long. I'm using HockeyApp for exception logging but experience the same behavior -- stack traces locally contain line numbers (even when built as "release" and deployed locally); exceptions on apps downloaded from the Play Store contain no line numbers.
Comment 8 Bernie Habermeier 2015-03-06 11:59:39 UTC
I don't think you'll need to include the full debugging symbols in your release build.  Check this thread here: http://forums.xamarin.com/discussion/comment/104665

But there is/was a bug (that appears to be now closed/fixed) that is related, and that was a fundamental issue here: https://bugzilla.xamarin.com/show_bug.cgi?id=19547

Without that piece, there is no way to get information we're looking for.  But after that, assuming you have the .dsym file -- you should be able to get full information we're looking for...
Comment 9 David Barrett 2015-03-06 12:02:45 UTC
Isn't that for iOS?
Comment 10 Bernie Habermeier 2015-03-06 12:04:11 UTC
Oh -- hah!  Good point.  Possibly related though.  Sorry didn't check the platform.  Blind desperation...
Comment 13 Jonathan Pryor 2015-09-09 07:51:16 UTC
*** Bug 33806 has been marked as a duplicate of this bug. ***
Comment 14 David 2015-12-18 21:48:15 UTC
Any news on this? I have exactly the same problem and try to solve it as Justin Toth explained it. That is not cool. Has anyone found a workaround other than abandoning Xamarin?
Comment 16 Eric 2017-03-24 11:12:38 UTC
I am using Xamarin.Forms SDK 2.3.3.193 and still have that issue. Whe will it be fixed?
Comment 17 Justin Toth 2017-03-24 13:27:19 UTC
We've been waiting 3 years, so don't hold your breath... :(
Comment 18 Eric 2017-03-24 13:28:05 UTC
Ha, ha ha :-)!!!
Comment 19 Tom Opgenorth 2017-06-28 19:34:59 UTC
Sounds like this feature has been added, see the Xamarin.Android 7.3 release notes at https://developer.xamarin.com/releases/android/xamarin.android_7/xamarin.android_7.3/#Release_Stack_Traces
Comment 20 Justin Toth 2017-07-18 20:59:45 UTC
Were you able to get it working? I'm not clear on a few things:

1. Where exactly we set $(DebugSymbols) to True and $(Optimize) to True?

2. If we manually have to call the mono-symbolicate tool to get line numbers, which defeats the purpose. We use HockeyApp and want the line numbers in their crashes, how can this be done?
Comment 21 Michael Carter 2017-07-18 22:04:36 UTC
@Justin

I don't think there's UI for it (at least not in Visual Studio that I can see). You need to manually edit your .csproj file:

  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <DebugSymbols>true</DebugSymbols>
    <DebugType>full</DebugType>
    <Optimize>true</Optimize>
    ...
  </PropertyGroup>

As for having to use the mono-symbolicate tool, yes it's not ideal, but better than we had it. At least now we can get the line numbers.

First make sure you have the tool. It should be in C:\Program Files (x86)\MSBuild\Xamarin\Android\

If you don't see it, you may need to actually install Xamarin.Android from the Xamarin site. I didn't have it when I installed the mobile development workload in VS 2017 and couldn't figure out why it was missing.

Once you enable DebugSymbols in your Release build, you will now see the mSYM folder generated in your Archives directory.

This is located in (placeholders added):

C:\Users\USERNAME\AppData\Local\Xamarin\Mono for Android\Archives\DATE\PROJNAME DATETIME.apkarchive\mSYM\PACKAGEID.apk.mSYM\

This is the folder you will need to pass to mono-symbolicate along with a text file that has your stack trace.

I also recommend you use the -q (quiet) switch. mono-symbolicate tries to find symbols for all 3rd party libs in your app, and if you don't have symbols it will add warnings in your output.

I agree that it would be nice if tools like HockeyApp would automatically add the line numbers, but until that happens, it shouldn't be too difficult to create a Chrome extension that "scrapes" the stack trace from the web page and passes it to a local service that calls mono-symbolicate for you.

I'll probably write something like that for our own use, so I'll post a link if I do.

Good luck!
Comment 22 Justin Toth 2017-07-19 14:01:27 UTC
Thanks for the tips, however I'm having trouble finding the mSYM files... I added the necessary elements to the release propertygroup, and then compiled my app in Release Mode. I'm on Mac, and can't find any paths similar to the windows path that you mentioned. Also, if I do a search for mSYM via Finder, nothing comes up.
Comment 23 Michael Carter 2017-07-19 14:15:07 UTC
I haven't tried on a Mac. I'm out of the office today, but I'll take a look when I get back and get you the correct paths for macOS.
Comment 24 René Simonsen 2017-09-07 22:03:46 UTC
I also installed Xamarin from the VS2017 payload, and I cannot find mono-symbolicate. Where is it/where do I get it?
Comment 25 Michael Carter 2017-09-07 22:22:10 UTC
You have to download and install Xamarin from the Xamarin site in order to get those tools. Apparently the VS 2017 workload is a subset.

https://www.xamarin.com/download

BTW: I wrote a tool that will call mono-symbolicate with the correct parameters. You simply pass it the package ID and version (build) and it will locate the mSYM folder.

I created a local service and wrote a Chrome extension that will read the obfuscated stack trace from the web page (in this case, Crashlytics) and it would send that to the local service, which ran mono-symbolicate, then returned the cleaned stack trace which the extension would update on the web page. It actually works really well.

NOTE: If you're using Crashlytics, the stack trace that is sent to the cloud is not in the proper format for mono-symbolicate. It uses the Java format, so it doesn't include the file hash or offset. I had to use reflection to modify the RegEx the NuGet package used for parsing the stack trace.

Anyway, it's still kind of rough, but I'll try and get it posted to GitHub in case anyone is interested.
Comment 26 René Simonsen 2017-09-07 22:40:34 UTC
That link to download, just gets me a VS 2017 installer. Can I download the tools elsewhere?

Also, that tool sounds really nice! Any chance for HockeyApp aswell? :)
Comment 27 Michael Carter 2017-09-07 22:45:28 UTC
Try this link. You'll need to have a Xamarin account.

https://store.xamarin.com/account/my/subscription/downloads

I'm not using HockeyApp at the moment, but I can setup a free account. It's just a matter of updating the selector for finding the stack trace node.
Comment 28 René Simonsen 2017-09-07 23:11:21 UTC
Thank you. That link seems to work still. If you want, I can send you what you need for HockeyApp selector. But I guess this isn't the most relevant place
Comment 29 Justin Toth 2017-09-07 23:32:33 UTC
+1 for HockeyApp support, that'd be amazing!
Comment 30 René Simonsen 2017-09-07 23:35:18 UTC
In HockeyApp there's actually 3 different views for stacktrace.

1: Rich view (syntax highlighting, would be too advanced to replace I guess)
2: Raw view (pre tag inside a div with class="raw-wrapper")
3: Log view - almost the same as Raw view, but with some device info at the top.

I'm guessing for ease of implementation, just replacing #2 would be fine.
Comment 31 René Simonsen 2017-09-07 23:47:20 UTC
Here's a screenshot of dom tree around the stacktrace on #2
http://i.imgur.com/TAmxxgq.png
Comment 32 René Simonsen 2017-09-08 00:15:40 UTC
Xamarin installer failed :/ I guess because I don't have old VS on my pc.
Comment 33 René Simonsen 2017-09-08 10:01:49 UTC
Could anyone maybe zip the Xamarin folder from MSBuild and upload it? I would appreciate it!

Also, @Justin - would you release your service + chrome plugin?
Comment 34 Michael Carter 2017-09-08 14:27:19 UTC
I posted the helper service to GitHub. https://github.com/kiliman/mono-symbolicate-helper

The Chrome extension isn’t ready for prime time yet, but you can still use the helper service by making a REST call to the symbolicate endpoint. The instructions are in the README.

Let me know how it works for you. Good luck!
Comment 35 Michael Carter 2017-09-11 23:17:42 UTC
I have published a Chrome extension that will automatically run mono-symbolicate on stack traces within HockeyApp. It actually works really well. It updates both the rich formatted and raw versions of the stack trace.

Check it out at https://chrome.google.com/webstore/category/extensions?hl=en-US&gl=US

You will need to install the helper service from https://github.com/kiliman/mono-symbolicate-helper

Enjoy!
Comment 37 Justin Toth 2017-09-12 13:50:37 UTC
Thanks @Michael Carter for your work on this, I'm excited to try it. I installed the Chrome Extension, now I'm trying to get the service up and running. After building it in Visual Studio for Mac, I opened the config file and am trying to figure out what to set the archivePath and configPath to. I'm not really clear because you were doing this on Windows, and I'm on OSX. The mono-symbolicate tool must exist somewhere on OSX, as I can use it from command line, however it doesn't come up when I do a search in Finder so I'm unclear how to proceed...
Comment 38 Justin Toth 2017-09-12 14:01:12 UTC
I found the Mono for Android folder, /Users/{username}/.local/share/Xamarin/Mono for Android, however there is no Archives subfolder in there.
Comment 39 René Simonsen 2017-09-12 14:27:57 UTC
@Justin Toth: In VS for Windows, when I've Archived a version - there's a button to open folder. Isn't there something similar in VS for Mac?
If there is, you may need to go up a few levels to get the root of all archive folders.
Comment 40 Justin Toth 2017-09-12 14:42:27 UTC
Got it, thanks. /Users/{username}/Library/Developer/Xamarin/Archives

Now I just need to figure out where mono-symbolicate is...
Comment 41 Michael Carter 2017-09-12 14:45:58 UTC
@Justin

At the console type: 

   which mono-symbolicate

This will give you the path of the command.

If you have any issues with the helper service, please file as issues on GitHub so we don't spam this bug report.

Good luck!