Bug 15202 - [regression] Crash reports no longer contain file/line number information
Summary: [regression] Crash reports no longer contain file/line number information
Status: RESOLVED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: XI runtime ()
Version: 7.0.0.x
Hardware: PC Mac OS
: --- normal
Target Milestone: 7.0.6
Assignee: Zoltan Varga
URL:
: 15812 ()
Depends on:
Blocks:
 
Reported: 2013-10-04 08:19 UTC by Rolf Bjarne Kvinge [MSFT]
Modified: 2016-01-11 17:32 UTC (History)
10 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Rolf Bjarne Kvinge [MSFT] 2013-10-04 08:19:21 UTC
Repro:
* Clone https://github.com/rolfbjarne/TestApp
* Run on a device.
* Click one of the buttons to make it crash.
* Open the crash report in Xcode.

Result: 

14  testapp  0x0018a70a mono_arm_throw_exception (exceptions-arm.c:161)
15  testapp  0x00132a94 ___lldb_unnamed_function5991$$testapp + 64
16  testapp  0x0015598c ___lldb_unnamed_function8419$$testapp + 68
17  testapp  0x0015ae94 ___lldb_unnamed_function8529$$testapp + 240
18  testapp  0x0015c0cc ___lldb_unnamed_function8554$$testapp + 92
19  testapp  0x00117534 ___lldb_unnamed_function5453$$testapp + 196
20  testapp  0x001a72a6 mono_jit_runtime_invoke (mini.c:6471)
Comment 1 asp_net 2013-11-28 04:40:37 UTC
I am hitting this, too. Please try to get it fixed asap!
Comment 2 Zoltan Varga 2013-12-01 08:19:13 UTC
I can no longer reproduce this with master.
Comment 3 asp_net 2013-12-01 16:38:47 UTC
Thanks for the update - when will that be available via alpha/beta channel?
Comment 4 Rolf Bjarne Kvinge [MSFT] 2013-12-02 06:24:55 UTC
I can still reproduce (but I had to enable LLVM first).

There seems to be two problems in fact (not sure if they're related or not):

1) The normal debug builds will get the function name right, but no file / line number information:

8   testapp  0x00037138 TestApp.AppDelegate:NativeCrash + 152
9   testapp   0x00037d84 TestApp.AppDelegate:<FinishedLaunching>m__2 + 120
10  testapp  0x001530cc MonoTouch_Dialog_StringElement_Selected_MonoTouch_Dialog_DialogViewController_MonoTouch_UIKit_UITableView_MonoTouch_Foundation_NSIndexPath + 68
11  testapp  0x001585d4 MonoTouch_Dialog_DialogViewController_Selected_MonoTouch_Foundation_NSIndexPath + 240
12  testapp  0x0015980c MonoTouch_Dialog_DialogViewController_Source_RowSelected_MonoTouch_UIKit_UITableView_MonoTouch_Foundation_NSIndexPath + 92

2) If I select the Release configuration and enable LLVM, I get the same issue as in the original description.

Tested with monotouch/fc0a1e46be.
Comment 5 Zoltan Varga 2013-12-02 14:54:33 UTC
The stacktrace in the ios console contains file/line numbers, but the crash report in the 'Device Logs' section does not.
Comment 6 asp_net 2013-12-02 15:22:35 UTC
Please make sure it is available within Instruments, too (I assume it is fetched from the .dSYM there?). Currently it is not possible to trace a memory leak (for example) down to the point in source causing it. What is a serious issue for everyone wanting to optimize their applications.
Comment 7 Zoltan Varga 2013-12-02 18:24:57 UTC
The file/line number support was broken when we switched to using clang as the assembler, we emit the generated machine code using .byte directives and clang doesn't consider those instructions, so it ignores the .loc directives emitted before those.
We can hack around that by emitting a nop at each pc which has a line number, but it will slow down code in release mode. Alternatively, llvm has an assembler called llvm-mc, and we can switch to that but it is risky, and it might not support arm64 at all.
Comment 8 Rolf Bjarne Kvinge [MSFT] 2013-12-03 04:53:32 UTC
I tried using clang's -integrated-as (which is supposedly using llvm-mc), and it didn't work either (I also tried -no-integrated-as, but nothing changed).
Comment 9 Zoltan Varga 2013-12-03 07:04:20 UTC
clang might be using an older version of llvm which had this problem.
Comment 10 Rolf Bjarne Kvinge [MSFT] 2013-12-04 12:15:28 UTC
Have you been able to make llvm-mc work?

I get an error:

./llvm-mc: error: invalid target 'armv7'.

And from the documentation it looks like it's not supported for arm: http://llvm.org/docs/CodeGenerator.html#target-feature-matrix
Comment 11 Zoltan Varga 2013-12-04 16:21:51 UTC
I used llvm-mc -arch=arm -filetype=obj. That seemed to work, but I'm not sure how reliable it is.
Comment 12 Rolf Bjarne Kvinge [MSFT] 2013-12-05 10:47:35 UTC
So I got llvm-mc to work, but nothing changed. I tried the llvm-mc we build as a part of our monotouch build, and also the latest version from their git. I also tried the latest clang version from their git, no difference, still no file names or line numbers.

BTW llvm-mc does not like the assembly files we create when LLVM is enabled, it reports a lot of errors.
Comment 13 Zoltan Varga 2013-12-05 12:38:21 UTC
The alternative is to add a hack which inserts a nop before each line number positions, that relatively easy to implement. Did we ever support file/line numbers in release mode ?
Comment 14 Rolf Bjarne Kvinge [MSFT] 2013-12-05 16:50:53 UTC
Yes, we did support file/line numbers in release mode (quite essential for crash reporting in released apps in fact).

Yet it would be an improvement if we could at least get debug builds to work properly.
Comment 15 Zoltan Varga 2013-12-06 09:31:10 UTC
This is somewhat better now with 98dbdd9cc505b7ccf150b641221139b00ec13c69. I get file/line numbers in debug mode.
Comment 16 asp_net 2013-12-13 15:44:10 UTC
Any news on this? It is currently almos impossible to debug an app regarding memory issues without getting to the source trough Instruments. Again, that's serious!
Comment 17 Rolf Bjarne Kvinge [MSFT] 2013-12-13 20:48:52 UTC
Zoltan, I can confirm it's better now, I get file/line numbers for some frames in both debug and release mode.

Sample crash (from the original test case): https://gist.github.com/rolfbjarne/dc52ea9e797a0c7eee00

Note how frame 9 is symbolicated correctly, while frames 10-12 are still lacking file/line number information.
Comment 18 Rolf Bjarne Kvinge [MSFT] 2013-12-13 20:49:05 UTC
@asp_net, the improvement mentioned in comment 15 will be included in 7.0.6 and 7.1 (soon to be released).
Comment 19 Zoltan Varga 2013-12-15 19:29:18 UTC
Made some progress in mono 3312a1af0c5bae33ec76b3ce45fb892d7c68447c, now the dSYM file contains debug info for all assemblies, not just the main one.  Some frames still have no line number info because they make indirect calls, and the AOT compiler emits those using .byte directives, not assembly, i.e. comment #7.
Comment 20 Zoltan Varga 2014-01-14 22:25:07 UTC
Fixed the majority of this in mono f18648d642ca963c8dc02c5b41bb70276b65c207/mt fa6a7b38f86fb1806678917aa37005b0817efbf5. Debug builds now work ok with the testcase, release builds still have some lldb_unnamed symbols, but those look like wrappers, c# frames have normal names and line number info.
Comment 21 Rolf Bjarne Kvinge [MSFT] 2014-01-20 04:20:32 UTC
*** Bug 15812 has been marked as a duplicate of this bug. ***
Comment 22 Rolf Bjarne Kvinge [MSFT] 2014-01-21 06:34:37 UTC
Fixes from comment #19 and comment 20 backported to the 7.1 and 7.0.6 branches (the fix from comment #15 is already in both).

mono/monotouch-7.0.6-branch: 04f55b81b2b14dd200c009cf2e4d75c996b84a97
monotouch/monotouch-7.0.6-branch: 2ed9bc96337da73a7e5d00ed6c77f63f2489bb24
monotouch/ios7.1: 5d420dd6f5964931c6f07cc401639f5a180c4e8c
Comment 23 GouriKumari 2014-01-21 15:24:07 UTC
@Rolf:
Could you take a look at the crash reports before and after applying the fix mentioned in Comment #22. I couldn't verify the bug since it looked similar.  

TestApp CrashReport with MT 7.0.6 2ed9bc96337da73a7e5d00ed6c77f63f2489bb24: 
https://gist.github.com/GouriKumari/0646b09a9e59443b9d3c

TestApp CrashReport with beta 7.0.6 : 
https://gist.github.com/GouriKumari/b22210c5b77aca1e04cf
Version: 7.0.6.167 (Enterprise Edition)
Hash: 9ec960a
Comment 24 Rolf Bjarne Kvinge [MSFT] 2014-01-22 09:08:38 UTC
Interestingly only Xcode 5.1 DP4 seem to symbolicate correctly, Xcode 5.0 does not.

Gouri, can you try again with Xcode 5.1? Note that you don't have to rebuild nor change anything else in the XI project, you only get the crash report using Xcode 5.1.
Comment 25 GouriKumari 2014-01-22 11:32:57 UTC
CrashReport from Xcode51-DP4
https://gist.github.com/GouriKumari/148e17963a1193e93ea9

Test Env:
Xamarin Studio
Version 4.3.1 (build 5)
Installation UUID: 5ed3a124-4b77-4c6f-beb9-c830fd815e2a
Runtime:
	Mono 3.2.6 ((no/9b58377)
	GTK+ 2.24.23 theme: Raleigh
	GTK# (2.12.0.0)
	Package version: 302060000

Apple Developer Tools
Xcode 5.1 (5051.4)
Build 5B90f

Xamarin.iOS
Version: 7.1.3.7 (Enterprise Edition)
Hash: f69fbdf
Branch: 
Build date: 2014-21-01 16:35:21-0500
Comment 26 Rolf Bjarne Kvinge [MSFT] 2014-01-22 13:14:35 UTC
Yeah, that's a lot better.

This sounds like Apple fixed a bug in Xcode 5.1, so I suppose we'll just have to document that Xcode 5.0 won't be able to symbolicate properly.
Comment 27 Zoltan Varga 2014-01-22 14:37:44 UTC
Rolf: I have xcode 5.0, and symbolifications works ok for me.
Comment 28 Rolf Bjarne Kvinge [MSFT] 2014-01-23 09:53:57 UTC
Zoltan, did you test using mt master or the 7.0.6 branch?
Comment 31 Zoltan Varga 2014-01-23 13:24:56 UTC
Rolf: I tested using master. It looks like the 7.0.6 branch (2ed9bc96337da73a7e5d00ed6c77f63f2489bb24) contains both the required changes, so I don't know what the problem is. Also, what config doesn't work (debug/release/release-llvm) ?
Comment 32 Rolf Bjarne Kvinge [MSFT] 2014-01-23 19:23:55 UTC
Xcode 5.0/master/Debug: OK.
Xcode 5.0/master/Release: OK (a couple of wrapper_* methods shows up as lldb_unnamed_function).
Xcode 5.0/master/Release+llvm: many lldb_unnamed_functions.

Xcode 5.1/master/Debug: OK.
Xcode 5.1/master/Release: OK
Xcode 5.1/master/Release+llvm: OKish (functions symbolicated as C names instead of C# names, no filename/line numbers)

Xcode 5.0/7.0.6/Debug: OK
Xcode 5.0/7.0.6/Release: OK (a couple of wrapper_* methods shows up as lldb_unnamed_function).
Xcode 5.0/7.0.6/Release+llvm: many lldb_unnamed_functions.

Xcode 5.1/7.0.6/Debug: OK
Xcode 5.1/7.0.6/Release: OK
Xcode 5.1/7.0.6/Release+llvm:  OKish (functions symbolicated as C names instead of C# names, no filename/line numbers)

To sum it up:
* Debug works in all cases.
* Release has a couple of minor lldb_unnamed_function w/Xcode 5.0 (but not 5.1).
* Release+llvm is quite unusable with Xcode 5.0, while it works OKish with Xcode 5.1
* I couldn't find any difference between 7.0.6 and master.

So any remaining issues should not hold up 7.0.6, since they haven't been fixed in master either.

Links to crash reports:

Xcode5.0/master/Debug: https://gist.github.com/rolfbjarne/214fe32f9159bc3b4bb6
Xcode5.0/master/Release: https://gist.github.com/rolfbjarne/21ffd3d32ae7d9bf04ee
Xcode5.0/master/Release+llvm: https://gist.github.com/rolfbjarne/8b9f671c0d132af40924
Xcode5.1/master/Debug: https://gist.github.com/rolfbjarne/4a55a37696b3da224db5
Xcode5.1/master/Release: https://gist.github.com/rolfbjarne/ed1688267487a9e7964b
Xcode5.1/master/Release+llvm: https://gist.github.com/rolfbjarne/5e8879d65b50488a9d80

Xcode5.0/7.0.6/Debug: https://gist.github.com/rolfbjarne/8a7d384492712d78eb1c
Xcode5.0/7.0.6/Release: https://gist.github.com/rolfbjarne/f45d4a36b28398cc4d39
Xcode5.0/7.0.6/Release+llvm: https://gist.github.com/rolfbjarne/92c2fb3bc691837fa2db
Xcode5.1/7.0.6/Debug: https://gist.github.com/rolfbjarne/83de2d2c353a71886a12
Xcode5.1/7.0.6/Release: https://gist.github.com/rolfbjarne/05e961fe20ff94b13c4d
Xcode5.1/7.0.6/Release+llvm: https://gist.github.com/rolfbjarne/344635ac8ef64c5de1fc
Comment 33 Zoltan Varga 2014-01-24 02:05:50 UTC
File/line info for LLVM functions was never supported, we didn't emit debug information for LLVM code.
Comment 34 Nate Cook 2015-02-26 11:37:32 UTC
Are there any plans to emit debug information for LLVM code in the future?
Comment 35 Zoltan Varga 2015-02-26 11:47:54 UTC
Yes, it will be supported in our next major release.