Bug 53286 - mono-2017-02 + xamarin-android results in apparent StackOverflow
Summary: mono-2017-02 + xamarin-android results in apparent StackOverflow
Status: RESOLVED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: JIT ()
Version: unspecified
Hardware: PC Mac OS
: --- normal
Target Milestone: ---
Assignee: Andi McClure
URL:
Depends on:
Blocks:
 
Reported: 2017-03-13 18:28 UTC by Jonathan Pryor
Modified: 2017-03-20 16:31 UTC (History)
3 users (show)

Tags: 2017-02
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Jonathan Pryor 2017-03-13 18:28:27 UTC
There is a xamarin-android PR which uses mono's 2017-02 branch:

  https://github.com/xamarin/xamarin-android/pull/445

The most recent Jenkins build has symptoms which suggest a stack overflow during process startup on an Android emulator:

  https://jenkins.mono-project.com/job/xamarin-android-pr-builder/661/

Specifically, if you look at the full build log:

  https://jenkins.mono-project.com/job/xamarin-android-pr-builder/661/consoleText

Search backwards from the end for `logcat -d`, then forward to `beginning of crash`:

> --------- beginning of crash
> W/VoiceInteractionManagerService( 1223): no available voice recognition services found for user 0
> D/DefContainer( 2151): Copying /data/local/tmp/Xamarin.Android.Locale_Tests-Signed.apk to base.apk
> D/PackageManager( 1223): Renaming /data/app/vmdl406539729.tmp to /data/app/Xamarin.Android.Locale_Tests-1
> V/BackupManagerService( 1223): restoreAtInstall pkg=Xamarin.Android.Locale_Tests token=3 restoreSet=0
> V/BackupManagerService( 1223): Finishing install immediately
> D/BackupManagerService( 1223): Received broadcast Intent { act=android.intent.action.PACKAGE_ADDED dat=package:Xamarin.Android.Locale_Tests flg=0x4000010 (has extras) }
> W/VoiceInteractionManagerService( 1223): no available voice recognition services found for user 0
> I/ActivityManager( 1223): Force stopping Mono.Android_Tests appid=10053 user=0: start instr
> I/ActivityManager( 1223): Start proc Mono.Android_Tests for added application Mono.Android_Tests: pid=2236 uid=10053 gids={50053, 9997, 3003} abi=x86
> --------- beginning of crash
> F/libc    ( 2236): Fatal signal 11 (SIGSEGV), code 2, fault addr 0xbf69dff0 in tid 2236 (o.Android_Tests)
> I/DEBUG   (  930): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
> I/DEBUG   (  930): Build fingerprint: 'generic_x86/sdk_phone_x86/generic_x86:5.0.2/LSY64/1772600:eng/test-keys'
> I/DEBUG   (  930): Revision: '0'
> I/DEBUG   (  930): ABI: 'x86'
> I/DEBUG   (  930): pid: 2236, tid: 2236, name: o.Android_Tests  >>> Mono.Android_Tests <<<
> I/DEBUG   (  930): signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xbf69dff0
> I/DEBUG   (  930):     eax 0000000e  ebx b76dd800  ecx 00000002  edx b76c0cc0
> I/DEBUG   (  930):     esi 00000020  edi 00000014
> I/DEBUG   (  930):     xcs 00000073  xds 0000007b  xes 0000007b  xfs 00000007  xss 0000007b
> I/DEBUG   (  930):     eip b769e8ff  ebp b76ea46c  esp bf69dff0  flags 00210212
> I/DEBUG   (  930): 
> I/DEBUG   (  930): backtrace:
> I/DEBUG   (  930):     #00 pc 0009a8ff  /system/lib/libc.so (je_calloc+127)
> I/DEBUG   (  930):     #01 pc 00012745  /system/lib/libc.so (calloc+37)
> I/DEBUG   (  930):     #02 pc 00487408  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (g_calloc+88)
> I/DEBUG   (  930):     #03 pc 004874b5  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (monoeg_malloc0+53)
> I/DEBUG   (  930):     #04 pc 002f60c3  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_metadata_parse_mh_full+243)
> I/DEBUG   (  930):     #05 pc 002bc292  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_method_get_header_checked+914)
> I/DEBUG   (  930):     #06 pc 0003fbf7  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_method_to_ir+743)
> I/DEBUG   (  930):     #07 pc 0001e247  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mini_method_compile+5223)
> I/DEBUG   (  930):     #08 pc 000235f9  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_compile_method_inner+2377)
> I/DEBUG   (  930):     #09 pc 00029532  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_compile_method_with_opt+1698)
> I/DEBUG   (  930):     #10 pc 00028e78  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_compile_method+88)
> I/DEBUG   (  930):     #11 pc 001522f9  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (common_call_trampoline+3177)
> I/DEBUG   (  930):     #12 pc 00151649  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_magic_trampoline+137)
> I/DEBUG   (  930):     #13 pc 00010187  <unknown>
> I/DEBUG   (  930):     #14 pc 000b7a8f  [anon:libc_malloc]
> I/DEBUG   (  930):     #15 pc 000037a3  <unknown>
> I/DEBUG   (  930):     #16 pc 000038c1  <unknown>
> I/DEBUG   (  930):     #17 pc 0002c90a  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_runtime_invoke+1386)
> I/DEBUG   (  930):     #18 pc 00369a09  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (do_runtime_invoke+249)
> I/DEBUG   (  930):     #19 pc 00362a61  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_runtime_invoke_checked+193)
> I/DEBUG   (  930):     #20 pc 0028adfb  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_get_exception_type_initialization_checked+539)
> I/DEBUG   (  930):     #21 pc 00363cca  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (get_type_init_exception_for_vtable+362)
> I/DEBUG   (  930):     #22 pc 00363289  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_runtime_class_init_full+521)
> I/DEBUG   (  930):     #23 pc 0006a237  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_method_to_ir+174375)
> I/DEBUG   (  930):     #24 pc 0001e247  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mini_method_compile+5223)
> I/DEBUG   (  930):     #25 pc 000235f9  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_compile_method_inner+2377)
> I/DEBUG   (  930):     #26 pc 00029532  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_compile_method_with_opt+1698)
> I/DEBUG   (  930):     #27 pc 00028e78  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_jit_compile_method+88)
> I/DEBUG   (  930):     #28 pc 001522f9  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (common_call_trampoline+3177)
> I/DEBUG   (  930):     #29 pc 00151649  /data/app/Mono.Android_Tests-1/lib/x86/libmonosgen-2.0.so (mono_magic_trampoline+137)

The "second" callstack block ("mono_jit_runtime_invoke+1386") is repeated twice.

Later, we see:

> --------- beginning of main
> E/mono-rt ( 2305):   at System.TypeInitializationException..ctor (string,System.Exception) [0x00000] in <ef00e35a9b7f4305ad66a1412a410e6c>:0
> E/mono-rt ( 2305):   at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object_object (object,intptr,intptr,intptr) <0x000c1>
> E/mono-rt ( 2305):   at <unknown> <0xffffffff>
> E/mono-rt ( 2305):   at System.Environment.GetResourceString (string,object[]) [0x00000] in <ef00e35a9b7f4305ad66a1412a410e6c>:0
> E/mono-rt ( 2305):   at System.TypeInitializationException..ctor (string,System.Exception) [0x00000] in <ef00e35a9b7f4305ad66a1412a410e6c>:0
> E/mono-rt ( 2305):   at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object_object (object,intptr,intptr,intptr) <0x000c1>
> E/mono-rt ( 2305):   at <unknown> <0xffffffff>
> E/mono-rt ( 2305):   at System.Environment.GetResourceString (string,object[]) [0x00000] in <ef00e35a9b7f4305ad66a1412a410e6c>:0
> E/mono-rt ( 2305):   at System.TypeInitializationException..ctor (string,System.Exception) [0x00000] in <ef00e35a9b7f4305ad66a1412a410e6c>:0
> E/mono-rt ( 2305):   at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object_object (object,intptr,intptr,intptr) <0x000c1>
> E/mono-rt ( 2305):   at <unknown> <0xffffffff>

which *really* looks like a stack overflow.

Finally, we see:

https://github.com/mono/referencesource/blob/mono/mscorlib/system/environment.cs#L114-L131

  // We have a somewhat common potential for infinite 
  // loops with mscorlib's ResourceManager.  If "potentially dangerous"
  // code throws an exception, we will get into an infinite loop
  // inside the ResourceManager and this "potentially dangerous" code.
  // Potentially dangerous code includes the IO package, CultureInfo,
  // parts of the loader, some parts of Reflection, Security (including 
  // custom user-written permissions that may parse an XML file at
  // class load time), assembly load event handlers, etc.  Essentially,
  // this is not a bounded set of code, and we need to fix the problem.
  // Fortunately, this is limited to mscorlib's error lookups and is NOT
  // a general problem for all user code using the ResourceManager.
  
  // The solution is to make sure only one thread at a time can call 
  // GetResourceString.  Also, since resource lookups can be 
  // reentrant, if the same thread comes into GetResourceString
  // twice looking for the exact same resource name before 
  // returning, we're going into an infinite loop and we should 
  // return a bogus string.  

That sounds suspiciously related to what we observe above.
Comment 1 Jonathan Pryor 2017-03-14 19:41:45 UTC
Additionally, this appears to be specific to the emulator; it doesn't reproduce on a Nexus 6P or 5x.

"Saner" (lol?) repro instructions:

0. Run macOS and have HAXM installed.

1. Download: https://xamjenkinsartifact.blob.core.windows.net/xamarin-android/xamarin-android/oss-xamarin.android_v7.2.99.39_Darwin-x86_64_pr_445_merge_04bfbd7.zip

2. Checkout git@github.com:xamarin/xamarin-android.git

3. Remove/comment out the following lines:

https://github.com/xamarin/xamarin-android/blob/mono-2017-02/src/Mono.Android/Test/Mono.Android-Tests.csproj#L103-L122

This is removing the @(ProjectReference) items, which aren't needed for this repro.

4. Follow: https://github.com/xamarin/xamarin-android/blob/master/Documentation/UsingJenkinsBuildArtifacts.md#command-line-use-linux-and-macos

4.a:

> $HOME/Downloads/oss-xamarin.android_v7.2.99.39_Darwin-x86_64_pr_445_merge_04bfbd7/bin/debug/bin/xabuild \
    /t:SignAndroidPackage \
    src/Mono.Android/Test/*.csproj

(4.a) should create xamarin-android/bin/TestDebug/Mono.Android_Tests-Signed.apk

5. Run:
> android create avd --abi "x86" -f -n "XamarinAndroidUnitTestRunner" -t "android-21"

6. Run:
> emulator -avd XamarinAndroidUnitTestRunner

7. Run:
> adb install xamarin-android/bin/TestDebug/Mono.Android_Tests-Signed.apk

8. Run:
> adb shell am instrument  -w "Mono.Android_Tests/xamarin.android.runtimetests.TestInstrumentation"

Given that the emulator we're using is an x86 emulator, and it works on arm64 devices, I have to guess that this is x86 related.
Comment 2 Jonathan Pryor 2017-03-14 20:10:00 UTC
More helpful tips:

1. Enable line numbers in stack trace:

> adb shell setprop debug.mono.debug 1

2. If (when?) you need to edit mscorlib sources, run the following from the `mono/mcs/class/corlib` directory:

> make PROFILE=monodroid clean all && adb push ../lib/monodroid/mscorlib.dll  /data/data/Mono.Android_Tests/files/.__override__ && adb push ../lib/monodroid/mscorlib.pdb /data/data/Mono.Android_Tests/files/.__override__

Then you can re-run Step (8) from Comment #1 to use the updated mscorlib.dll.
Comment 3 Jonathan Pryor 2017-03-14 21:19:52 UTC
@andi.mcclure: Alternate repro steps:

1. Build the xamarin-android repo with the mono-2017-02 branch, as per Comment #0.


2. Build the unit tests:
> tools/scripts/xabuild /t:SignAndroidPackage src/Mono.Android/Test/Mono.Android-Tests.csproj

# Note: ensure that *no Android devices are attached*, so that the emulator is launched.
3. Start the x86 emulator:
> xbuild /t:AcquireAndroidTarget tests/RunApkTests.targets


4. Install the test on the emulator:
> adb install bin/TestDebug/Mono.Android_Tests-Signed.apk

5. Run the tests:
> adb shell am instrument  -w "Mono.Android_Tests/xamarin.android.runtimetests.TestInstrumentation"
Comment 4 Jonathan Pryor 2017-03-20 16:31:47 UTC
For closure, the "apparent stack overflow" was caused by a TLS (Thread Local Storage) bug, fixed in one or both of:

https://github.com/mono/mono/commit/cd2579f7ceb62241c3a549bddcdd6a71f1b10625
https://github.com/mono/mono/commit/24e82a538588116cbd490eac44ff98fc9f4a9817

After these changes were applied, Android/x86 is able to execute the unit tests.