Bug 57872 - block_guard_restore_aligment_on_exit.exe crashes consistently on FullAOT ARM runs on Jenkins
Summary: block_guard_restore_aligment_on_exit.exe crashes consistently on FullAOT ARM ...
Status: RESOLVED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: JIT ()
Version: master
Hardware: PC Mac OS
: --- normal
Target Milestone: ---
Assignee: Zoltan Varga
URL:
Depends on:
Blocks:
 
Reported: 2017-06-29 22:41 UTC by Alexander Köplinger [MSFT]
Modified: 2017-07-07 12:45 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Alexander Köplinger [MSFT] 2017-06-29 22:41:53 UTC
E.g. https://jenkins.mono-project.com/job/test-mono-mainline-fullaot/label=debian-8-armhf/355/testReport/MonoTests/runtime/block_guard_restore_aligment_on_exit_exe/:

This crashes on armel, armhf and arm64 100% of the time and seems to be the only test that keeps us from getting green ARM FullAOT runs on Jenkins.

>   Id   Target Id         Frame 
>   4    Thread 0xb67ff450 (LWP 31099) "SGen worker" 0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
>   3    Thread 0xb6d07450 (LWP 31101) "Finalizer" 0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
>   2    Thread 0xb48ff450 (LWP 31103) "mono" 0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> * 1    Thread 0xb6fb1000 (LWP 31081) "mono" 0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> 
> Thread 4 (Thread 0xb67ff450 (LWP 31099)):
> #0  0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1  0xb6edf144 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2  0x00196460 in mono_os_cond_wait (mutex=0x29b804 <lock>, cond=0x29b820 <work_cond>) at ../../mono/utils/mono-os-mutex.h:173
> #3  get_work (job=<optimized out>, do_idle=<optimized out>, work_context=<optimized out>, worker_index=<optimized out>) at sgen-thread-pool.c:164
> #4  thread_func (worker_index=0) at sgen-thread-pool.c:193
> #5  0xb6edbf88 in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #6  0xb6e520fc in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> 
> Thread 3 (Thread 0xb6d07450 (LWP 31101)):
> #0  0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1  0xb6ee08c2 in do_futex_wait () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2  0xb6ee092e in sem_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #3  0x001477de in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x29393c <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:210
> #4  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x29393c <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:43
> #5  finalizer_thread (unused=<optimized out>) at gc.c:862
> #6  0x0012c352 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:839
> #7  start_wrapper (data=0x31e3f8) at threads.c:901
> #8  0xb6edbf88 in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #9  0xb6e520fc in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> 
> Thread 2 (Thread 0xb48ff450 (LWP 31103)):
> #0  0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1  0xb6ee2306 in waitpid () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2  0x0008b842 in mono_handle_native_crash (signal=signal@entry=0x1b2b54 "SIGSEGV", ctx=ctx@entry=0xb48fe548, info=info@entry=0xb48fe4c8) at mini-exceptions.c:2736
> #3  0x0002db28 in mono_sigsegv_signal_handler (_dummy=11, _info=0xb48fe4c8, context=0xb48fe548) at mini-runtime.c:3128
> #4  0xb6de3b00 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
> #5  0x0012cb7a in mono_os_mutex_unlock (mutex=<optimized out>) at ../../mono/utils/mono-os-mutex.h:121
> #6  mono_coop_mutex_unlock (mutex=<optimized out>) at ../../mono/utils/mono-coop-mutex.h:70
> #7  unlock_thread (thread=0xb68004b0) at threads.c:424
> #8  mono_thread_resume_interruption (exec=-1233124024) at threads.c:4526
> #9  0x00000000 in ?? ()
> 
> Thread 1 (Thread 0xb6fb1000 (LWP 31081)):
> #0  0xb6ee3544 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #1  0xb6edf144 in pthread_cond_wait@@GLIBC_2.4 () from /lib/arm-linux-gnueabihf/libpthread.so.0
> #2  0x00198382 in mono_os_cond_wait (mutex=0x29b99c <signal_mutex>, cond=0xbeb4c320) at mono-os-mutex.h:173
> #3  mono_os_event_wait_multiple (events=events@entry=0xbeb4c384, nevents=nevents@entry=1, waitall=waitall@entry=1, timeout=timeout@entry=4294967295, alertable=1) at os-event-unix.c:190
> #4  0x00198650 in mono_os_event_wait_one (event=event@entry=0xb46007f8, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at os-event-unix.c:94
> #5  0x001a5856 in mono_thread_info_wait_one_handle (thread_handle=thread_handle@entry=0xb46007f0, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at mono-threads.c:1632
> #6  0x0012db52 in mono_join_uninterrupted (error=0xbeb4c3ac, ms=<optimized out>, thread_to_join=0xb46007f0) at threads.c:1777
> #7  ves_icall_System_Threading_Thread_Join_internal (this_obj=<optimized out>, ms=<optimized out>) at threads.c:1834
> #8  0xb4da03d4 in wrapper_managed_to_native_System_Threading_Thread_JoinInternal_System_Threading_Thread_int () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-armhf/mcs/class/lib/testing_aot_full/mscorlib.dll.so
> #9  0xb4da0454 in System_Threading_Thread_Join () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-armhf/mcs/class/lib/testing_aot_full/mscorlib.dll.so
> #10 0xb4ac789c in Driver_Main () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-armhf/mono/tests/block_guard_restore_aligment_on_exit.exe.so
> #11 0xb50e4fc0 in wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-armhf/mcs/class/lib/testing_aot_full/mscorlib.dll.so
> #12 0x0002d42c in mono_jit_runtime_invoke (method=<optimized out>, obj=0xbeb4c6bc, params=<optimized out>, exc=<optimized out>, error=0xbeb4c98c) at mini-runtime.c:2769
> #13 0x0014db10 in do_runtime_invoke (method=0x31f748, obj=0x0, params=0xbeb4c96c, exc=0x0, error=0xbeb4c98c) at object.c:2828
> #14 0x0015016a in mono_runtime_invoke_checked (method=0x31f748, obj=0x0, params=0xbeb4c96c, error=0xbeb4c98c) at object.c:2982
> #15 0x001501fa in do_exec_main_checked (method=0x31f748, args=<optimized out>, error=0xbeb4c98c) at object.c:4699
> #16 0x000687a2 in mono_jit_exec (domain=domain@entry=0x2a6f78, assembly=assembly@entry=0x31fd78, argc=argc@entry=1, argv=argv@entry=0xbeb4ccb8) at driver.c:1028
> #17 0x000699ce in main_thread_handler (user_data=<synthetic pointer>) at driver.c:1097
> #18 mono_main (argc=<optimized out>, argv=<optimized out>) at driver.c:2179
> #19 0x00029992 in mono_main_with_options (argv=0xbeb4cca4, argc=6) at main.c:46
> #20 main (argc=6, argv=0xbeb4cca4) at main.c:339
> 
> =================================================================
> Got a SIGSEGV while executing native code. This usually indicates
> a fatal error in the mono runtime or one of the native libraries 
> used by your application.
> =================================================================
Comment 1 Zoltan Varga 2017-07-06 19:11:48 UTC
Should be fixed by mono master 8aaf80d0112c6bcac934670cb2cac7ee1cbf26a0/75d5e92311d0b4a3630dcb78a41d7f5aeab009f0.
Comment 2 Alexander Köplinger [MSFT] 2017-07-06 23:48:48 UTC
Thanks! It indeed fixes the issue on armel and armhf, but arm64 is still failing the test.

Here's the new stacktrace from https://jenkins.mono-project.com/job/test-mono-mainline-fullaot/label=debian-8-arm64/363/:

>                                                 EEE
> What now?
> in the finally block
> [New LWP 14906]
> [New LWP 14905]
> [New LWP 14904]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
> 0x0000007fa1db23b4 in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
>   Id   Target Id         Frame 
>   4    Thread 0x7fa13ff1c0 (LWP 14904) "SGen worker" 0x0000007fa1db23b4 in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
>   3    Thread 0x7f9f2671c0 (LWP 14905) "Finalizer" 0x0000007fa1db461c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
>   2    Thread 0x7f9f0671c0 (LWP 14906) "mono" mono_arm_handler_block_trampoline_helper (ptr=<optimized out>) at tramp-arm64.c:105
> * 1    Thread 0x7fa1eba000 (LWP 14884) "mono" 0x0000007fa1db23b4 in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
> 
> Thread 4 (Thread 0x7fa13ff1c0 (LWP 14904)):
> $1 = void
> 
> Thread 3 (Thread 0x7f9f2671c0 (LWP 14905)):
> 
> "Finalizer" tid=0x0x7f9f2671c0 this=0x0x7f9f2d8278 , thread handle : 0x7f980010e0, state : not waiting
> 
> $2 = void
> 
> Thread 2 (Thread 0x7f9f0671c0 (LWP 14906)):
> 
> "<unnamed thread>" tid=0x0x7f9f0671c0 this=0x0x7f9f2d83c0 , thread handle : 0x7f900010e0, state : not waiting
> 
> $3 = void
> 
> Thread 1 (Thread 0x7fa1eba000 (LWP 14884)):
> 
> "<unnamed thread>" tid=0x0x7fa1eba000 this=0x0x7f9f2d8130 , thread handle : 0x753e90, state : waiting
>   at <unknown> <0xffffffff>
>   at (wrapper managed-to-native) System.Threading.Thread.JoinInternal (System.Threading.Thread,int) <0x00007>
>   at System.Threading.Thread.Join () [0x00000] in <c50bb8d21057420a8b6dfa5673ed48c9#11B42CDE-AC47-A193-AAD0-596634B252A5>:0
>   at Driver.Main () [0x0003c] in <93ea97e0ce48499e816408844cffe619#11B42CDE-AC47-A193-AAD0-596634B252A5>:0
>   at (wrapper runtime-invoke) object.runtime_invoke_dynamic (intptr,intptr,intptr,intptr) [0x0001e] in <c50bb8d21057420a8b6dfa5673ed48c9#11B42CDE-AC47-A193-AAD0-596634B252A5>:0
> 
> $4 = void
> 
> Thread 4 (Thread 0x7fa13ff1c0 (LWP 14904)):
> #0  0x0000007fa1db23b4 in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x000000000060c7e4 in mono_os_cond_wait (mutex=0x750f60 <lock>, cond=0x750f90 <work_cond>) at ../../mono/utils/mono-os-mutex.h:173
> #2  get_work (job=<optimized out>, do_idle=<optimized out>, work_context=<optimized out>, worker_index=<optimized out>) at sgen-thread-pool.c:164
> #3  thread_func (worker_index=0) at sgen-thread-pool.c:193
> #4  0x0000007fa1dade34 in start_thread () from /lib/aarch64-linux-gnu/libpthread.so.0
> #5  0x0000007fa1d244f0 in clone () from /lib/aarch64-linux-gnu/libc.so.6
> 
> Thread 3 (Thread 0x7f9f2671c0 (LWP 14905)):
> #0  0x0000007fa1db461c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x0000007fa1db46cc in sem_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
> #2  0x000000000059d01c in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x7420e8 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:210
> #3  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x7420e8 <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:43
> #4  finalizer_thread (unused=<optimized out>) at gc.c:862
> #5  0x0000000000576658 in start_wrapper_internal (stack_ptr=<optimized out>, start_info=0x0) at threads.c:839
> #6  start_wrapper (data=0x7b01c0) at threads.c:901
> #7  0x0000007fa1dade34 in start_thread () from /lib/aarch64-linux-gnu/libpthread.so.0
> #8  0x0000007fa1d244f0 in clone () from /lib/aarch64-linux-gnu/libc.so.6
> 
> Thread 2 (Thread 0x7f9f0671c0 (LWP 14906)):
> #0  mono_arm_handler_block_trampoline_helper (ptr=<optimized out>) at tramp-arm64.c:105
> #1  0x00000000004d89e4 in mono_arch_setup_resume_sighandler_ctx (ctx=0x7f90001eb0, func=0x7f9f0678b0) at exceptions-arm64.c:613
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> 
> Thread 1 (Thread 0x7fa1eba000 (LWP 14884)):
> #0  0x0000007fa1db23b4 in pthread_cond_wait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
> #1  0x000000000060f604 in mono_os_cond_wait (mutex=0x7511c8 <signal_mutex>, cond=0x7ffe8d28a0) at mono-os-mutex.h:173
> #2  mono_os_event_wait_multiple (events=events@entry=0x7ffe8d28e8, nevents=nevents@entry=1, waitall=waitall@entry=1, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at os-event-unix.c:190
> #3  0x000000000060f7fc in mono_os_event_wait_one (event=event@entry=0x7f900010f0, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at os-event-unix.c:94
> #4  0x0000000000622540 in mono_thread_info_wait_one_handle (thread_handle=thread_handle@entry=0x7f900010e0, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at mono-threads.c:1632
> #5  0x00000000005789c8 in mono_join_uninterrupted (error=0x7ffe8d29b8, ms=<optimized out>, thread_to_join=0x7f900010e0) at threads.c:1777
> #6  ves_icall_System_Threading_Thread_Join_internal (this_obj=<optimized out>, ms=<optimized out>) at threads.c:1834
> #7  0x0000007f9f55076c in wrapper_managed_to_native_System_Threading_Thread_JoinInternal_System_Threading_Thread_int () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-arm64/mcs/class/lib/testing_aot_full/mscorlib.dll.so
> #8  0x0000007f9f5507ec in System_Threading_Thread_Join () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-arm64/mcs/class/lib/testing_aot_full/mscorlib.dll.so
> #9  0x0000007fa1806760 in Driver_Main () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-arm64/mono/tests/block_guard_restore_aligment_on_exit.exe.so
> #10 0x0000007f9f832da4 in wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr () from /media/ssd/jenkins/workspace/test-mono-mainline-fullaot/label/debian-8-arm64/mcs/class/lib/testing_aot_full/mscorlib.dll.so
> #11 0x0000000000426b30 in mono_jit_runtime_invoke (method=<optimized out>, obj=0x0, params=0x7ffe8d2f98, exc=<optimized out>, error=0x7ffe8d2fd8) at mini-runtime.c:2770
> #12 0x00000000005a58b0 in do_runtime_invoke (method=0x7bc218, obj=0x46af98 <mono_main+4384>, params=0x7ffe8d3040, exc=0x7fa14003e8, error=0x7ffe8d2fd8) at object.c:2828
> #13 0x00000000005a9148 in do_exec_main_checked (method=0x7bc218, args=<optimized out>, error=0x7ffe8d2fd8) at object.c:4699
> #14 0x000000000046989c in mono_jit_exec (domain=domain@entry=0x761800, assembly=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffe8d3450) at driver.c:1028
> #15 0x000000000046af98 in main_thread_handler (user_data=<synthetic pointer>) at driver.c:1097
> #16 mono_main (argc=<optimized out>, argv=<optimized out>) at driver.c:2179
> #17 0x00000000004218bc in mono_main_with_options (argv=0x7ffe8d3428, argc=6) at main.c:46
> #18 main (argc=0, argv=0x2f2f2f2f2f2f2f2f) at main.c:339
Comment 3 Alexander Köplinger [MSFT] 2017-07-07 12:45:12 UTC
https://github.com/mono/mono/commit/fd5843e964bfaf313e8e7f8672bf2ebe24e9315c fixes the issue on ARM64, thanks!