Bug 58186 - Mono 4.8.0 fails to build on Ubuntu (s390x) with SIGSEGV
Summary: Mono 4.8.0 fails to build on Ubuntu (s390x) with SIGSEGV
Status: RESOLVED ANSWERED
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: 4.8.0 (C9)
Hardware: Other Linux
: --- normal
Target Milestone: Community
Assignee: Neale Ferguson
URL:
Depends on:
Blocks:
 
Reported: 2017-07-17 12:12 UTC by Vaibhav Vaingankar
Modified: 2017-08-30 09:01 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
stacktrace for the error (11.68 KB, text/plain)
2017-07-17 12:12 UTC, Vaibhav Vaingankar
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED ANSWERED

Description Vaibhav Vaingankar 2017-07-17 12:12:14 UTC
Created attachment 23594 [details]
stacktrace for the error

Mono 4.8.0.524 fails to build on Ubuntu distros for s390x architecture
please see the attached build logs
Comment 1 Neale Ferguson 2017-07-20 15:15:01 UTC
I am not sure what the environmental differences between my environment and the OPs but I get a clean build on my Ubuntu system:

Linux linux02 3.16.0-4-s390x #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) s390x GNU/Linux
gcc version 5.3.1 20160101 (Debian 5.3.1-5) 
glibc 2.24-12 
Mono JIT compiler version 4.8.1 (mono-4.8.0-branch/22a39d7 Fri Jul 14 15:44:05 EDT 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       normal
	Notifications: epoll
	Architecture:  s390x
	Disabled:      none
	Misc:          softdebug 
	GC:            sgen (concurrent by default)

I suggest invoking the failing mono command with gdb and determining exactly where it is crashing.
Comment 2 Vaibhav Vaingankar 2017-07-21 06:32:59 UTC
Debug info from gdb:

[New LWP 52009]
[New LWP 52010]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".
0x000003fffcfe1046 in __waitpid (pid=<optimized out>, stat_loc=0x3ffffcd751c, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29      ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
  Id   Target Id         Frame
* 1    Thread 0x3fffce3e740 (LWP 45492) "mono" 0x000003fffcfe1046 in __waitpid (pid=<optimized out>, stat_loc=0x3ffffcd751c, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:29
  2    Thread 0x3fffc7ff910 (LWP 45514) "SGen worker" 0x000003fffcfdc7d4 in __pthread_cond_wait (cond=0x2aa00448508 <work_cond>, mutex=0x2aa00448538 <lock>) at pthread_cond_wait.c:186
  3    Thread 0x3fffac5a910 (LWP 45515) "Finalizer" 0x000003fffcfdef92 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x2aa00439484 <finalizer_sem+4>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205

Thread 3 (Thread 0x3fffac5a910 (LWP 45515)):
#0  0x000003fffcfdef92 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x2aa00439484 <finalizer_sem+4>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0x2aa00439480 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x000003fffcfdf07a in __new_sem_wait_slow (sem=0x2aa00439480 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x000002aa00243ebe in mono_os_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x2aa00439480 <finalizer_sem>) at ../../mono/utils/mono-os-semaphore.h:209
#4  mono_coop_sem_wait (flags=MONO_SEM_FLAGS_ALERTABLE, sem=0x2aa00439480 <finalizer_sem>) at ../../mono/utils/mono-coop-semaphore.h:40
#5  finalizer_thread (unused=<optimized out>) at gc.c:905
#6  0x000002aa0021b2ac in start_wrapper_internal (stack_ptr=0x3fffac59e90, start_info=0x0) at threads.c:760
#7  start_wrapper (data=<optimized out>) at threads.c:824
#8  0x000002aa002f343e in inner_start_thread (data=0x2aa171eb5d0) at mono-threads.c:1181
#9  0x000003fffcfd6934 in start_thread (arg=0x3fffac5a910) at pthread_create.c:335
#10 0x000003fffcf2dd1a in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone.S:74

Thread 2 (Thread 0x3fffc7ff910 (LWP 45514)):
#0  0x000003fffcfdc7d4 in __pthread_cond_wait (cond=0x2aa00448508 <work_cond>, mutex=0x2aa00448538 <lock>) at pthread_cond_wait.c:186
#1  0x000002aa002c66c0 in mono_os_cond_wait (mutex=0x2aa00448538 <lock>, cond=0x2aa00448508 <work_cond>) at ../../mono/utils/mono-os-mutex.h:150
#2  thread_func (thread_data=0x0) at sgen-thread-pool.c:110
#3  0x000003fffcfd6934 in start_thread (arg=0x3fffc7ff910) at pthread_create.c:335
#4  0x000003fffcf2dd1a in thread_start () at ../sysdeps/unix/sysv/linux/s390/s390-64/clone.S:74

Thread 1 (Thread 0x3fffce3e740 (LWP 45492)):
#0  0x000003fffcfe1046 in __waitpid (pid=<optimized out>, stat_loc=0x3ffffcd751c, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x000002aa0011cc14 in mono_handle_native_sigsegv (signal=<optimized out>, ctx=<optimized out>, info=<optimized out>) at mini-exceptions.c:2469
#2  0x000002aa000832da in mono_sigsegv_signal_handler (_dummy=<optimized out>, _info=0x3ffffcd7f70, context=0x3ffffcd7ff0) at mini-runtime.c:2904
#3  <signal handler called>
#4  0x000002aa00307410 in monoeg_g_slist_free (list=0x3f3851ec00000000) at gslist.c:121
#5  0x000002aa0009af9c in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=<optimized out>, end_bblock=0x2aa176335c0, return_var=<optimized out>, inline_args=0x0, inline_offset=0, is_virtual_call=0) at method-to-ir.c:8629
#6  0x000002aa001693da in mini_method_compile (method=method@entry=0x2aa17626838, opts=opts@entry=370223487, domain=domain@entry=0x2aa1717e080, flags=flags@entry=JIT_FLAG_RUN_CCTORS, parts=parts@entry=0, aot_method_index=-1) at mini.c:3561
#7  0x000002aa0016b742 in mono_jit_compile_method_inner (method=0x2aa17626838, target_domain=0x2aa1717e080, opt=<optimized out>, error=0x3ffffcd8b40) at mini.c:4258
#8  0x000002aa00081ff0 in mono_jit_compile_method_with_opt (method=0x2aa17626838, opt=370223487, error=0x3ffffcd8b40) at mini-runtime.c:1934
#9  0x000002aa0011e7da in common_call_trampoline (regs=regs@entry=0x3ffffcd8cb0, code=code@entry=0x3fffa07256e "\271\004", m=m@entry=0x2aa17626838, vt=vt@entry=0x0, vtable_slot=<optimized out>, vtable_slot@entry=0x0, error=0x3ffffcd8b40) at mini-trampolines.c:699
#10 0x000002aa0011ef1c in mono_magic_trampoline (regs=0x3ffffcd8cb0, code=0x3fffa07256e "\271\004", arg=0x2aa17626838, tramp=<optimized out>) at mini-trampolines.c:829
#11 0x000003fffce2e0e2 in ?? ()
#12 0x000003fffa07256e in ?? ()
#13 0x000003fffa06f096 in ?? ()
Comment 3 Neale Ferguson 2017-07-28 16:41:17 UTC
Would you:

(a) git log mono/mini/mini.c | head -10
(b) Show me lines 3550-3570 in mono/mini/mini.c

Neale
Comment 4 Vaibhav Vaingankar 2017-07-31 05:33:12 UTC
$ git log mono/mini/mini.c | head -10
commit ed0fb9dea072737e74a5c2302d14dd3ccd5cb23a
Author: Zoltan Varga <vargaz@gmail.com>
Date:   Mon Feb 13 14:35:41 2017 -0500

    [jit] Disable llvm for overly complex methods, to avoid compilation slowdowns. Fixes #52475.

commit 498a47eab324c39dfacb47a2533a88606760efc6
Author: Zoltan Varga <vargaz@gmail.com>
Date:   Wed Jan 11 12:12:32 2017 -0500

----------
$ vi mono/mini/mini.c

3547         if (mono_compile_is_broken (cfg, cfg->method, TRUE)) {
3548                 if (mini_get_debug_options ()->break_on_unverified)
3549                         G_BREAKPOINT ();
3550                 return cfg;
3551         }
3552
3553         /*
3554          * create MonoInst* which represents arguments and local variables
3555          */
3556         mono_compile_create_vars (cfg);
3557
3558         mono_cfg_dump_create_context (cfg);
3559         mono_cfg_dump_begin_group (cfg);
3560
3561         MONO_TIME_TRACK (mono_jit_stats.jit_method_to_ir, i = mono_method_to_ir (cfg, method_to_compile, NULL, NULL, NULL, NULL, 0, FALSE));
3562         mono_cfg_dump_ir (cfg, "method-to-ir");
3563
3564         if (cfg->gdump_ctx != NULL) {
3565                 /* workaround for graph visualization, as it doesn't handle empty basic blocks properly */
3566                 mono_insert_nop_in_empty_bb (cfg);
3567                 mono_cfg_dump_ir (cfg, "mono_insert_nop_in_empty_bb");
3568         }
3569
3570         if (i < 0) {
3571                 if (try_generic_shared && cfg->exception_type == MONO_EXCEPTION_GENERIC_SHARING_FAILED) {
3572                         if (compile_aot) {
3573                                 if (MONO_METHOD_COMPILE_END_ENABLED ())
Comment 5 Neale Ferguson 2017-07-31 14:52:19 UTC
Are you sure you are on the right branch? Here is what I see:

[nealef@linux02 - mono] git branch
* mono-4.8.0-branch

commit 22a39d7631b1a9ef899433bd28adb993a7bdb39c
Author: Alexis Christoforides <alexis@thenull.net>
Date:   Wed Apr 5 19:03:18 2017 -0400

    Bump version -> 4.8.1
Comment 6 Vaibhav Vaingankar 2017-08-01 04:11:54 UTC
Here is the branch I am on:

$ git branch
  master
* mono-4.8.0-branch
Comment 7 Neale Ferguson 2017-08-01 23:24:40 UTC
I started from a clean slate:

[nealef@linux02 - ~] git clone https://github.com/mono/mono.git                     
Cloning into 'mono'...
remote: Counting objects: 1131439, done.
Receiving objects:  15% (169716/1131439), 63.75 MiB | 15.93 MiB/s    
remote: Total 1131439 (delta 0), reused 1 (delta 0), pack-reused 1131438
Receiving objects: 100% (1131439/1131439), 442.91 MiB | 20.02 MiB/s, done.
Resolving deltas: 100% (918233/918233), done.
Checking connectivity... done.
Checking out files: 100% (54971/54971), done.
[nealef@linux02 - ~] cd mono
[nealef@linux02 - mono] git checkout mono-4.8.0-branch
Branch mono-4.8.0-branch set up to track remote branch mono-4.8.0-branch from origin.
Switched to a new branch 'mono-4.8.0-branch'
[nealef@linux02 - mono] git log
commit 22a39d7631b1a9ef899433bd28adb993a7bdb39c
Author: Alexis Christoforides <alexis@thenull.net>
Date:   Wed Apr 5 19:03:18 2017 -0400

    Bump version -> 4.8.1

:
:
[nealef@linux02 - mono] ./autogen.sh --with-libgc-threads=pthreads --enable-parallel-mark --with-tls=__thread --with-sgen-default-concurrent=yes --with-orbis=yes
Running libtoolize...
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
:
:
Submodule path 'external/rx': checked out 'b29a4b0fda609e0af33ff54ed13652b6ccf0e05e'
Git submodules updated successfully

        mcs source:    mcs

   Engine:
	Host:	       s390x-ibm-linux-gnu
	Target:	       s390x-ibm-linux-gnu
	GC:	       sgen (concurrent by default) and Included Boehm GC with typed GC and parallel mark
	TLS:           __thread
	SIGALTSTACK:   yes
	Engine:        Building and using the JIT
	BigArrays:     no
	DTrace:        no
	LLVM Back End: no (dynamically loaded: no)

   Libraries:
	.NET 4.x:        yes
	Xamarin.Android: no
	Xamarin.iOS:     no
	Xamarin.WatchOS: no
	Xamarin.TVOS:    no
	Xamarin.Mac:     no
	mobile_static:   no
	JNI support:     IKVM Native
	libgdiplus:      assumed to be installed
	zlib:            system zlib
	BTLS:            no
	

Now type `make' to compile
[nealef@linux02 - mono] make get-monolite-latest
rm -fr /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite-*
mkdir -p /mnt/DREAMStorage/nealef/mono/mcs/class/lib
test ! -d /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite || test ! -d /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite.old || rm -fr /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite.old
test ! -d /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite || mv -f /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite /mnt/DREAMStorage/nealef/mono/mcs/class/lib/monolite.old
cd /mnt/DREAMStorage/nealef/mono/mcs/class/lib && { (wget -O- http://download.mono-project.com/monolite/monolite-156-latest.tar.gz || curl http://download.mono-project.com/monolite/monolite-156-latest.tar.gz) | gzip -d | tar xf - ; }
--2017-08-01 18:58:19--  http://download.mono-project.com/monolite/monolite-156-latest.tar.gz
Resolving download.mono-project.com (download.mono-project.com)... 72.21.81.200, 2606:2800:11f:17a5:191a:18d5:537:22f9
Connecting to download.mono-project.com (download.mono-project.com)|72.21.81.200|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4569148 (4.4M) [application/x-gzip]
Saving to: 'STDOUT'

-                           100%[=============================================>]   4.36M  5.20MB/s   in 0.8s   

2017-08-01 18:58:20 (5.20 MB/s) - written to stdout [4569148/4569148]

cd /mnt/DREAMStorage/nealef/mono/mcs/class/lib && mv -f monolite-* monolite
[nealef@linux02 - mono] make
:
MDOC    [net_4_x] mono-file-formats.tree
make[3]: Leaving directory '/mnt/DREAMStorage/nealef/mono/docs'
make -f ./docs.make topdir=./../mcs srcdir=. mono-tools.tree
make[3]: Entering directory '/mnt/DREAMStorage/nealef/mono/docs'
MDOC    [net_4_x] mono-tools.tree
make[3]: Leaving directory '/mnt/DREAMStorage/nealef/mono/docs'
mkdir -p deploy
cp -f ./api-style.css deploy
make -f ./docs.make topdir=./../mcs srcdir=. monoapi.tree
make[3]: Entering directory '/mnt/DREAMStorage/nealef/mono/docs'
MDOC    [net_4_x] monoapi.tree
Warning: File `DoesNotExist' referenced in TOC but it doesn't exist. It will be ignored.
make[3]: Leaving directory '/mnt/DREAMStorage/nealef/mono/docs'
make[2]: Leaving directory '/mnt/DREAMStorage/nealef/mono/docs'
Making all in acceptance-tests
make[2]: Entering directory '/mnt/DREAMStorage/nealef/mono/acceptance-tests'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/mnt/DREAMStorage/nealef/mono/acceptance-tests'
Making all in llvm
make[2]: Entering directory '/mnt/DREAMStorage/nealef/mono/llvm'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/mnt/DREAMStorage/nealef/mono/llvm'
make[2]: Entering directory '/mnt/DREAMStorage/nealef/mono'
make[2]: Nothing to be done for 'all-am'.
make[2]: Leaving directory '/mnt/DREAMStorage/nealef/mono'
make[1]: Leaving directory '/mnt/DREAMStorage/nealef/mono'

However, I did get one crash during the build that wasn't the same as yours but when I issued make again it went to completion cleanly.
Comment 8 Neale Ferguson 2017-08-01 23:50:43 UTC
[nealef@linux02 - tests] make check | grep passed
394 test(s) passed, 0 test(s) did not pass.
212 test(s) passed, 0 test(s) did not pass.
1 test(s) passed, 0 test(s) did not pass.
1 test(s) passed, 0 test(s) did not pass.
6 test(s) passed, 0 test(s) did not pass.
6 test(s) passed, 0 test(s) did not pass.
4 test(s) passed, 0 test(s) did not pass.
Comment 9 Vaibhav Vaingankar 2017-08-02 05:52:51 UTC
I followed exact same steps as yours, yet I ended up with same crash.

apt-get install autoconf automake libtool  make git g++ wget curl
git clone https://github.com/mono/mono.git
cd mono/
git checkout mono-4.8.0-branch
./autogen.sh --with-libgc-threads=pthreads --enable-parallel-mark --with-tls=__thread --with-sgen-default-concurrent=yes --with-orbis=yes --disable-nls
make get-monolite-latest
make
Comment 10 Vaibhav Vaingankar 2017-08-08 12:26:58 UTC
I tried building on Debian 9, where build is clean. Issue is seen on Ubuntu systems, can you check if you can get clean build on Ubuntu system instead of Debian?

thanks,
vaibhav
Comment 11 Vaibhav Vaingankar 2017-08-30 09:00:34 UTC
The issue was with the gcc available in the repository. Had to build gcc from source. build is clean using newly built gcc.