Bug 17017 - ARMHF (ARM v7a): AOT'd mcs crashes for no apparent reason while compiling.
Summary: ARMHF (ARM v7a): AOT'd mcs crashes for no apparent reason while compiling.
Status: RESOLVED NORESPONSE
Alias: None
Product: Runtime
Classification: Mono
Component: JIT ()
Version: unspecified
Hardware: Other Linux
: --- normal
Target Milestone: ---
Assignee: Alex Rønne Petersen
URL:
Depends on:
Blocks:
 
Reported: 2014-01-02 04:27 UTC by Dylan Borg
Modified: 2015-10-28 06:01 UTC (History)
10 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
config.log for ./configure --with-tls=__thread (32.33 KB, application/x-tar-gz)
2014-01-04 03:09 UTC, Dylan Borg
Details
config.log - 2014-02-05 05:01:57 (549.16 KB, application/octet-stream)
2014-02-05 05:04 UTC, Horst Kautschitz
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NORESPONSE

Description Dylan Borg 2014-01-02 04:27:48 UTC
I tried compiling mono on my Panda ES (used git master). The VM parts compiled fine, so did the basic and build parts of the mcs directory. Then when the first compilation of mscorlib started using the just compiled and aot'd mcs make said an error had occurred:

MCS     [build] Mono.Posix.dll
MCS     [build] mcs.exe
AOT     [build] mscorlib.dll.so
AOT     [build] mcs.exe.so
MCS     [build] sn.exe
MCS     [net_2_0] mscorlib.dll
make[8]: *** [../../class/lib/net_2_0/tmp/mscorlib.dll] Error 1
make[7]: *** [do-all] Error 2
make[6]: *** [all-recursive] Error 1
make[5]: *** [all-recursive] Error 1
make[4]: *** [profile-do--net_2_0--all] Error 2
make[3]: *** [profiles-do--all] Error 2
make[2]: *** [all-local] Error 2
make[2]: Leaving directory `/mnt/hdd/mono/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/mnt/hdd/mono'
make: *** [all] Error 2

Compiling with "SKIP_AOT=true make" suppress the AOT step at the end of MCS compilation and allows the class libraries to be compiled normally. Installing the latter yielded a fully functional environment (tested console programs and asp.net hosting).
Comment 1 Alex Rønne Petersen 2014-01-02 17:07:44 UTC
Can you try running with MONO_LOG_LEVEL=debug?
Comment 2 Dylan Borg 2014-01-03 04:13:34 UTC
I get the following before the crash:
Mono: Assembly Loader loaded assembly from location: '/usr/lib/mono/4.5/System.dll'.
Mono: Config attempting to parse: '/usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll.config'.
Mono: Config attempting to parse: '/usr/etc/mono/assemblies/System/System.config'.
Mono: Assembly Ref addref System.Core[0x1db9730] -> System[0x1dbd028]: 2
Mono: Assembly Ref addref System[0x1dbd028] -> mscorlib[0x1d75c28]: 4
Mono: Assembly Ref addref mcs[0x1db05d8] -> System[0x1dbd028]: 3
Mono C# compiler version 3.2.7.0
Mono: GC_MAJOR: (user request) pause 1.83ms, total 1.89ms, bridge 0.00ms major 544K/304K los 0K/0K
Mono: Unloading domain mcs.exe[0x1d71c18], assembly mscorlib[0x1d75c28], ref_count=4
Mono: Unloading domain mcs.exe[0x1d71c18], assembly mcs[0x1db05d8], ref_count=1
Mono: Unloading assembly mcs [0x1db05d8].
Mono: Unloading domain mcs.exe[0x1d71c18], assembly System.Core[0x1db9730], ref_count=2
Mono: Unloading domain mcs.exe[0x1d71c18], assembly System[0x1dbd028], ref_count=3
Mono: Unloading image /usr/lib/mono/4.5/mcs.exe [0x1d74698].
Mono: Unloading assembly System.Core [0x1db9730].
Mono: Unloading image /usr/lib/mono/gac/System.Core/4.0.0.0__b77a5c561934e089/System.Core.dll [0x1db8d80].
Mono: Unloading assembly System [0x1dbd028].
Mono: Unloading image /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll [0x1dbc658].
Mono: Unloading assembly mscorlib [0x1d75c28].
Mono: Unloading image /usr/lib/mono/4.5/mscorlib.dll [0x1d752f8].
MCS     [net_2_0] mscorlib.dll
Mono: Assembly Loader probing location: './../../class/lib/build/mscorlib.dll'.
Mono: Image addref mscorlib[0x1023bd8] -> /mnt/hdd/mono/mcs/class/lib/build/mscorlib.dll[0x1022f48]: 2
make[7]: *** [../../class/lib/net_2_0/tmp/mscorlib.dll] Error 1
make[6]: *** [do-all] Error 2
make[5]: *** [all-recursive] Error 1
make[4]: *** [all-recursive] Error 1
make[3]: *** [profile-do--net_2_0--all] Error 2
make[2]: *** [profiles-do--all] Error 2
make[1]: *** [all-local] Error 2
make[1]: Leaving directory `/mnt/hdd/mono/runtime'
make: *** [dir-check] Error 2
Comment 3 Alex Rønne Petersen 2014-01-03 05:33:21 UTC
Nothing particularly useful there, unfortunately.

A few things:

1. How did you `configure` the runtime?
2. What ARM CPU does your board have?
3. What's the output of `uname -a` and `LD_SHOW_AUXV=1 /bin/true`?
4. Does `dmesg` show anything about the Mono process that errors out?
Comment 4 Dylan Borg 2014-01-03 06:46:56 UTC
1. ./configure --with-tls=__thread
2. TI OMAP 4460 (ARM v7A MPCore  2x1.2GHz)
3. Linux pandaboard 3.4.0-2-linaro-llt-omap #2~ci+121213205950-Ubuntu SMP PREEMPT Fri Dec 14 06:23:34 UTC 20 armv7l armv7l armv7l GNU/Linux

AT_HWCAP:    swp half thumb fastmult vfp edsp thumbee neon vfpv3
AT_PAGESZ:       4096
AT_CLKTCK:       100
AT_PHDR:         0x8034
AT_PHENT:        32
AT_PHNUM:        9
AT_BASE:         0xb6f3a000
AT_FLAGS:        0x0
AT_ENTRY:        0x8d11
AT_UID:          1001
AT_EUID:         1001
AT_GID:          1002
AT_EGID:         1002
AT_SECURE:       0
AT_RANDOM:       0xbe8047d9
AT_EXECFN:       /bin/true
AT_PLATFORM:     v7l

4. apparently nothing
Comment 5 Alex Rønne Petersen 2014-01-03 07:22:19 UTC
Nothing suspicious in that output either. Can you try to strace the Mono process and see if anything interesting pops up? Alternatively, run in GDB and break on exit, _Exit, abort, etc and see where they get called from.
Comment 6 Dylan Borg 2014-01-03 12:48:58 UTC
Does the AOT compiler assume double precision FP is available in hardware? If I am not mistaken my board only has half and single precision in hardware.
Comment 7 Dylan Borg 2014-01-03 12:57:34 UTC
After testing using the csharp REPL, I have found out that trying to remove whole numbers out of double.MaxValue/float.MaxValue does not reduce the amount. So the runtime is not preserving the right semantics.
Comment 8 Dylan Borg 2014-01-03 13:01:33 UTC
Further investigation reveals that division works for large floats but not addition/subtraction of small numbers.
Comment 9 Alex Rønne Petersen 2014-01-03 21:34:23 UTC
> Does the AOT compiler assume double precision FP is available in hardware? If I
am not mistaken my board only has half and single precision in hardware.

Yes, we assume there is both single-precision and double-precision support if a VFP unit is available, but that has nothing to do with AOT. We produce the same code. Not having double-precision support in VFP sounds quite... odd. The AAPCS explicitly mentions d0-d15 in the ABI, not even conditionally.

> After testing using the csharp REPL, I have found out that trying to remove
whole numbers out of double.MaxValue/float.MaxValue does not reduce the amount.
So the runtime is not preserving the right semantics.

This behavior is what you'll see on x86 too.

> Further investigation reveals that division works for large floats but not
addition/subtraction of small numbers.

Can you post an example REPL session?
Comment 10 Alex Rønne Petersen 2014-01-03 21:34:43 UTC
By the way, can you attach the full `configure` log?
Comment 11 Dylan Borg 2014-01-04 03:09:40 UTC
Created attachment 5757 [details]
config.log for ./configure --with-tls=__thread
Comment 12 Alex Rønne Petersen 2014-01-04 05:19:29 UTC
Nothing unusual there. So, you'll have to GDB it to figure out what's going wrong.
Comment 13 Brandon White 2014-01-08 18:44:27 UTC
I'm running into a very similar issue on a Technologic Systems board:

root@ts4700:~/mono# uname -a
Linux ts4700 2.6.34-ts471x #88 PREEMPT Thu Aug 15 10:29:09 MST 2013 armv5tejl GNU/Linux



Making all in runtime
make[2]: Entering directory `/root/mono/runtime'
if test -w /root/mono/mcs; then :; else chmod -R +w /root/mono/mcs; fi
cd /root/mono/mcs && make --no-print-directory -s NO_DIR_CHECK=1 PROFILES='net_2_0 net_3_5 net_4_0 net_4_5  ' CC='gcc' all-profiles
Bootstrap compiler: Mono C# compiler version 3.0.6.0
MCS     [build] mscorlib.dll
System.Collections/Queue.cs(28,0): error CS1525: Unexpected symbol `E'
System.Collections/Queue.cs(42,15): error CS1040: Preprocessor directives must appear as the first non-whitespace character on a line
System.Collections/Queue.cs(44,7): error CS1040: Preprocessor directives must appear as the first non-whitespace character on a line
System.Collections/Queue.cs(46,9): error CS1040: Preprocessor directives must appear as the first non-whitespace character on a line
System.Collections/Queue.cs(150,53): warning CS0078: The `l' suffix is easily confused with the digit `1' (use `L' for clarity)
System.Collections/Queue.cs(386,1): error CS1035: End-of-file found, '*/' expected
Compilation failed: 5 error(s), 1 warnings
make[8]: *** [../../class/lib/build/tmp/mscorlib.dll] Error 1
Comment 14 Alex Rønne Petersen 2014-01-09 05:54:40 UTC
I'm not sure that's related. Are you using a monolite compiler or a system compiler provided by some distro package? If the former, open a separate issue; if the latter, just update the system compiler or remove it in favor of monolite (`make get-monolite-latest`).
Comment 15 Brandon White 2014-01-09 09:36:52 UTC
Thanks for the response.  I've actually tried it both ways.  In the dump in my previous comment, that is using the Debian package.  I'm using Debian Wheezy but installed mono-mcs from the 'testing' repo.

When this didn't work for me, I removed the mono-mcs package and performed the 'make get-monolite-latest'.  I got the same result.  I can indeed confirm that the version was more recent.
Comment 16 Alex Rønne Petersen 2014-01-09 10:34:50 UTC
OK. But I think that this is a separate issue, as that board seems to be ARM v5 while this issue is about ARM v7. So please open a separate issue and assign/CC me.
Comment 17 Brandon White 2014-01-09 10:38:03 UTC
Will do.  I started a fresh build from a fresh master pull this morning.  When it concludes, I will open a new issue with the details.
Comment 18 Horst Kautschitz 2014-02-05 05:01:57 UTC
Same here with latest checkout on linaro-ubuntu linux

--- INFO ---
uname -a:
Linux linaro-ubuntu-desktop 3.0.68-smx6_03 #1 SMP Fri Apr 5 15:40:28 CEST 2013 armv7l armv7l armv7l GNU/Linux

--
LD_SHOW_AUXV=1 /bin/true:
AT_HWCAP:    swp half thumb fastmult vfp edsp neon vfpv3
AT_PAGESZ:       4096
AT_CLKTCK:       100
AT_PHDR:         0x8034
AT_PHENT:        32
AT_PHNUM:        9
AT_BASE:         0x2aaf6000
AT_FLAGS:        0x0
AT_ENTRY:        0x8d11
AT_UID:          0
AT_EUID:         0
AT_GID:          0
AT_EGID:         0
AT_SECURE:       0
AT_RANDOM:       0x7e89d8b4
AT_EXECFN:       /bin/true
AT_PLATFORM:     v7l

---
mono checked out 27836846a5b0c0e4dd4a7605f094cbc330e733d6
Submodule path 'external/Lucene.Net': checked out '88fb67b07621dfed054d8d75fd50672fb26349df'
Submodule path 'external/Newtonsoft.Json': checked out '471c3e0803a9f40a0acc8aeceb31de6ff93a52c4'
Submodule path 'external/aspnetwebstack': checked out 'e77b12e6cc5ed260a98447f609e887337e44e299'
Submodule path 'external/cecil': checked out '89cfb1d4079002394010fd351628ad057c94422f'
Submodule path 'external/debian-snapshot': checked out '9342f8f052f81deaba789f030db23a88b4369724'
Submodule path 'external/entityframework': checked out 'a5faddeca2bee08636f1b7b3af8389bd4119f4cd'
Submodule path 'external/ikvm': checked out '92df59f033ef81a9b5a9cc6b09d44ed18d244517'
Submodule path 'external/rx': checked out '00c1aadf149334c694d2a5096983a84cf46221b8'
Git submodules updated successfully


./autogen.sh --prefix=/opt/evo/opt/mono/1.3.7-beta

   mcs source:    mcs

   Engine:
        GC:            sgen and bundled Boehm GC with typed GC and parallel mark
        TLS:           __thread
        SIGALTSTACK:   yes
        Engine:        Building and using the JIT
        oprofile:      no
        BigArrays:     no
        DTrace:        no
        LLVM Back End: no (dynamically loaded: no)

   Libraries:
        .NET 2.0/3.5:  yes
        .NET 4.0:      yes
        .NET 4.5:      yes
        MonoDroid:     no
        MonoTouch:     no
        Xamarin.Mac:   no
        JNI support:   IKVM Native
        libgdiplus:    assumed to be installed
        zlib:          system zlib


--->>>
--->>> FIRST CASE: COMPILING WITHOUT MONO INSTALLED ---
--->>>


make get-monolite-latest
make EXTERNAL_MCS="${PWD}/mcs/class/lib/monolite/gmcs.exe"


--->>>
--->>> SECOND CASE: COMPILING WITH MONO INSTALLED ---
--->>>

apt-get install mono-mcs mono-gmcs
... autogen ... (as above)
make


---
Output for FIRST and SECOND:

MCS     [build] mcs.exe
AOT     [build] mscorlib.dll.so
AOT     [build] mcs.exe.so
MCS     [build] gacutil.exe
MCS     [build] sn.exe
MCS     [build] culevel.exe
CompileUplevel.cs(943,34): warning CS0618: `System.CodeDom.Compiler.CodeDomProvider.CreateGenerator()' is obsolete: `ICodeGenerator is obsolete'
Compilation succeeded - 1 warning(s)
mkdir -p -- ../../class/lib/net_2_0/tmp/
MCS     [net_2_0] mscorlib.dll
make[8]: *** [../../class/lib/net_2_0/tmp/mscorlib.dll] Error 1
make[7]: *** [do-all] Error 2
make[6]: *** [all-recursive] Error 1
make[5]: *** [all-recursive] Error 1
make[4]: *** [profile-do--net_2_0--all] Error 2
make[3]: *** [profiles-do--all] Error 2
make[2]: *** [all-local] Error 2
make[2]: Leaving directory `/media/sd/usr/src/mono/3/mono-git/mono-2/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/media/sd/usr/src/mono/3/mono-git/mono-2'
make: *** [all] Error 2

---
Output with make SKIP_AOT=true without make clean:

if test -w /media/sd/usr/src/mono/3/mono-git/mono-2/mcs; then :; else chmod -R +w /media/sd/usr/src/mono/3/mono-git/mono-2/mcs; fi
cd /media/sd/usr/src/mono/3/mono-git/mono-2/mcs && make --no-print-directory -s NO_DIR_CHECK=1 PROFILES='net_2_0 net_3_5 net_4_0 net_4_5 xbuild_12   ' CC='gcc' all-profiles
Bootstrap compiler: Mono C# compiler version 2.10.8.1
MCS     [net_2_0] mscorlib.dll
make[8]: *** [../../class/lib/net_2_0/tmp/mscorlib.dll] Error 1
make[7]: *** [do-all] Error 2
make[6]: *** [all-recursive] Error 1
make[5]: *** [all-recursive] Error 1
make[4]: *** [profile-do--net_2_0--all] Error 2
make[3]: *** [profiles-do--all] Error 2
make[2]: *** [all-local] Error 2
make[2]: Leaving directory `/media/sd/usr/src/mono/3/mono-git/mono-2/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/media/sd/usr/src/mono/3/mono-git/mono-2'
make: *** [all] Error 2


--->>>
--->>> THIRD CASE: C'OMPILING WITH MONO INSTALLED ---
--->>>

apt-get install mono-mcs mono-gmcs
./autogen.sh --prefix=/opt/evo/opt/mono --disable-nls --with-profile2=no --with-moonlight=no --enable-minimal=aot
make

---
Output:
Compilation succeeded - 5 warning(s)
MCS     [build] Mono.Posix.dll
MCS     [build] mcs.exe
AOT     [build] mscorlib.dll.so
AOT     [build] mcs.exe.so
MCS     [build] sn.exe
mkdir -p -- ../../class/lib/net_4_0/tmp/
MCS     [net_4_0] mscorlib.dll
make[8]: *** [../../class/lib/net_4_0/tmp/mscorlib.dll] Error 1
make[7]: *** [do-all] Error 2
make[6]: *** [all-recursive] Error 1
make[5]: *** [all-recursive] Error 1
make[4]: *** [profile-do--net_4_0--all] Error 2
make[3]: *** [profiles-do--all] Error 2
make[2]: *** [all-local] Error 2
make[2]: Leaving directory `/media/sd/usr/src/mono/3/mono-git/mono-2/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/media/sd/usr/src/mono/3/mono-git/mono-2'
make: *** [all] Error 2


Also trying to do with apt-get install mono-complete - same issue

Add config.log
Comment 19 Horst Kautschitz 2014-02-05 05:04:44 UTC
Created attachment 5977 [details]
config.log - 2014-02-05 05:01:57
Comment 20 Andres G. Aragoneses 2014-02-11 04:57:31 UTC
Horst, can you try with `make V=1`? It will give more detailed output.
Comment 21 Horst Kautschitz 2014-02-19 05:45:35 UTC
Switched now over to debian wheezy armhf distribution for testing. Used hardware is udoo (http://www.udoo.org) platform. Build works out of the box.
Comment 22 Zoltan Varga 2014-02-28 15:03:32 UTC
I can't reproduce this using mono master under debian wheezy.
Comment 23 Wolfgang Thomas 2014-03-25 10:45:34 UTC
I am facing the same issue on a Armada XP (ARM v7-A):
Linux ubuntu 3.2.40 #1 SMP Wed May 8 12:49:29 IDT 2013 armv7l armv7l armv7l GNU/Linux

make all-local
make[8]: Entering directory `/root/mono/mcs/class/corlib'
MONO_PATH="./../../class/lib/build:$MONO_PATH" /root/mono/runtime/mono-wrapper  ./../../class/lib/build/mcs.exe /codepage:65001 -unsafe -nostdlib -nowarn:612,618 -d:INSIDE_CORLIB -d:LIBC  -d:NET_1_1 -d:NET_2_0 -nowarn:1699 -nostdlib -lib:./../../class/lib/net_2_0  -debug -optimize  /noconfig -resource:resources/collation.core.bin -resource:resources/collation.tailoring.bin -resource:resources/collation.cjkCHS.bin -resource:resources/collation.cjkCHT.bin -resource:resources/collation.cjkJA.bin -resource:resources/collation.cjkKO.bin -resource:resources/collation.cjkKOlv2.bin --runtime:v2 -target:library -out:../../class/lib/net_2_0/tmp/mscorlib.dll  @corlib.dll.sources
make[8]: *** [../../class/lib/net_2_0/tmp/mscorlib.dll] Error 1
make[8]: Leaving directory `/root/mono/mcs/class/corlib'
make[7]: *** [do-all] Error 2
make[7]: Leaving directory `/root/mono/mcs/class/corlib'
make[6]: *** [all-recursive] Error 1
make[6]: Leaving directory `/root/mono/mcs/class'
make[5]: *** [all-recursive] Error 1
make[5]: Leaving directory `/root/mono/mcs'
make[4]: *** [profile-do--net_2_0--all] Error 2
make[4]: Leaving directory `/root/mono/mcs'
make[3]: *** [profiles-do--all] Error 2
make[3]: Leaving directory `/root/mono/mcs'
make[2]: *** [all-local] Error 2
make[2]: Leaving directory `/root/mono/runtime'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/mono'
make: *** [all] Error 2


If I can help by providing more traces please let me know.
Comment 24 Wolfgang Thomas 2014-04-11 19:20:10 UTC
Same problem on an Armada XP development boards (also an ARM v7A).
The compilation succeeds with  "make SKIP_AOT=true".
Comment 25 Wolfgang Thomas 2014-04-11 19:22:05 UTC
Sorry, I named the wrong board in comment 24. I wanted to refer to a Xilinx Zynq evaluation board, where I experienced the same problem.
Comment 26 Andres G. Aragoneses 2014-04-12 08:14:50 UTC
In reply to comment#24:
> The compilation succeeds with "make SKIP_AOT=true"

If the compilation succeeds with this, then I'm wondering if the distro that Horst mentioned in comment#21 is using this flag to avoid the problem? If it is, then it's just a workaround for the real bug, that could simply be generating incorrect AOT .so files for this architecture.

In reply to comment#22:
> I can't reproduce this using mono master under debian wheezy.

Zoltan, given that you have debian-wheeze in ARMHF, to confirm my theory above, can you verify if you reproduce other bug I filed? It is bug 18956.

Thanks
Comment 27 Zoltan Varga 2014-04-12 12:31:05 UTC
AOT on an armhf system works fine for me.
Comment 28 Alex Rønne Petersen 2015-05-07 20:25:50 UTC
Is this still an issue on Mono master?
Comment 29 Alex Rønne Petersen 2015-10-28 06:01:49 UTC
Closing due to inactivity. Feel free to reopen if this issue still exists.