Bug 49824 - Access violation - code c0000005
Summary: Access violation - code c0000005
Status: RESOLVED NOT_ON_ROADMAP
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: 4.8.0 (C9)
Hardware: PC Windows
: --- normal
Target Milestone: ---
Assignee: Niklas Therning
URL:
Depends on:
Blocks:
 
Reported: 2016-12-13 15:15 UTC by Mikkel Kruse Johnsen
Modified: 2017-05-18 07:23 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Embedded C++ source code. (5.40 KB, text/x-c++src)
2017-02-09 10:04 UTC, Mikkel Kruse Johnsen
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NOT_ON_ROADMAP

Description Mikkel Kruse Johnsen 2016-12-13 15:15:45 UTC
Mono 4.8.0.374 crashes on Windows 10.

Compiled on Fedora 24 using mingw32 cross compiler.
(I have been doing this for years and works fine for 4.6 versions of Mono and before)


# mono.exe mozroots.exe

CommandLine: "C:\Program Files (x86)\XMedicus\bin\mono.exe" mozroots.exe

************* Symbol Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
ModLoad: 00400000 00430000   mono-sgen.exe
ModLoad: 77e60000 77fe2000   ntdll.dll
ModLoad: 74fb0000 75090000   WOW64_IMAGE_SECTION
ModLoad: 74fb0000 75090000   C:\WINDOWS\SysWoW64\KERNEL32.DLL
ModLoad: 75090000 75231000   C:\WINDOWS\SysWoW64\KERNELBASE.dll
ModLoad: 761b0000 7626e000   C:\WINDOWS\SysWoW64\msvcrt.dll
ModLoad: 6c540000 6dbb1000   C:\Program Files (x86)\XMedicus\bin\libmonosgen-2.0.dll
ModLoad: 76700000 77ad9000   C:\WINDOWS\SysWoW64\SHELL32.dll
ModLoad: 74a00000 74a77000   C:\WINDOWS\SysWoW64\ADVAPI32.dll
ModLoad: 77c20000 77c61000   C:\WINDOWS\SysWoW64\sechost.dll
ModLoad: 76270000 762a6000   C:\WINDOWS\SysWoW64\cfgmgr32.dll
ModLoad: 74d70000 74e31000   C:\WINDOWS\SysWoW64\RPCRT4.dll
ModLoad: 753a0000 7590e000   C:\WINDOWS\SysWoW64\windows.storage.dll
ModLoad: 74900000 7491e000   C:\WINDOWS\SysWoW64\SspiCli.dll
ModLoad: 748f0000 748fa000   C:\WINDOWS\SysWoW64\CRYPTBASE.dll
ModLoad: 74b50000 74d61000   C:\WINDOWS\SysWoW64\combase.dll
ModLoad: 762b0000 7630a000   C:\WINDOWS\SysWoW64\bcryptPrimitives.dll
ModLoad: 76310000 763f0000   C:\WINDOWS\SysWoW64\ucrtbase.dll
ModLoad: 77ae0000 77b25000   C:\WINDOWS\SysWoW64\powrprof.dll
ModLoad: 77b30000 77c1a000   C:\WINDOWS\SysWoW64\ole32.dll
ModLoad: 766b0000 766db000   C:\WINDOWS\SysWoW64\GDI32.dll
ModLoad: 006b0000 00783000   C:\Program Files (x86)\XMedicus\bin\libgcc_s_sjlj-1.dll
ModLoad: 63b40000 63b4f000   C:\Program Files (x86)\XMedicus\bin\iconv.dll
ModLoad: 75240000 75286000   C:\WINDOWS\SysWoW64\shlwapi.dll
ModLoad: 75da0000 75eff000   C:\WINDOWS\SysWoW64\USER32.dll
ModLoad: 71c40000 71c8e000   C:\WINDOWS\SysWoW64\MSWSOCK.DLL
ModLoad: 74e40000 74ea3000   C:\WINDOWS\SysWoW64\WS2_32.dll
ModLoad: 77d00000 77e5b000   C:\WINDOWS\SysWoW64\gdi32full.dll
ModLoad: 64b40000 64b54000   C:\Program Files (x86)\XMedicus\bin\libwinpthread-1.dll
ModLoad: 766e0000 766f5000   C:\WINDOWS\SysWoW64\win32u.dll
ModLoad: 765b0000 76644000   C:\WINDOWS\SysWoW64\OLEAUT32.dll
ModLoad: 75910000 7598b000   C:\WINDOWS\SysWoW64\msvcp_win.dll
ModLoad: 76650000 7665d000   C:\WINDOWS\SysWoW64\kernel.appcore.dll
ModLoad: 74b40000 74b46000   C:\WINDOWS\SysWoW64\PSAPI.DLL
ModLoad: 74970000 749f8000   C:\WINDOWS\SysWoW64\shcore.dll
ModLoad: 742f0000 742f8000   C:\WINDOWS\SysWoW64\VERSION.dll
ModLoad: 74700000 74724000   C:\WINDOWS\SysWoW64\WINMM.DLL
ModLoad: 76690000 7669f000   C:\WINDOWS\SysWoW64\profapi.dll
ModLoad: 744f0000 74513000   C:\WINDOWS\SysWoW64\WINMMBASE.dll
(17f4.2480): Break instruction exception - code 80000003 (first chance)
eax=00000000 ebx=00000000 ecx=829f0000 edx=00000000 esi=00801e78 edi=00391000
eip=77f06f9c esp=0062fa50 ebp=0062fa7c iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!LdrpDoDebuggerBreak+0x2b:
77f06f9c cc              int     3
0:000> g
ModLoad: 76660000 76685000   C:\WINDOWS\SysWoW64\IMM32.DLL
(17f4.2b5c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000000 ebx=75185444 ecx=ffffffff edx=00000007 esi=02ad2690 edi=6c6c6490
eip=00000002 esp=0463ff28 ebp=02ad2658 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
00000002 ??              ???
Comment 1 Mikkel Kruse Johnsen 2016-12-14 10:10:07 UTC
Seems to only be a problem on 32bit mono version.

The 64bit mono version works fine. (Compiled with Mingw64)
Comment 2 Mikkel Kruse Johnsen 2016-12-16 14:33:44 UTC
We have just tracked it down, to be a problem with cairo and the graphic card in the machine.

Running in safe mode will not cause this error.

So I am closing the bug.

It did happen on 4.8 aswell.
Comment 3 Mikkel Kruse Johnsen 2016-12-17 10:36:47 UTC
This is still a problem. I acidentally clised this
Comment 4 Niklas Therning 2016-12-19 07:46:34 UTC
Can you please try with a Visual Studio compiled 32-bit Mono and see if that works? You can find alpha builds at http://www.mono-project.com/download/alpha/.
Comment 5 Mikkel Kruse Johnsen 2016-12-19 07:59:08 UTC
It works with the Visual Studio compiled 32-bit Mono.

I just don't find it very easy to compile with Visual Studio and I need to create a RPM package, for use with cross packaging.
Comment 6 Niklas Therning 2016-12-22 09:19:56 UTC
What does "cross packaging mean"? Any reason you cannot use the precompiled binaries and repackage those?

I'm afraid Visual Studio is the only toolchain we target and run the test suites on. Because of the Mono build system's autotools/automake dependency we also need to make sure the code compiles with Mingw under Cygwin. But we never use those binaries.

If you provide some more information we might be able to help you understand why it crashes. First step would be to build Mono with debug symbols and run mozroots.exe in gdb to get nice backtraces when the crash occurs.
Comment 7 Mikkel Kruse Johnsen 2017-02-02 08:23:19 UTC
I have the same/similar problem using the precompiled mono 4.8.0.478.

The "mono.exe mozroots.exe" do not fail.

But my own program runs for a while and then crashes with a "Access violation - code c0000005"

I have a problem debugging and getting a call stack. It must happen in a thread.

Should I use gdb or WinDbg ?

Any quick information on how to get a nice call stack ?

(Using Mono 4.6.1.5 there is no problem, everything runs fine)
Comment 8 Mikkel Kruse Johnsen 2017-02-02 13:54:50 UTC
On Linux there is no problem with Mono 4.8.0.478. So I'm positive that the problem is with Mono on Windows.

It could off course be a Gtk problem, but since it works perfectly with  Mono 4.6.1.5 on Windows. It must be a Mono 4.8 problem.
Comment 9 johan.lorensson 2017-02-03 15:37:53 UTC
Hi Mikkel,

If you are running the prebuild Windows 32-bit application it is best debugged using either windbg or Visual Studio. Please note that since you are running a 32-bit application on a 64-bit windows OS it is easier to use a 32-bit version of windbg or you need to do some additional commands to switch to 32-bit debugging mode + registering the debugger for post mortem debugging of 32-bit applications. There are some more details around the different available configuration options on this page:

https://msdn.microsoft.com/en-us/library/windows/hardware/ff542967(v=vs.85).aspx

In short, to setup windbg as your post mortem debugger, you can run it with -I argument, windbg.exe -I, this will register it as the post mortem debugger on the system. NOTE, since you would like to debug a 32-bit application you need to run the 32-bit version of windbg.exe. If you run the 64 bit version you will just setup the post mortem debugger for 64-bit applications.

Run the application and wait for the crash to reproduce, when that happens the system should launch and attach the post mortem debugger to the process and you should have access to the full process from within windbg. If it take some time to reproduce the crash you could also attach windbg directly to the running process, by using File->Attach to a process… from within a 32-bit windbg instance.

If you have problems with stackwalks and symbols you will need to make sure windbg can find and load all needed symbols. For the mono binaries the symbols are located in the mono bin and lib folder (from the same dist version as the binaries), add those paths to the symbol configuration in windbg. You could also be missing OS symbols. There is a shortcut in windbg to setup the OS symbol path. By entering symfix+ in the command window you will download symbols from Microsoft symbol servers. You should also add the paths to mono’s pdb file directories. That could be done in the File->Symbol File Path… dialog (you can force reload all symbols from this dialog as well). This is how it could look if you write .sympath in the command window when OS and mono symbol paths set as above:

Symbol search path is: C:\Program Files (x86)\Mono\bin;C:\Program Files (x86)\Mono\lib;srv*
Expanded Symbol search path is: c:\program files (x86)\mono\bin;c:\program files (x86)\mono\lib;cache*;SRV*https://msdl.microsoft.com/download/symbols

************* Symbol Path validation summary **************
Response                         Time (ms)     Location
OK                                             C:\Program Files (x86)\Mono\bin
OK                                             C:\Program Files (x86)\Mono\lib
Deferred                                       srv*

In order to test symbol resolution you can run the following commands (will test one system library and the mono binaries):

.reload /f kernel32.dll
.reload /f mono-sgen.exe
.reload /f mono-2.0-sgen.dll

If all the above worked as expected you can examine the complete process memory and threads. Once in windbg it is possible to save dump files that can be used later for further analysis.

For more information on the dump command, please have a look at, https://msdn.microsoft.com/en-us/library/windows/hardware/ff562428(v=vs.85).aspx. Example of a dump command, .dump /mfh myfile.dmp

A couple of comments/questions:

Mono 4.8 has been upgraded to a later Visual Studio version (2015) compared to earlier releases. The default mono.exe is also using the sgen GC instead of boehm GC that I believed was used by default in 4.6.

Does the application work as expected under 64-bit Mono 4.8? Can this be viewed as a 32-bit only problem? Have you tried to run with boehm GC instead of sgen GC in 4.8 to see if you get a different behavior?
Comment 10 Mikkel Kruse Johnsen 2017-02-09 10:02:29 UTC
I'm tracking to problem here:

1) Mono 4.8 cross compiled on Linux (Fedora 25) with Mingw32/64.

The compiled mono.exe (sgen) crashes when running "mono.exe mozroots.exe" it shows the info correctly and everything works, but Windows says that the program has stopped working. So it must be a problem when the program shutsdown.

My own program starts fine and is running for 1-2 min and the crashes. This happens when using "mono-sgen.exe xcare.exe" and also when using and C++ binary that embeds mono.


2) Using precompiled mono from official site 4.8.0.483.

I copy "libmonosgen-2.0.dll", "mono-2.0-sgen.dll" and "mono-sgen.dll" to my own program.

Now "mono-sgen.exe mozroots.exe" works fine, no message about it stopped working.

My own program starting it with C++ binary embedding mono, starts fine and running for 1-2 min crashes.
Starting it with "mono-sgen.exe xcare.exe" it now works and do not crash anymore.


So the problem must be in embedding mono.

I can't debug since "libmonosgen-2.0.pdb" is not found. Is it not installed with the mono installer.
Comment 11 Mikkel Kruse Johnsen 2017-02-09 10:04:26 UTC
Created attachment 19769 [details]
Embedded C++ source code.

This is my embedded C++ source code.

...

	MonoAllocatorVTable mem_vtable = {custom_malloc};
	mono_set_allocator_vtable (&mem_vtable);
...

Was something I just added, but the result is the same.
Comment 12 johan.lorensson 2017-03-01 10:26:40 UTC
Hi Mikkel,

When linking and embedding prebuild MSVC mono you should only need to link towards the import library, mono-2.0-sgen.lib (for sgen GC) and then have mono-2.0-sgen.dll and mono-2.0-sgen.pdb (for debug symbols) in the path or in the same directory as your embedding binary. You should not link or use libmonosgen-2,0.dll since that is the Cygwin mingw gcc build mono runtime. When running mono-sgen.exe, it will use the MSVC build mono-2.0.dll.

The prebuild MSVC binary mono-2.0.dll is linked towards VS2015 c-runtime that needs to be installed on the target system, but that should already have been taken care of by the mono installer, but if you need to redist your solution you will need to include the VS2015 c-runtime redist as part of your installation (if using the prebuild MSVC mono runtime). For more information around MSVC c-runtime redist, look here https://msdn.microsoft.com/en-us/library/ms235316.aspx?f=255&MSPPError=-2147217396.

If you build your own mono from source on Windows using supplied MSVC build configurations there is now an option to link towards a static c-runtime and the removes the need to redist c-runtime binaries. There are pro's and con's on both methods (some are explained in the link above).

A couple of things to keep in mind when embedding mono on Windows (NOTE, this is currently just FYI). Since different modules (exe,DLL's) in a process can link different c-runtime versions, it is important to not transfer ownership of resource like memory and file handles between modules (exe, DLL's). For example, file handles opened in one module linked towards one c-runtime version can't be used by a different module linked towards a different c-runtime etc. The same goes for memory ownership, memory allocated by one c-runtime can't be freed by a different c-runtime (different heaps owning the memory). It is however ok to pass and use memory between modules if the ownership rules are honored. Violating these rules will cause undefined behavior and most likely crash the process sooner or later.

NOTE, the above is not mono specific, it applies to all applications using modules linked towards different c-runtimes. Normally the API surface between modules will make sure the rules are honored, but there might be scenarios when the embedder need to keep this in mind as well.

Another scenario getting into the multiple c-runtime issues described above is by building the exe using one compiler like gcc (will use gcc's c-runtime) and then link and load a module build using a different compiler, MSVC for example. This will work as expected if the rules around resources and ABI are followed.

One way to eliminate this issue is to build the embedding module (the exe in this case) using MSVC (VS2015) and then link towards the prebuild MSVC (VS2015) mono runtime. NOTE, since the prebuild mono runtime is linked towards the release version of the c-runtime (and if the problem is due to violating the rules around resource sharing) you will need to verify the theory by running the release version of the application. If you run the default MSVC debug configuration your application will be linked towards the debug version of the MSVC c-runtime causing the same issues with multiple c-runtimes as described above (in this case the debug and release versions of the c-runtime).

Above might sound complex and problematic if you never been exposed to it. Normally the above is not an issue if the rules are correctly followed, but it might be worth keeping in mind especially if you are extending mono by supplying alternative implementations, callbacks or other hooks.

For this specific issue, I believe the first thing to do is to switch to the prebuild MSVC mono runtime, since it looks like you still use the gcc build version (libmonosgen-2.0.dll) of mono runtime. If the issue is around the embedding together with the gcc build mono runtime, then you are still getting into the same problem. When running mono-sgen.exe it will use the prebuild MSVC mono runtime and if you don't have the issue running mono-sgen.exe, then first step is to link and use prebuild MSVC (you will also get working symbols for the debugger). Hopefully just switching to the prebuild MSVC version of mono runtime will resolve the issue.
Comment 13 johan.lorensson 2017-03-02 10:35:29 UTC
References to mono-2.0.dll in above comment should be mono-2.0-sgen.dll.
Comment 14 Mikkel Kruse Johnsen 2017-05-18 07:23:32 UTC
I now compile Mono on Linux using Mingw32/64 but I copy:

mono-2.0-sgen.dll
mono-sgen.exe

From windows to my Linux machine and replace the Linux build files with these ones.

I also package VCRuntime140 into the package just in case.

I hope that Mono will be fixed in the future to be able to work when compiled with Mingw32/64.

(The Ming64 build works, it is only the Muingw32 that fails).

Also these.
mono-2.0-sgen.lib
mono-2.0-sgen.pdb
mono-sgen.pdb