Bug 949 - managed-to-native wrapper fails to call a function pointer inside a structure defined in managed code
Summary: managed-to-native wrapper fails to call a function pointer inside a structure...
Status: RESOLVED INVALID
Alias: None
Product: Runtime
Classification: Mono
Component: Interop ()
Version: unspecified
Hardware: PC Windows
: --- critical
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2011-09-20 21:36 UTC by Vinicius Fonseca
Modified: 2011-10-24 10:02 UTC (History)
2 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED INVALID

Description Vinicius Fonseca 2011-09-20 21:36:51 UTC
First of all, I'm sorry for not being able to provide a simpler test case.

I'm working on a project that aims to be the first portable open-source FFmpeg wrapper in C#.NET (source code attached). This project contains a lot of FFmpeg structs marked with the [LayoutKind.Sequential] attribute for interop. The solution file also contains a very simple console project that creates a video file through the wrapper. The solution works great in Windows / MS .NET Framework 2.0, but it doesn't work in Linux / Mono 2.10.5. The FFmpeg libraries were built for Windows and Linux from the same source files. 

Expected Result:
----------------

To work perfectly as it works in Windows / MS .NET Framework, that is, to output a video file which plays the runtime-generated frames by the console project included in this solution.


Obtained Result:
----------------

The following output (I've compiled FFmpeg with debug-level=3 in order to obtain this detailed output):

vinicius@ubuntu:~$ sudo ./FFmpegNet.Console.exe test.avi
[sudo] password for vinicius:
Stacktrace:

  at (wrapper managed-to-native) FFmpegNet.Interop.FFmpeg.avcodec_encode_video (FFmpegNet.Interop.Codec.AVCodecContext&,byte[],int,FFmpegNet.Interop.Codec.AVFrame&) <0xffffffff>
  at FFmpegNet.VideoFileWriter.write_video_frame (FFmpegNet.WriterPrivateData&) <0x000a7>
  at FFmpegNet.VideoFileWriter.WriteVideoFrame (System.Drawing.Bitmap,System.TimeSpan) <0x004df>
  at FFmpegNet.VideoFileWriter.WriteVideoFrame (System.Drawing.Bitmap) <0x00043>
  at FFmpegNet.Console.Program.Main (string[]) <0x0010f>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        /usr/bin/cli() [0x80e126c]
        /usr/bin/cli() [0x812046c]
        /usr/bin/cli() [0x805fe7d]
        [0xb776240c]

Debug info from gdb:

[Thread debugging using libthread_db enabled]
[New Thread 0xb7085b70 (LWP 10805)]
0xb7762424 in __kernel_vsyscall ()
  Id   Target Id         Frame
  2    Thread 0xb7085b70 (LWP 10805) "cli" 0xb7762424 in __kernel_vsyscall ()
* 1    Thread 0xb7588b60 (LWP 10803) "cli" 0xb7762424 in __kernel_vsyscall ()

Thread 2 (Thread 0xb7085b70 (LWP 10805)):
#0  0xb7762424 in __kernel_vsyscall ()
#1  0xb7711d35 in sem_wait@@GLIBC_2.1 () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x0820b7a8 in mono_sem_wait ()
#3  0x0815061c in ?? ()
#4  0x081d942d in ?? ()
#5  0x08205da1 in ?? ()
#6  0x082274e8 in ?? ()
#7  0xb770bd31 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#8  0xb765b0ce in clone () from /lib/i386-linux-gnu/libc.so.6
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 1 (Thread 0xb7588b60 (LWP 10803)):
#0  0xb7762424 in __kernel_vsyscall ()
#1  0xb7712a8b in read () from /lib/i386-linux-gnu/libpthread.so.0
#2  0x080e144f in ?? ()
#3  0x0812046c in ?? ()
#4  0x0805fe7d in ?? ()
#5  <signal handler called>
#6  0x00000000 in ?? ()
#7  0xb64a966b in ff_thread_get_buffer (avctx=0xbf85f320, f=0x9255a50) at libavcodec/pthread.c:787
#8  0xb646e480 in alloc_frame_buffer (s=0x924b100, pic=0x9255a50) at libavcodec/mpegvideo.c:231
#9  0xb646e690 in ff_alloc_picture (s=0x924b100, pic=0x9255a50, shared=0) at libavcodec/mpegvideo.c:277
#10 0xb647fb1f in load_input_picture (s=0x924b100, pic_arg=0xbf85f120) at libavcodec/mpegvideo_enc.c:874
#11 0xb64812d4 in MPV_encode_picture (avctx=0xbf85f210, buf=0x87010 "", buf_size=460800, data=0xbf85f120) at libavcodec/mpegvideo_enc.c:1256
#12 0xb6528e00 in avcodec_encode_video (avctx=0xbf85f210, buf=0x87010 "", buf_size=460800, pict=0xbf85f120) at libavcodec/utils.c:629
#13 0xb4eccc8c in ?? ()
#14 0xb70b4b38 in ?? ()
#15 0xb70b4008 in ?? ()
#16 0xb70b3b14 in ?? ()
#17 0xb70a6e08 in ?? ()
#18 0xb70a6f17 in ?? ()
#19 0x08064afc in ?? ()
#20 0x081a3d6f in mono_runtime_invoke ()
#21 0x081a6b65 in mono_runtime_exec_main ()
#22 0x080bb61b in mono_main ()
#23 0x0805990f in ?? ()
#24 0xb75a2113 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6
#25 0x080599d5 in ?? ()
Backtrace stopped: Not enough registers or memory available to unwind further

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

---

Everything that I can tell by reading this output is that the error is thrown at line 787 of the file libavcodec/pthread.c, which reads:

	return avctx->get_buffer(avctx, f);

This is relative to a function pointer passed by a struct defined in managed code (my code), in the file AVCodecContext.cs, line 404, which reads:

	public IntPtr get_buffer_ptr;

This should be a simple function pointer used by the library itself. Maybe someone who understands the inner-workings of mono can figure it out just by reading these information that I've collected, without having to test it (which is really a pain in the a55)...


-------------------------------------------------------------------------------------

Here is some information about my test environments:



Windows environment:
--------------------

C:\Users\Vinicius>ver
Microsoft Windows [Version 6.1.7601]

Windows 7 x64
Visual Studio 2010
MS .NET Framework 2.0


Linux Environment:
------------------

Ubuntu 11.10

vinicius@ubuntu:~$ mono --version
Mono JIT compiler version 2.10.5 (Debian 2.10.5-1)
Copyright (C) 2002-2011 Novell, Inc, Xamarin, Inc and Contributors. www.mono-pro       ject.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  x86
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            Included Boehm (with typed GC and Parallel Mark)

vinicius@ubuntu:~$ uname -a
Linux ubuntu 3.0.0-9-generic-pae #15-Ubuntu SMP Tue Aug 30 16:33:52 UTC 2011 i686 i686 i386 GNU/Linux


FFmpeg version (both Windows and Linux):
----------------------------------------

git-a304071
	libavutil    50. 40. 1 / 50. 40. 1
	libavcodec   52.120. 0 / 52.120. 0
	libavformat  52.108. 0 / 52.108. 0
	libavdevice  52.  4. 0 / 52.  4. 0
	libavfilter   1. 79. 0 /  1. 79. 0
	libswscale    0. 13. 0 /  0. 13. 0

FFmpeg configured with:
	--disable-static
	--enable-shared
	--enable-gpl
	--enable-version3
	--enable-runtime-cpudetect
	--enable-memalign-hack
	--enable-avisynth
	--enable-bzlib
	--enable-frei0r
	--enable-libopencore-amrnb
	--enable-libopencore-amrwb
	--enable-libfreetype
	--enable-libgsm
	--enable-libmp3lame
	--enable-libopenjpeg
	--enable-librtmp
	--enable-libschroedinger
	--enable-libspeex
	--enable-libtheora
	--enable-libvorbis
	--enable-libvpx
	--enable-libx264
	--enable-libxavs
	--enable-libxvid
	--enable-zlib
	--cross-prefix=i686-w64-mingw32-
	--target-os=mingw32
	--arch=x86_32
	--extra-cflags="-I/home/kyle/software/ffmpeg/external-libraries/win32/include"
	--extra-ldflags="-L/home/kyle/software/ffmpeg/external-libraries/win32/lib"
	--pkg-config=pkg-config


The source code for this FFmpeg build can be found at:
	http://hawkeye.arrozcru.org/source/ffmpeg/ffmpeg-git-a304071.tar.bz2
	
This version of FFmpeg was built on:
	Ubuntu Desktop 11.10: http://www.ubuntu.com/desktop
	
The cross-compile toolchain used to compile this FFmpeg was:
	MinGW-w64 r4138: http://mingw-w64.sourceforge.net/

The GCC version used to compile this FFmpeg was:
	GCC 4.5.2: http://gcc.gnu.org/
	
The external libaries compiled into this FFmpeg are:
	bzip2 1.0.6 http://www.bzip.org
	Frei0r 1.3 http://frei0r.dyne.org/
	opencore-amr 0.1.2 http://sourceforge.net/projects/opencore-amr/
	FreeType 2.4.4 http://www.freetype.org/
	gsm 1.0.13-3 http://libgsm.sourcearchive.com/
	LAME 3.98.4 http://lame.sourceforge.net/
	OpenJPEG 1.4 http://www.openjpeg.org/
	RTMP r568 http://rtmpdump.mplayerhq.hu/
	Schroedinger 1.0.10 http://diracvideo.org/
	Speex 1.2rc1 http://www.speex.org/
	Theora 1.1.1 http://www.theora.org/
	Vorbis 1.3.2 http://www.vorbis.com/
	libvpx 0.9.6 http://www.webmproject.org/code/
	x264 git-aa21558b http://www.videolan.org/developers/x264.html
	XAVS r51 http://xavs.sourceforge.net/
	Xvid 1.3.1 http://www.xvid.org/
	zlib 1.2.5 http://zlib.net/

-----------------------------------------------------------------------------

Thanks!
Comment 1 Vinicius Fonseca 2011-09-20 21:46:40 UTC
I'm not being able to attach any files to this bug (maybe because IE9). Please ask me the file if you are interested in testing this, I'll be happy to provide it. Thanks.
Comment 2 Zoltan Varga 2011-09-22 19:59:44 UTC
These problems are historically caused by incorrect managed structure definitions which don't match their native counterparts on some platforms. Some things to watch out for:
- 32/64 bit issues in general.
- the 'long' data type. See:
http://www.mono-project.com/Interop_with_Native_Libraries#Longs
Comment 3 Zoltan Varga 2011-09-27 15:19:43 UTC
-> NEEDINFO.
Comment 4 Vinicius Fonseca 2011-10-06 10:21:09 UTC
Hi Zoltan!

Thank you for the quick response and sorry about my delayed reply.

I've reviewed my managed structures searching for any non-compatible data type or misaligned (or misordered) property, but everything really looks OK. The link that you provided was really interesting, but the nice people who wrote the FFmpeg code was very cautious about these data types, avoiding using the "long" and "ulong" data types by using "t_int64" and "t_uint64" instead.

It can be (and probably is) something wrong in my managed code, but, after reviewing the code, I'm really stucked right now.

Thank you so much anyway.

Regards,

Vinicius
Comment 5 Vinicius Fonseca 2011-10-21 22:45:02 UTC
Hi again Zoltan!

I finally found what is wrong with my program. I still don't know if this can be considered a Mono bug, a .Net bug, or something that I simply should be aware of. Please let me know what you think.

The test case is very simple:

----------------------------------------------------------
using System;
using System.Runtime.InteropServices;

namespace TestCase {
	class Program {

		[StructLayout(LayoutKind.Sequential)]
		unsafe struct Test {
			public fixed ulong test[4];
			public int temp;
		}

		static void Main(string[] args) {

			Console.WriteLine(Marshal.SizeOf(typeof(Test)));
			Console.ReadKey();
		}
	}
}
----------------------------------------------------------

Result in Win 7 x64 with MS .NET Framework 2.0 : 40
Result in Ubuntu with Mono 2.10.5              : 36

Notice that removing the second field (int temp), the returned size is always 32 on both platforms. Marshal.SizeOf(typeof(int)) is 4 on both platforms. My code is being compiled for x86 only. For me it looks like Mono is doing it right, but my code is working only on Windows...(?)

Thanks a lot!

Best regards,

Vinicius
Comment 6 Zoltan Varga 2011-10-22 04:13:20 UTC
This is due to alignment differences between 32 bit and 64 bit machines. On 32 bit machines, the 'test' field has alignment 4, causing the whole structure to have 4 alignment. On 64 bit systems, it is 8, causing the increase to the size of the structure, since the structure size must be a multiple of the alignment.
Comment 7 Vinicius Fonseca 2011-10-22 13:50:10 UTC
Hi Zoltan!

Thank you for the reply. That's what I thought, but when I use LayoutKind.Explicit (code below), both platforms returns 40 as the structure size. Is that correct?

[StructLayout(LayoutKind.Explicit)]
unsafe struct Test {
	[FieldOffset(0)]
	public fixed ulong test[4];
	[FieldOffset(32)]
	public int temp;
}

Maybe this is because I'm compiling this file on the x64 machine?

Thanks again!

Regards,

Vinicius
Comment 8 Zoltan Varga 2011-10-22 14:47:53 UTC
No idea. It should match the size/layout of the unmanaged structure.
Comment 9 Vinicius Fonseca 2011-10-23 18:42:46 UTC
Hi Zoltan!

I went ahead and tested this case (source code I provided in comment 5) on three different environments (always compiling as x86-only):

Win 7 x64 (MS .NET 2.0) result   : 40 bytes
Win XP x86 (MS .NET 2.0) result  : 40 bytes
Ubuntu 11.10 (Mono 2.10.5) result: 36 bytes

Defining the LayoutKind.Explicit, like I did in in comment 7:

Win 7 x64 (MS .NET 2.0) result   : 40 bytes
Win XP x86 (MS .NET 2.0) result  : 40 bytes
Ubuntu 11.10 (Mono 2.10.5) result: 40 bytes

Defining the LayoutKind.Sequential and Pack=1:

Win 7 x64 (MS .NET 2.0) result   : 36 bytes
Win XP x86 (MS .NET 2.0) result  : 36 bytes
Ubuntu 11.10 (Mono 2.10.5) result: 36 bytes
 
I have a very strong feeling that Mono has StructLayoutAttribute.Pack = 1 as its default value for StructLayoutAttribute.Pack, and totally ignores the default value of StructLayoutAttribute.Pack = 0 (even when explicitly defined). I'm saying this not only because of the results that I got, but also because of this bug entry:

http://lists.ximian.com/pipermail/mono-bugs/2008-December/083580.html

Best regards,

Vinicius




---

using System;
using System.Runtime.InteropServices;

namespace TestCase {
    class Program {

        [StructLayout(LayoutKind.Sequential)]
        unsafe struct Test {
            public fixed ulong test[4];
            public int temp;
        }

        static void Main(string[] args) {

            Console.WriteLine(Marshal.SizeOf(typeof(Test)));
            Console.ReadKey();
        }
    }
}
Comment 10 Zoltan Varga 2011-10-23 19:07:47 UTC
I don't think this is a bug. On 32 bit linux, 'long' values are aligned on 4 bytes in
native code, while on 32 bit/64 bit windows, they are aligned at 8 bytes. This
is why the struct size is 36 bytes on linux, and 40 on windows.
Comment 12 Zoltan Varga 2011-10-24 04:23:05 UTC
Marshal.SizeOf () returns the size of the native representation of a structure, which of course could be different between platforms.
Comment 13 Vinicius Fonseca 2011-10-24 09:11:29 UTC
Thank you very much for the clarification. Please let me know if I have to take any actions to close this bug, like changing status or something like that...
Comment 14 Zoltan Varga 2011-10-24 10:02:35 UTC
-> NOTABUG.