Bug 21492 - SIGSEGV when calling libsqlite on multiple background threads
Summary: SIGSEGV when calling libsqlite on multiple background threads
Status: RESOLVED INVALID
Alias: None
Product: Android
Classification: Xamarin
Component: Mono runtime / AOT Compiler ()
Version: 4.14.0
Hardware: PC Mac OS
: High normal
Target Milestone: ---
Assignee: Jonathan Pryor
URL:
Depends on:
Blocks:
 
Reported: 2014-07-21 17:18 UTC by Brendan Zagaeski (Xamarin Team, assistant)
Modified: 2014-07-24 22:06 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Test case (77.73 KB, application/zip)
2014-07-21 17:18 UTC, Brendan Zagaeski (Xamarin Team, assistant)
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED INVALID

Description Brendan Zagaeski (Xamarin Team, assistant) 2014-07-21 17:18:06 UTC
Created attachment 7440 [details]
Test case

It appears that in certain circumstances (maybe when p/invoking native functions on background threads?), Xamarin.Android sometimes writes to memory it shouldn't. In the attached test case, the inputs to the `sqlite3_prepare_v2()` method seem to be getting corrupted.

The corresponding Java test case (also included) does not have the problem.


## Steps to reproduce

Build and run the attached "Xamarin" test case on an armeabi-v7a device (not emulator). You can also run the attached "Java" test case for comparison. To run the "Java" test case: (1) open it in Eclipse, (2) allow the automatic build process to complete, (3) launch the application using the "Debug" icon.


## Test case details

The test case first calls `sqlite3_open_v2()`, and then repeatedly calls `sqlite3_prepare_v2()` in several background threads. To make the Java and Xamarin test cases as similar as possible, I made a tiny little "sqlite_test" library that hard-codes the input values for these two native methods. Instructions to build "sqlite_test" with the Android NDK are included in the zip file (in the README). But the test case includes a pre-built version of the library in both the Xamarin and Java projects, so there's no need to build the "sqlite_test" library before running either project.

The original full app that showed the problem used the SQLite.NET component and `ThreadPool.QueueUserWorkItem()`.


## Result (Java)

The program runs without error.


## Result (Xamarin)

The app will occasionally run without error, but usually one or more of the background threads crash with something similar to this:

> [SQLiteLog] (1) no such table: tbl1
> [mono-rt] Stacktrace:
> [mono-rt] 
> [mono-rt]   at <unknown> <0xffffffff>
> [mono-rt]   at (wrapper managed-to-native) AndroidApp1.MainActivity.Prepare () <IL 0x00022, 0xffffffff>
> [mono-rt]   at AndroidApp1.MainActivity/AsyncRunQuery.DoInBackground (Java.Lang.Object[]) [0x00009] in /Volumes/Cases/Android_SQL_SIGSEGV_MLindestam_2014_07_13/AndroidApp1/AndroidApp1/MainActivity.cs:67
> [mono-rt]   at Android.OS.AsyncTask.n_DoInBackground_arrayLjava_lang_Object_ (intptr,intptr,intptr) [0x00020] in /Users/builder/data/lanes/monodroid-mlion-monodroid-4.12-series/163212a9/source/monodroid/src/Mono.Android/platforms/android-19/src/generated/Android.OS.AsyncTask.cs:232
> [mono-rt]   at (wrapper dynamic-method) object.8c2d1b2e-a867-4e29-bccd-c4175a52646f (intptr,intptr,intptr) <IL 0x00017, 0x00047>
> [mono-rt]   at (wrapper native-to-managed) object.8c2d1b2e-a867-4e29-bccd-c4175a52646f (intptr,intptr,intptr) <IL 0x00025, 0xffffffff>
> [mono-rt] 
> [mono-rt] =================================================================
> [mono-rt] Got a SIGSEGV while executing native code. This usually indicates
> [mono-rt] a fatal error in the mono runtime or one of the native libraries 
> [mono-rt] used by your application.
> [mono-rt] =================================================================
> [mono-rt]


## Additional information

If you limit the number of background threads to 1, the problem does not occur:

> for (var i = 0; i < 1; i++) {
>   new AsyncRunQuery().Execute();
> }


In my tests, 2 threads was sufficient to produce the problem.


## Additional Error Output (Xamarin)

Sometimes the program produces other error messages. Here are a few examples, each from a different run. All of these are excerpts from the full output.

###
> [mono-rt] Stacktrace:
> [mono-rt] 
> [mono-rt]   at <unknown> <0xffffffff>
> [libc] @@@ ABORTING: HEAP MEMORY CORRUPTION IN dlmalloc addr=0x59b4ff90


###
> @@@ ABORTING: INVALID HEAP ADDRESS IN dlfree addr=0x4114d4d8


###
> @@@ ABORTING: INVALID HEAP ADDRESS IN dlfree addr=0x51629ff0


###
> [] * Assertion at /Users/builder/data/lanes/monodroid-mlion-monodroid-4.12-series/163212a9/source/mono/mono/io-layer/wthreads.c:108, condition `ok' not met


###
> [SQLiteLog] (1) no such table: ����������������j


###
> [SQLiteLog] (1) no such table: SELECT * FROM tbl1
> [SQLiteLog] (1) no such table: tbl1
> [SQLiteLog] (1) no such table: tbl1
> [mono-rt] Stacktrace:
> [mono-rt] 
> [mono-rt]   at <unknown> <0xffffffff>[SQLiteLog] (1) no such table: SELECT * FROM tbl1
 


## Version information

### Device

LG Optimus L9 (LGMS769), Android 4.1.2

I wasn't able to reproduce on emulators.


### Xamarin.Android

Tested on: 4.12.6-1 and 4.14.0-51
Comment 2 Brendan Zagaeski (Xamarin Team, assistant) 2014-07-21 19:12:00 UTC
## Partial workaround for the test case

Place the call to `Prepare()` in a `lock`:

> lock(myObject) {
>     Prepare();
> }


This prevents the crashes.

Note that `sqlite3_threadsafe()` returns "2" in my test setup. So libsqlite was compiled with thread safety enabled.
Comment 3 Prashant manu 2014-07-24 03:26:31 UTC
We have checked with the project (XamarainSqliteTest> AndroidApp1) and able to reproduce this issue (2 out of 6 attempts) on Device Samsung Galaxy Note 3 (android 4.4.2)

Supplement Info:
Application Output: https://gist.github.com/sunil360/95001dd8f198583d00bd
IDE log: https://gist.github.com/sunil360/19f8d5e4294fbd2bbf0b

Environment Info:
Xamarin Studio
Version 5.2 (build 384)
Installation UUID: 561c7a69-0a91-4bae-ad7c-f0c79d594337
Runtime:
	Mono 3.6.0 ((no/f540f8a)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 306000039

Apple Developer Tools
Xcode 5.1.1 (5085)
Build 5B1008

Xamarin.iOS
Version: 7.2.6.26 (Business Edition)
Hash: f10e5ed
Branch: 
Build date: 2014-07-23 11:06:08-0400

Xamarin.Mac
Version:

Xamarin.Android
Version: 4.14.0 (Business Edition)
Android SDK: /Users/tajinder/Desktop/android-sdk-macosx
	Supported Android versions:
		2.1   (API level 7)
		2.2   (API level 8)
		2.3   (API level 10)
		3.1   (API level 12)
		3.2   (API level 13)
		4.0   (API level 14)
		4.0.3 (API level 15)
		4.1   (API level 16)
		4.2   (API level 17)
		4.3   (API level 18)
		4.4   (API level 19)
Java SDK: /usr
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

Build Information
Release ID: 502000384
Git revision: 09e06441e39ea272ddda63758fd53013157f7e45
Build date: 2014-07-16 11:15:54-04
Xamarin addins: 2d11dd4dad5b35ca83f24d70e965792a47be8291

Operating System
Mac OS X 10.8.4
Darwin Tajinders-iMac.local 12.4.2 Darwin Kernel Version 12.4.2
    Mon Jun 17 18:00:12 PDT 2013
    root:xnu-2050.45.8~1/RELEASE_X86_64 x86_64
Comment 4 Jonathan Pryor 2014-07-24 19:15:42 UTC
I do not believe that this is a Mono bug, per-se. I believe that it is an application bug, and the Java version is "lucky."

sqlite3_threadsafe() returns 2 on my device: http://www.sqlite.org/c3ref/threadsafe.html

I believe that this means that my libsqlite.so was built with -DSQLITE_THREADSAFE=2:

http://www.sqlite.org/compile.html#threadsafe

Which in turn gets confusing:

> When compiled with SQLITE_THREADSAFE=2, SQLite can be used in a 
> multithreaded program so long as no two threads attempt to use the 
> same database connection (or any prepared statements derived from 
> that database connection) at the same time.

which I believe can be interpreted as meaning "don't create a DB connection on one thread and share on another."

With lots of investigation, I found the following:

If I move the Open() invocation outside of OnCreate() and into AsyncTask.DoInBackground, causing the same thread to both open the database and prepare the statements on the database, then the code works without the crash.

If I instead alter sqlite_test_open() so that it passes SQLITE_OPEN_FULLMUTEX to sqlite3_open_v2(), then it likewise works without a crash:

http://www.sqlite.org/c3ref/c_open_autoproxy.html
http://www.sqlite.org/threadsafe.html

    sqlite3_open_v2("/data/data/AndroidApp1.AndroidApp1/files/test.db3",
            &dbref, 
            2 | 4 | 0x00010000,
            NULL);

If I attach gdb for the stack trace of the original crash...I get a stack which is of no help at all:

> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 16558]
> 0x409046c0 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> (gdb) bt
> #0  0x409046c0 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #1  0x409048c4 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #2  0x409048e4 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #3  0x4092993e in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #4  0x4092c1a2 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #5  0x4092ce50 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #6  0x4092d05c in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #7  0x4092d2fa in sqlite3_prepare () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #8  0x4092d370 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #9  0x4092ff46 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #10 0x409301e8 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #11 0x4093022c in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #12 0x40930524 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #13 0x4093066a in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #14 0x408ffe16 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #15 0x4090076e in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #16 0x4091c14c in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #17 0x40929db4 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #18 0x4092c1e6 in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #19 0x4092ce2a in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #20 0x4092d05c in ?? () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #21 0x4092d1c6 in sqlite3_prepare_v2 () from /Users/jon/Downloads/bxc-21492/SqliteInvalidAddress/XamarinSqliteTest/AndroidApp1/gdb-symbols/libsqlite.so
> #22 0x4d03f5ee in ?? ()
> #23 0x4d03f5ee in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Many other threads were similarly corrupt, so a post-SIGSEGV investigation doesn't get far.
Comment 5 Brendan Zagaeski (Xamarin Team, assistant) 2014-07-24 20:09:43 UTC
Oh, that sounds exactly right! Many thanks for the investigation! If only I had clicked the link for the SQLITE_THREADSAFE compiler macro while working on this!

From the "threadsafe.html" page alone, it sounded like as long as `sqlite3_threadsafe()` returned 1 or 2, you could use threads however you wanted.
Comment 6 Brendan Zagaeski (Xamarin Team, assistant) 2014-07-24 20:13:58 UTC
From the `sqlite3_threadsafe()` "threadsafe.html" page alone, that is:
http://www.sqlite.org/c3ref/threadsafe.html
Comment 7 Brendan Zagaeski (Xamarin Team, assistant) 2014-07-24 22:06:31 UTC
Just as one additional bit of verification and documentation, I tried adding the `SQLiteOpenFlags.FullMutex` flag to the `new SQLiteConnection()` call in the original full SQLite.NET app that, and that does indeed prevent the crashes.