Bug 2473 - Webservices 2.0 Async download: memory is not released, app gets stuck or is running OOM. Demo included
Summary: Webservices 2.0 Async download: memory is not released, app gets stuck or is ...
Status: RESOLVED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: XI runtime ()
Version: 5.0
Hardware: PC Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2011-12-12 18:14 UTC by René Ruppert
Modified: 2011-12-14 06:55 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Monodevelop project to demonstrate the issue (76.43 KB, application/zip)
2011-12-12 18:14 UTC, René Ruppert
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description René Ruppert 2011-12-12 18:14:56 UTC
Created attachment 1037 [details]
Monodevelop project to demonstrate the issue

Setup:
iPad2, iOS 5.0, iOS SDK 5.0.1, MT 5.0.1, Deployment Target 5.0

The demo project:
my app is calling a web service that delivers a string as response. The string is XML that contains base 64 encoded data representing binary files that can be downloaded. So much for the use case.

The download is served in chunks. The client can define the chunk size. For desktop applications we usually use 1MB per chunk.
For the iPad I chose 200kb because it has less memory but the test case also supports 1MB per chunk; that brings it to its limit quicker.

The test connects to the web service and starts downloading the same file over and over again. The file has appr. 40MB. With 1MB per chunk we end up with 40 calls to the server.

There are TWO ISSUES:

1. The app sometimes completely fails to init. It starts downloading chunk 0 but never returns. It is stuck somewhere in MT code. It looks like a similar issue that existed in MT 4 and was caused by a socket deadlock. This happens more often when using 1MB chunks instead of 200kb chunks.

2. If the download starts: After a while the test crashes. Console shows that it was killed due to OOM. It received three memory warnings but memory was not reclaimed.

If you run the test under Instruments' Activity Monitor, you can see that Real Memory grows and grows, up to 300MB. That's when either the app gets stuck or receives memory warnings but does never release any memory. It stays high until the app gets killed. Calling GC.Collect() manually does not change this behavior.

The question is: where does that memory go? The strings returned by the web service call are not hold in references anywhere. The all finished and the result is dropped.

If I run the same code in the Simulator, memory stays low and constant all the time. Same for Windows. That's why I assume a bug/an issue with Monotouch.

Also noteworthy: If I use the synchronous method DMOGetChunkedWithChunkSize() instead of the async one memory STAYS VERY LOW and CONSTANT. The app never gets stuck.

The implementation:
Hitting the button fires up a thread. That thread does an async call to the server. It does a loop using Thread.Sleep() until the chunk has been downloaded.
Once a flag has been set by the completion handler, the next chunk is downloaded. I used this weird looking Thread construction because I wanted to replicate the situation of the real project. However you can leave away the Thread and it will fail just the same.
Comment 1 René Ruppert 2011-12-12 18:17:55 UTC
I just noticed that there seems to be a bug in our Webservice. It looks like after having downloaded the file once and immediately beginning another download creates an invalid response.
But this into the cause of the problem of course. Even with only only one download it is very obvious that memory consumption is enorm and goes easily up to some hundred MB where as it stays low when using the synchronous version.
Comment 2 René Ruppert 2011-12-12 18:44:11 UTC
And one more note: the "while(true) { Thread.Sleep() }" loop contains a check in the real code if cancellation has been requested. Otherwise I would use an AutoReset event (in fact I did, but it does not influence memory consumption).
Comment 3 René Ruppert 2011-12-12 18:52:59 UTC
I have tried to update to MT 5.1.1 beta but there I get an exception when I hit the button:

Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>: Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>:   at System.Threading.Thread.GetNSAutoreleasePool () [0x00000] in <filename unknown>:0 
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>:   at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>: [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>:   at System.Threading.Thread.GetNSAutoreleasePool () [0x00000] in <filename unknown>:0 
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>:   at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>: Terminating runtime due to unhandled exception
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>: Stacktrace:
Dec 13 00:50:45 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>: Native stacktrace:
Dec 13 00:50:46 unknown UIKitApplication:webservicetest[0x4f9][6063] <Notice>:
Comment 4 Sebastien Pouliot 2011-12-13 08:34:00 UTC
wrt comment #3 this is already fixed (and will be part of 5.1.2) and caused by the linker removing NSAutoreleasePool (it's used by reflection so the linker needed to know a bit more about it). 

You can either turn off linking or (better) use NSAutoreleasePool somewhere in your application so the linker won't remove it.
Comment 5 René Ruppert 2011-12-13 08:36:58 UTC
I stick with MT 5.0.1 for now. I only wanted to see if the memory issues is maybe fixed.
Comment 6 Rolf Bjarne Kvinge [MSFT] 2011-12-13 17:17:46 UTC
I've found it, it's a bug in the class libraries. Every result from the server is kept in a hash table, which is why it's not freed by the GC.

Thanks a lot for the test case, it made it a lot easier to reproduce and track down!

Fixed in d3e516763 (master) and f4433694 (2.10).
Comment 7 René Ruppert 2011-12-14 04:21:11 UTC
Very cool! Thank You! But I have more questions regarding this:

1. I still need to use the timer trick, otherwise the async request gets stuck randomly. This is not the case when using the synchronous method. Is this also fixed?

2. The demo project from above works without issues if I connect to our internal development platform. Memory consumption is constant. The only differences I see: our internal server uses a certificate signed by us and I implement the ServicePoint callback to tell the client that the certificate is okay and the connection is of course faster over the LAN.

3. My workaround for now is to dispose the service instance after each call and to trigger a GC.Collect. This keeps my memory at around 50MB. Is there anything else I can do until a new MT version is available?
Comment 8 René Ruppert 2011-12-14 04:21:43 UTC
Oh, and are you done testing on our platform? Because then I will close your test account.
Comment 9 Rolf Bjarne Kvinge [MSFT] 2011-12-14 06:55:57 UTC
1. I have not been able to reproduce the hang, it has always started downloading for me. Can you try again when the next beta comes out and if it's still happening file another bug?

2. This is somewhat inexplicable, but I guess it goes through some other code path that doesn't trigger this bug.

3. That's exactly the workaround, the hash table I mentioned is on the service instance, so if you let the service instance be GC'ed, the hash table (and all the data in it) will also be freed.

And yes, I'm done testing :)