Bug 30640 - Local variable is not collected and leads to crash related to memory use
Summary: Local variable is not collected and leads to crash related to memory use
Status: RESOLVED ANSWERED
Alias: None
Product: iOS
Classification: Xamarin
Component: General ()
Version: XI 8.10
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2015-06-01 15:20 UTC by John Miller [MSFT]
Modified: 2015-06-03 23:07 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Native Test Case (157.29 KB, application/zip)
2015-06-01 15:20 UTC, John Miller [MSFT]
Details
Xamarin Test Case (143.37 KB, application/zip)
2015-06-01 15:20 UTC, John Miller [MSFT]
Details
Crash Log (8.15 KB, application/octet-stream)
2015-06-01 15:21 UTC, John Miller [MSFT]
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED ANSWERED

Description John Miller [MSFT] 2015-06-01 15:20:10 UTC
Created attachment 11408 [details]
Native Test Case

**Overview:**

   A local variable contains a long string value and does not seem to be collected when method is finished doing its work. 

**Steps to Reproduce:**

   1. Run the attached Xamarin sample on an iOS Device (Tested on iPad 4th gen)
   2. Move the slider up to 10 iterations
   3. Press Go
   4. Repeat steps 3 a few times once it's finished doing it's work

**Actual Results:**

   The app will crash after first receiving some memory warnings. 
   http://screencast.com/t/f0gvNB9j

**Expected Results:**

   No crash. Should be able to run this many times without an issue, similar to the native sample project attached. 

**Build Date & Platform:**

   XI 8.10.0.3
   iOS 8+
   iPad 4th Gen

**Additional Information:**

   I could not reproduce the issue on my iPhone 5s and the customer could not reproduce on an iPad Air. I am wondering if that is because they are 64 bit devices. 
   It's possible this could crash on a 32bit iPhone too. 
   I've also attached the crash log generated from the iPad test. 
   The projects can be modified for iPhone by simply changing the supported devices to iPhone and specifying the Main Interface to the "Main" storyboard. 
   A native Xcode project is provided to show a comparison with how that works and produces the expected results.
Comment 1 John Miller [MSFT] 2015-06-01 15:20:54 UTC
Created attachment 11409 [details]
Xamarin Test Case
Comment 2 John Miller [MSFT] 2015-06-01 15:21:16 UTC
Created attachment 11410 [details]
Crash Log
Comment 3 Rolf Bjarne Kvinge [MSFT] 2015-06-02 04:07:56 UTC
The problem is that the peak memory usage is above what the device allows (not that the memory isn't collected when done with it).

Profiling with instruments yields this graph: https://www.dropbox.com/s/wg9qa8z6bclnu92/Screenshot%202015-06-02%2009.48.51.png?dl=0, which clearly shows the memory being freed.

The reason it works with newer devices, is that newer devices have more memory.

The Objective-C sample isn't identical, because strings are stored as UTF8 in Objective-C (so a 50mb text file will use 50mb of memory), while managed strings uses UTF16 (iow a 50mb text file uses 100mb of memory).

So for instance the (commented out) implementation using a byte array will use:
* 50mb for the byte array.
* 100mb for the string once converted.
* 100mb + 1 byte for the char array Console.WriteLine creates (it creates an array of chars and appends the newline character to that array before writing it).
* 100mb for an internal StringBuilder Xamarin.iOS uses when redirecting Console.WriteLine to the native NSLog (so that Console.WriteLines show up in the device log).
* 100mb when the above StringBuilder is converted to a string.

Our recommendation is to not use big chunks of memory, but instead process data in a streaming pattern, which avoids the whole problem.
Comment 4 David Ingham 2015-06-02 05:19:52 UTC
Forcing the compile to Armv7 + Armv7s (instead of Armv7 + ARM64) does replicate the problem on an iPad Air.

So question is how do we get this working on older devices?

>>  The reason it works with newer devices, is that newer devices have more memory.

INCORRECT. 4th Gen iPad and iPad Air both contain 1GB Memory.

>> The Objective-C sample isn't identical, because strings are 
>> stored as UTF8 in Objective-C (so a 50mb text file will use 
>> 50mb of memory), while managed strings uses UTF16 (iow a 
>> 50mb text file uses 100mb of memory).

Changing the native example to read it as UTF16 still has no negative effect to the native example.

>> Our recommendation is to not use big chunks of memory, 
>> but instead process data in a streaming pattern, which 
>> avoids the whole problem.

Even doing it with a substantially smaller buffer (I was just eliminating more memory being created for a string builder, memory re-allocation when building the string etc since 50 mb should more than comfortably get loaded in a device with 1gb memory.

All that aside it doesn't explain why using the simple core .net functions like: System.File.ReadAllText fails. Why does it success on attempt one... then sometimes fail at attempt two.... or three...  it smacks of something not freeing up properly between each run since the previous should not be out of scope.

I’m not happy with this being marked as resolved without first checking that a satisfactory resolution has been achieved!
Comment 5 Rolf Bjarne Kvinge [MSFT] 2015-06-02 09:17:37 UTC
(In reply to comment #4)
> Forcing the compile to Armv7 + Armv7s (instead of Armv7 + ARM64) does replicate
> the problem on an iPad Air.
> 
> So question is how do we get this working on older devices?
> 
> >>  The reason it works with newer devices, is that newer devices have more memory.
> 
> INCORRECT. 4th Gen iPad and iPad Air both contain 1GB Memory.

You're correct, I can reproduce in 32-bit mode on my iPad Air. The memory usage in Instruments doesn't look different between the 32-bit and 64-bit apps though, so this could be an iOS difference (maybe the 64-bit version allows more memory to be used).

> >> The Objective-C sample isn't identical, because strings are 
> >> stored as UTF8 in Objective-C (so a 50mb text file will use 
> >> 50mb of memory), while managed strings uses UTF16 (iow a 
> >> 50mb text file uses 100mb of memory).
> 
> Changing the native example to read it as UTF16 still has no negative effect to
> the native example.
> 
> >> Our recommendation is to not use big chunks of memory, 
> >> but instead process data in a streaming pattern, which 
> >> avoids the whole problem.
> 
> Even doing it with a substantially smaller buffer (I was just eliminating more
> memory being created for a string builder, memory re-allocation when building
> the string etc since 50 mb should more than comfortably get loaded in a device
> with 1gb memory.

Loading a 50mb byte array, isn't the problem, the problem is that you're not using 50mb, but a lot more (adding up the numbers I gave previously get you to 450mb + ~240mb of memory the native NSLog implementation requires to print the string to the device log [1]).

That's 690mb of memory. IMHO the amazing part is that this is working on 64-bit devices in the first place.

> All that aside it doesn't explain why using the simple core .net functions
> like: System.File.ReadAllText fails. Why does it success on attempt one... then
> sometimes fail at attempt two.... or three...  it smacks of something not
> freeing up properly between each run since the previous should not be out of
> scope.

The GC isn't 100% deterministic, you can't let a variable out of scope and then depend on the corresponding memory being freed immediately.

[1] I don't know how Apple implemented NSLog, but Instruments show two CFStrings of 44mb + 66mb + a malloc of 132mb.
Comment 6 David Ingham 2015-06-03 09:59:02 UTC
I’m still not happy with this, if I get rid of the console.writeline to get rid of the extra 100’s of mb memory used for that it still gets killed for memory consumption. Natively I can process a file of 120mb on the device and it’s just fine.

I’ve pinned down a boundary where the behaviour seems to come into play….. if I have a file that’s  33,554,432 bytes in length then it will read this into a string with 100 iterations. By increasing the file size by just one byte to 33,554,433 bytes it can’t even manage 5 iterations! I’ve updated the example here:

https://github.com/ingybing/LargeFile-Xamarin.git

This feels more like one you breach this boundary something is going wrong. Especially since it only seems to happen when running as a 32 bit build. If you run it as a 64 bit build its ok.