Bug 37950 - Mono using a lot more memory than .NET when zipping a file using ZipArchive
Summary: Mono using a lot more memory than .NET when zipping a file using ZipArchive
Status: RESOLVED FIXED
Alias: None
Product: Class Libraries
Classification: Mono
Component: General ()
Version: 4.2.0 (C6)
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Marek Safar
URL:
Depends on:
Blocks:
 
Reported: 2016-01-23 02:16 UTC by Jon Goldberger [MSFT]
Modified: 2017-05-18 13:12 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Jon Goldberger [MSFT] 2016-01-23 02:16:22 UTC
## Description

With a simple Console app using the following code:

>using (Stream outStream = new FileStream("../../Out.zip", FileMode.Create, FileAccess.Write)){
>	using (ZipArchive archive = new ZipArchive(outStream, ZipArchiveMode.Create, true))
>	{
>		using (Stream inStream = new FileStream("../../IMG_0099.MOV", FileMode.Open, FileAccess.Read))
>		{
>			ZipArchiveEntry entry = archive.CreateEntry("IMG_0099.MOV");
>			using (Stream entryStream = entry.Open())
>			{
>				inStream.CopyTo(entryStream);
>			}
>		}
>	}
>}

running from XS on Mac the process uses about 385 MB, whereas the same code run in Visual Studio with .NET uses about 3.6 MB.

The test project has a .MOV file in it that is 116 MB. I  observed the memory usage for the process in Activity Monitor on my Mac and in Task Manager on Windows. 

Comment from the reporting customer:
"My users are reporting out of memory crashes when compressing data. I investigated it and found that when compressing data with System.IO.Compression.ZipArchive in Xamarin, the memory usage is proportional to the size of the data being compressed.  For example if I want to compress a 200MB file, the runtime consumes in excess of 400MB of RAM. Microsoft's implementation of System.IO.Compression.ZipArchive uses constant amount of memory irrespective of compressed data size. This is correct behavior, because the deflate algorithm used by zip format only requires a fixed amount of memory. There must be an issue in Xamarin version of ZipArchive that needlessly allocates memory proportionally to input data size. Here's sample code exhibiting large memory usage (if "In.dat" is 200MB, it will use about 400MB of memory)"

## Steps to reproduce

Test project (too big too attach due to included movie file for zipping purposes): 
https://www.dropbox.com/s/psrdv1sj0f7zjzt/TestZipMemoryUsage.zip?dl=0

1. To first observe memory usage in .NET, open the test project in Visual Studio (It's a console app) and run it.

2. Open Task Manager and observe the memory used by the process. It should show up in Task Manager as vshost32.exe  (32bit). On my end memory usage was about 3.6 MB

3. Now open the project in Xamarin Studio on a Mac and run the project. 

4. Open Activity Monitor->Memory tab and observe the memory used by the process. It will show up as process mono-sgen, but there may be more than one process with that name. To make sure you are observing the correct process, get the PID of the process in Terminal, so open Terminal if need be and enter:
ps -ax | grep mono-sgen
and look for the process that ends with "TestZipMemoryUsage.exe" and note the PID for that process and compare to the PID in Activity Monitor.


Expected result: process memory usage will be about the same as in .NET

Actual result: process using much more memory than with .NET.


## My environment

Mac:

=== Xamarin Studio ===

Version 5.10.1 (build 6)
Installation UUID: 964c531b-d928-456b-a9ae-e1f82266b360
Runtime:
	Mono 4.2.1 (explicit/6dd2d0d)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 402010102

=== Xamarin.Profiler ===

Version: 0.22.0.0
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Xamarin.Android ===

Version: 6.0.0.34 (Business Edition)
Android SDK: /Users/apple/Library/Developer/Xamarin/android-sdk-mac_x86
	Supported Android versions:
		4.0.3 (API level 15)
		4.1   (API level 16)
		4.2   (API level 17)
		4.3   (API level 18)
		4.4   (API level 19)
		5.0   (API level 21)
		5.1   (API level 22)
		6.0   (API level 23)

SDK Tools Version: 24.4.1
SDK Platform Tools Version: 23.1
SDK Build Tools Version: 23.0.2

Java SDK: /usr
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

=== Xamarin Android Player ===

Version: 0.6.5
Location: /Applications/Xamarin Android Player.app

=== Apple Developer Tools ===

Xcode 7.2 (9548)
Build 7C68

=== Xamarin.iOS ===

Version: 9.4.0.0 (Business Edition)
Hash: 7322991
Branch: master
Build date: 2015-12-08 16:20:29-0500

=== Xamarin.Mac ===

Version: 2.4.0.109 (Business Edition)

=== Xamarin Inspector ===

Version: 0.3.2.3
Hash: 1b526e6
Branch: master
Build date: Tue Nov 17 20:54:30 UTC 2015

=== Build Information ===

Release ID: 510010006
Git revision: 0b60eecdb531933734519c13257d16a780274aab
Build date: 2015-12-04 20:28:20-05
Xamarin addins: 9876fd7c9837977178411ec7375b4352c0a0d6af
Build lane: monodevelop-lion-cycle6-baseline

=== Operating System ===

Mac OS X 10.11.2
Darwin Jons-iMac.local 15.2.0 Darwin Kernel Version 15.2.0
    Fri Nov 13 19:56:56 PST 2015
    root:xnu-3248.20.55~2/RELEASE_X86_64 x86_64


Windows:

Microsoft Visual Studio Professional 2015
Version 14.0.23107.0 D14REL
Microsoft .NET Framework
Version 4.6.00079

Installed Version: Professional

Visual Basic 2015   00322-40000-00000-AA850
Microsoft Visual Basic 2015

Visual C# 2015   00322-40000-00000-AA850
Microsoft Visual C# 2015

Visual C++ 2015   00322-40000-00000-AA850
Microsoft Visual C++ 2015

Visual F# 2015 RC   00322-40000-00000-AA850
Microsoft Visual F# 2015 RC

Windows Phone SDK 8.0 - ENU   00322-40000-00000-AA850
Windows Phone SDK 8.0 - ENU

Application Insights Tools for Visual Studio Package   1.0
Application Insights Tools for Visual Studio

ASP.NET and Web Tools   14.0.20626.0
ASP.NET and Web Tools

ASP.NET Web Frameworks and Tools 2013   5.2.30624.0
For additional information, visit http://www.asp.net/

Common Azure Tools   1.5
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

GenerateUnitTest   1.0
Generates unit test code for methods in classes under test.

GitHub.VisualStudio   1.0
A Visual Studio Extension that brings the GitHub Flow into Visual Studio.

Microsoft Azure Mobile Services Tools   1.4
Microsoft Azure Mobile Services Tools

NuGet Package Manager   3.2.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

PreEmptive Analytics Visualizer   1.2
Microsoft Visual Studio extension to visualize aggregated summaries from the PreEmptive Analytics product.

SQL Server Data Tools   14.0.50616.0
Microsoft SQL Server Data Tools

Visual Studio Tools for Universal Windows Apps   14.0.23309.00 d14oob
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.

Xamarin   4.0.0.1717 (1390b70)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android   6.0.0.35 (d300845)
Visual Studio plugin to enable development for Xamarin.Android.

Xamarin.iOS   9.3.99.33 (ea30b32)
Visual Studio extension to enable development for Xamarin.iOS.
Comment 1 Jon Goldberger [MSFT] 2016-01-23 02:19:58 UTC
Additional comment from reporting customer:

"This is a problem in Mono
libraries also present on desktop (not only mobile) and seems to be
connected to internal use of MemoryStreams to cache the data."
Comment 2 Rodrigo Kumpera 2016-01-25 19:43:53 UTC
Hey João,

Can you take a look at this?
Comment 3 kalita 2016-02-23 11:53:18 UTC
Is there any progress on this. We are going to release an update for our app soon and this is blocking us. The only way is to go back to an external zip implementation like SharpZip.
Comment 4 João Matos 2016-02-23 12:13:14 UTC
Not yet, but I'll start looking if I can improve things here.
Comment 5 kalita 2016-03-16 15:51:11 UTC
I see almost another month has passed and nothing is happening.
Comment 6 Alexander Köplinger [MSFT] 2017-05-16 16:10:40 UTC
(moving to correct Class Libraries bugzilla component)

@João: did you end up making changes here?
Comment 7 Marek Safar 2017-05-18 13:12:56 UTC
This issue is already fixed in Mono 5.0. Compress 1GB file takes about 4MB of memory right now