Bug 43462 - [regression] System.IO.IOException: Too many open files
Summary: [regression] System.IO.IOException: Too many open files
Status: VERIFIED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: io-layer ()
Version: 4.6.0 (C8)
Hardware: PC Mac OS
: --- major
Target Milestone: Future Cycle (TBD)
Assignee: Bugzilla
URL:
Depends on:
Blocks: 52545
  Show dependency tree
 
Reported: 2016-08-17 12:33 UTC by Sebastien Pouliot
Modified: 2017-04-07 06:36 UTC (History)
8 users (show)

Tags:
Is this bug a regression?: Yes
Last known good build: Mono 4.4


Attachments
test.cs (1.44 KB, text/plain)
2016-09-22 12:03 UTC, Rolf Bjarne Kvinge [MSFT]
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Sebastien Pouliot 2016-08-17 12:33:34 UTC
The `test-build-unit-test-dev` step from [1] started failing when the bots where updated to mono 4.6. Prior to that the same code worked fine with mono 4.4.

Here's the stack trace from the latest build [2]

[1:44:56] ulimit -n 96 && /Users/builder/data/lanes/3426/76197024/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/bin/mtouch --dev libs/bug-35786.app --abi armv7,arm64 libs/bug-35786.exe --sdkroot /Applications/Xcode8-beta6.app/Contents/Developer --sdk 10.0 -r:/Users/builder/data/lanes/3426/76197024/source/xamarin-macios/_ios-build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/lib/mono/Xamarin.iOS/Xamarin.iOS.dll
[1:44:56] Xamarin.iOS 9.99.3 using framework: /Applications/Xcode8-beta6.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS10.0.sdk
[1:45:10] error MT0000: Unexpected error - Please file a bug report at http://bugzilla.xamarin.com
[1:45:10] System.IO.IOException: Too many open files
[1:45:10]   at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00094] in /private/tmp/source-mono-4.6.0/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/System/System.Diagnostics/Process.cs:659 
[1:45:10]   at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x0012c] in /private/tmp/source-mono-4.6.0/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/System/System.Diagnostics/Process.cs:716 


I need to dig back to see why `ulimit -n 96` was added to the test. Consider this a warning that something changed and warn support (or mention it in the release notes) if it makes sense.


[1] https://wrench.internalx.com/Wrench/ViewTable.aspx?lane_id=3426&host_id=306

[2] https://wrench.internalx.com/Wrench/WebServices/Download.aspx?work_id=37233349&filename=test-build-unit-test-dev.log
Comment 1 Sebastien Pouliot 2016-08-17 12:40:18 UTC
Our test is related to https://bugzilla.xamarin.com/show_bug.cgi?id=35786 and I can set an higher limit.

OTOH

> @# ulimit -n=64 works while testing (with the fix for this bug), so use 96 to have some buffer.

that test did work, back in November 2015, using `64` and now fails at `96`, so maybe there's something wrong (or to document).
Comment 2 Sebastien Pouliot 2016-08-21 16:57:46 UTC
Changing ulimit to 128 still fails [1] and that used to work with 64. That looks bad for large customer applications.

[1] https://gist.github.com/spouliot/74a9ca6307256aebf7ea9e7b2ae36859
Comment 3 Sebastien Pouliot 2016-08-21 17:02:59 UTC
It does work with (default) 256 but fails with 192. 

https://gist.github.com/spouliot/d410e6548bd6c801943d698f4a2b878a

So running mtouch on mono 4.6 seems to require more than 3x the open files than 4.4 does.
Comment 4 Rolf Bjarne Kvinge [MSFT] 2016-08-22 09:56:38 UTC
This smells like a file descriptor leak inside mono somewhere.
Comment 5 Rodrigo Kumpera 2016-08-22 15:22:19 UTC
Hey Ludo,

This looks like your area.
Comment 6 Ludovic Henry 2016-08-22 19:04:49 UTC
Looks like a file descriptor leak, @rolf you are right.

IMO, there can be 2 reasons:
 - the application/bcl opens handles (via FileStream, Socket or Pipes) but do not Dispose them explicitly (most likely explanation)
 - the runtime effectively leaks handle

To help debug it, could you run it with `MONO_LOG_MASK=io-layer MONO_LOG_LEVEL=debug` ? That should print information about the handles that are allocated.
Comment 7 Sebastien Pouliot 2016-08-23 17:48:39 UTC
@Ludovik considering it worked fine with earlier versions of Mono, including 4.4.x, I think it's safe to assume #2 (leak).

I stopped the log after 202936466 bytes because I'm not sure you want them.

Might be easier to download the current XI .pkg (no build time required) from master, cycle8 (Xcode 7.3) or xcode8 (Xcode 8 beta).

Checkout xamarin-macios (master will do)

Then run `make` from xamarin-macios/tests/scripted/bug-35786

using mono 4.6 (and compare to 4.4).
Comment 8 Sebastien Pouliot 2016-08-23 19:27:50 UTC
https://github.com/mono/mono/pull/3426

^ that fix might be related or just a similar issue
Comment 9 Rolf Bjarne Kvinge [MSFT] 2016-09-22 12:03:40 UTC
Created attachment 17672 [details]
test.cs

Adding self-contained test case.

Execute:

    mcs test.cs
    ulimit -n 96
    mono test.exe

Result:

    [...]
    Iteration #44
    
    Unhandled Exception:
    System.IO.IOException: Too many open files
      at System.Diagnostics.Process.CreatePipe (System.IntPtr& read, System.IntPtr& write, System.Boolean writeDirection) [0x00014] in <affe4060066c42de8cdd6027cdb92b56>:0 
      at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x00154] in <affe4060066c42de8cdd6027cdb92b56>:0 

Workaround: run the GC for every iteration.
Comment 10 Ludovic Henry 2016-09-23 10:39:10 UTC
This is an expected leak: when calling Dispose on a Process (via the using), we do not close the StandardInput, StandardOutput and StandardError FileStream. That happens in the referencesource code, so I am going to mark this bug as "not fixing".

The fix as @rolf suggests in the test case is to run the GC, as it's going to make sure that the FileStream that are not referenced anymore, are being closed, and thus freeing the corresponding fd.
Comment 11 Rolf Bjarne Kvinge [MSFT] 2016-09-23 10:52:55 UTC
This needs to be documented as a functional breaking change then, because previously it was not necessary to manually close the streams.

On Windows it's probably not an issue because there's no limit on the number of open files.
Comment 12 Ludovic Henry 2016-09-23 12:50:03 UTC
@kumpera any suggestion on how we should document that?
Comment 13 Rodrigo Kumpera 2016-09-26 20:15:25 UTC
@ludovic, add to the release notes and ping Adrian about including that small piece on the C8 known issues.
Comment 14 Rolf Bjarne Kvinge [MSFT] 2016-09-29 14:28:16 UTC
I'm reopening this, since for the test sample the _only_ solution is to run the GC, and IMHO that's not an acceptable solution.

If I try to close standard output like this (https://gist.github.com/rolfbjarne/ce764e541799c9f0ac8392418a2175be):

    p.StandardOutput.Close ();

I get:

    System.InvalidOperationException: Cannot mix synchronous and asynchronous operation on process stream.
Comment 15 ben.mcmill 2017-03-29 00:42:09 UTC
I get the same error: 
    System.InvalidOperationException: Cannot mix synchronous and asynchronous operation on process stream.

Any idea on when this will be fixed, or how to handle in the meantime?
Comment 16 Rolf Bjarne Kvinge [MSFT] 2017-03-29 08:30:26 UTC
@Ben, we call GC.Collect to work around it: https://github.com/xamarin/xamarin-macios/blob/9f0fd32330e127b303874121f96c9e7d737c4297/tools/common/Driver.cs#L213
Comment 17 Rodrigo Kumpera 2017-04-06 20:41:18 UTC
We fixed that on C9.
Comment 18 Rolf Bjarne Kvinge [MSFT] 2017-04-07 06:36:37 UTC
The test case from comment #9 works with "ulimit -n 8" now.

> mono --version
Mono JIT compiler version 5.0.0.18 (2017-02/9ed0907 Mon Mar 27 19:45:32 EDT 20