Notice (2018-05-24): bugzilla.xamarin.com is now in
Please join us on
Visual Studio Developer Community and in the
Mono organizations on
GitHub to continue tracking issues. Bugzilla will remain
available for reference in read-only mode. We will continue to work
on open Bugzilla bugs, copy them to the new locations
as needed for follow-up, and add the new items under Related
Our sincere thanks to everyone who has contributed on this bug
tracker over the years. Thanks also for your understanding as we
make these adjustments and improvements for the future.
Please create a new report on
GitHub or Developer Community with
your current version information, steps to reproduce, and relevant error
messages or log files if you are hitting an issue that looks similar to
this resolved bug and you do not yet see a matching new report.
Created attachment 1084 [details]
Each threadpool thread can keep objects alive until more work is done by that thread. This is because the GC finds references to the objects on the stack (even if the code nulls out the reference there may be other temporaries on the stack created by gcc).
To reproduce: Get http://bugzilla.xamarin.com/attachment.cgi?id=181, and run the server project (TestTouchService) on a windows machine. Then get the client from here, and run it. Write the number of MB of data to get from the windows machine (25 seems to cause a decent spike in mem usage). When done, type GC a couple of times to run the GC. Note how the memory usage is still enormous compared to before downloading anything.
Created attachment 1085 [details]
Patch that helps
One solution (illustrated by the attached patch), is to refactor async_invoke_thread so that the objects are used in a separate method, so that when work is done and the thread is waiting for more work, the stack doesn't contain references to the objects anymore.
This patch improves memory usage (but doesn't quite fix it, that is likely another bug somewhere else though).
Created attachment 1086 [details]
- Right after startup: 8 MB
- Right after getting 25MB of data: 279 MB
- After running the GC a couple of times: 164 MB.
See attached screenshot from HeapShot, it shows inverse references for a StringBuilder that is kept alive by two AsyncResult and SocketAsyncResult instances (the <Misc Root> nodes are the stack referencing those objects).
With the patch from comment #1, the memory usage goes down to 71 MB after
running the GC.
Unfortunately the patch has other consequences (I can only run 1 request,
subsequent requests just sits there, doing nothing). This might be the symptom
of a more severe bug (depending on the threadpool threads to keep objects alive
on the stack... which will fail miserably with certain threadpool loads).
Lately we had weird memory problems when loading stuff asynchronously.
A series of operation that was taking a long time was moved from synchronous loading to asynchronous using BackgroundWorkers/NSThreads. When we moved to asynchronous we started receiving memory warnings and eventually crashes (when deploying on the device), whereas we don't have such problems when the operations are done synchronously on the main thread.
Could our issues be caused by the problem described in this case ?
Dynami, I do not know the problems you experienced are related to this bug, but I doubt it. Looks more like you did not register the threads (NSAutoReleasePool...).
Rolf, I guess you already know that running this server on Mono easily crashes (out of memory) due to XmlWriter leaking StringBuilders like a sieve...
Gonzalo, I actually never ran the server on Mono, I ran it on Windows :) In any case that sounds like another bug that should be fixed...
Rolf, I doubt the threadpool has anything to do with this problem. Your patch might fail because FILE is not thread-safe. We have never been good dealing with
large arrays and sys.xml has always been allocation-happy (huge byte array, then huge string...). Using large non-xml payloads with xsp works just fine and does NOT leak even after thousands of request have been made.
Gonzalo, I don't think you quite understood the problem. The issue is not leaking a bit in thousands of requests, the issue is leaking the last request (per threadpool thread). Doing another request will free the previous one (if it happens on the same threadpool thread).
The fact that non-xml payloads doesn't show the problem doesn't mean it isn't there - you can add counters to (Socket)AsyncResult's ctor and dtor and see that not all are freed after a request has finished. No need for any payload at all.
The FILE related parts of the patch are just debug-code, and they're obviously not meant to be part of the solution.
Created attachment 1120 [details]
epoll changes to 'Patch that helps'
Revert back the threadpool_init() line to its original version if needed.
Created attachment 1126 [details]
Patch that helps, v2
Second version of the threadpool patch, now it makes a difference on linux x86 too.
Unfortunately the regression still exists, but it's less reproducible.
Our iOS app is gobbling memory slowly but surely until eventually it's killed by the OS. This is basically the same code that already runs fine on other .net platforms and at this stage we suspect the Mono/MonoTouch platform, and this bug sounds suspicious as we do a lot of work on background threads.
Does this affect both Boehm and Sgen?
Tim: this particular bug wouldn't cause memory usage to grow slowly and steadily, it will keep exactly 1 object per thread-pool thread alive (and all the objects that one object references) - and only while that thread is waiting for more work. In other words it will leak if you do something on the threadpool once, but then stop using the threadpool, if you continuously keep using the threadpool you shouldn't notice anything.
If you think it is a bug in Mono/MonoTouch, don't hesitate to open another bug report or contact support where we can track the issue.
We don't use ThreadPool directly (though we do use threads extensively) however I understand that WebClient uses it, and we use WebClient a lot. Once data has been downloaded asynchronously with WebClient it stays in memory forever, which is a serious bug as it renders our software susceptible to be killed by the OS after initial setup which involves much downloading of data.
I have, however, created a separate bug for the WebClient issue. Someone commented on it that it might be this issue though.
Created attachment 1191 [details]
Disable the GC for threadpool threads while waiting for more work
Attached is another approach that works better: it disables the GC for threadpool threads while those threads are waiting for work.
With this patch no large objects survive for long in the test case provided previously.
Paolo, does the patch in the previous comment look ok?
(In reply to comment #15)
> Paolo, does the patch in the previous comment look ok?
Yes, please commit, thanks!
Thanks, committed to master (2d739f3, 8928c1b), 2.10 (b371946, 5092bb10) and mobile-master (851bbc06, eeb38948).