Bug 60871 - CPU pegged @ 100% w/ a lot of TCP sockets in CLOSE_WAIT state
Summary: CPU pegged @ 100% w/ a lot of TCP sockets in CLOSE_WAIT state
Status: RESOLVED DUPLICATE of bug 60872
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: 5.4 (2017-06)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2017-11-27 12:09 UTC by Mindaugas
Modified: 2017-11-27 12:14 UTC (History)
3 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED DUPLICATE of bug 60872

Description Mindaugas 2017-11-27 12:09:33 UTC
Hi, 

Using the script below, a service that is managed by mono run-time starts using all the CPU, has 200+ CLOSE_WAIT tcp connections which do not close, and becomes non responsive. The issue starts with a sudden increase in CPU usage and thread count (thread count increases slower than CPU but the onset point is the same - same time), an increase in CLOSE_WAIT() sockets durring that time is present (if close() syscall is issued w/ gdb then the execution returns back to normal);


Script:
#!/bin/bash
IP=$1;
ab -n 10000000 -c 100 -k http://$IP:55558/resourceIdEncoder/encodeCustomerId?CustomerId=256 &> /dev/null &
PID=$(echo $!);
sleep 58.5;
kill -2 $PID;


Steps taken:
- Launch the service with mono;
- Execute the scipt (above) choosing that service as a target;
- Wait till the problem appears (~ 32-26 hours);

Actual: 
- CLOSE_WAIT socket count is 200+ (a sudden increase is noticeable);
- CPU is pegged @ 100%


Version: 
[root@xxx ~]# /root/mono-debug-install-dir/bin/mono-sgen -V
Mono JIT compiler version 5.4.1.6 (tarball Tue Nov 14 04:43:11 EST 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen (concurrent by default)


Thread info:
https://pastebin.com/raw/EmMTu0B5


GDB: info threads:
https://pastebin.com/raw/zLwXEedc


GDB: thread apply all bt:
https://pastebin.com/raw/0cJUw0Ha


GDB: mono_stacktrace 25 (for the Threadpool worker):
https://pastebin.com/raw/hteNxvCv
Comment 1 Mindaugas 2017-11-27 12:14:20 UTC

*** This bug has been marked as a duplicate of bug 60872 ***