Bug 58343 - Visual Studio sometimes hangs (pauses, freezes) on `Xamarin.Messaging.Integration.BatchedObservableCollection<Xamarin.Messaging.Integration.Models.ServerData>.Find(System.Func<Xamarin.Messaging.Integration.Models.ServerData, bool> predicate)`
Summary: Visual Studio sometimes hangs (pauses, freezes) on `Xamarin.Messaging.Integra...
Status: VERIFIED FIXED
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: General ()
Version: 4.7.0 (15.4)
Hardware: PC Mac OS
: --- critical
Target Milestone: 15.4
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2017-07-24 21:26 UTC by Kyle White
Modified: 2017-09-29 06:55 UTC (History)
7 users (show)

Tags: AutomationDefect
Is this bug a regression?: Yes
Last known good build: d15-3


Attachments
Mini Dump (without heap) from VS2017. (4.65 MB, application/x-zip-compressed)
2017-07-24 21:30 UTC, Kyle White
Details
Mini Dump (without heap) from VS2015. (1.07 MB, application/x-zip-compressed)
2017-07-24 21:34 UTC, Kyle White
Details
Minidump without heap (4.30 MB, application/octet-stream)
2017-08-07 16:14 UTC, Kyle White
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Kyle White 2017-07-24 21:26:17 UTC
Visual Studio sometimes hangs (pauses, freezes) on `Xamarin.Messaging.Integration.BatchedObservableCollection<Xamarin.Messaging.Integration.Models.ServerData>.Find(System.Func<Xamarin.Messaging.Integration.Models.ServerData, bool> predicate)`

## Reproduction steps

This issue is cropping up on the functional automation tests very frequently starting last week. I haven't reproduced manually but it appears there is no pattern to when it happens. For example, two bots are currently stuck just after creating a tvOS app, and another bot is stuck while archiving an android app.

Although I don't have manual repro steps, if the stack/dump information is not enough to fix this issue, I can most likely provide access to a machine in the hung state (because it's happening so frequently in the auto runs).

## Except of Main Thread call stack 

>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.BatchedObservableCollection<Xamarin.Messaging.Integration.Models.ServerData>.Find(System.Func<Xamarin.Messaging.Integration.Models.ServerData, bool> predicate)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.ServerManager.Find(Xamarin.Messaging.Integration.Models.ServerData server, bool mustMatchHostName)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.ServerManager.AddOrUpdateServerInner.AnonymousMethod__0()	Unknown
>mscorlib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start<Xamarin.Messaging.Integration.ServerManager.(ref Xamarin.Messaging.Integration.ServerManager.<>c__DisplayClass23_0.<<AddOrUpdateServerInner>b__0>d stateMachine)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.ServerManager.AddOrUpdateServerInner.AnonymousMethod__0()	Unknown
>Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.ExecuteJob<Microsoft.VisualStudio.Threading.EmptyStruct>(System.Func<System.Threading.Tasks.Task> asyncMethod, Microsoft.VisualStudio.Threading.JoinableTask job)	Unknown
>Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.RunAsync(System.Func<System.Threading.Tasks.Task> asyncMethod, bool synchronouslyBlocking, Microsoft.VisualStudio.Threading.JoinableTaskCreationOptions creationOptions, System.Delegate entrypointOverride)	Unknown
>Merq.Async.Core.dll!Merq.AsyncManager.Run(System.Func<System.Threading.Tasks.Task> asyncMethod)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.ServerManager.AddOrUpdateServerInner(Xamarin.Messaging.Integration.Models.ServerData server)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.ServerManager.OnServiceFound(object sender, Xamarin.Messaging.Integration.ServerDiscoveryInfo args)	Unknown
>Xamarin.Messaging.Windows.dll!Xamarin.Messaging.Windows.Bonjour.BonjourDiscoveryService.OnResolved(System.Collections.Generic.IReadOnlyList<Zeroconf.IZeroconfHost> hosts)	Unknown
>Xamarin.Messaging.Windows.dll!Xamarin.Messaging.Windows.Bonjour.BonjourDiscoveryService.DiscoverHosts(object state)	Unknown

Full stack of main thread: https://gist.github.com/kdubau/32d1c9aeeeaebeacbe250a24bcdb3e1f

## More info

It feels like a deadlock, there are a ton of worker threads that look like this

>mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext)	Unknown
>mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)	Unknown
>mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
>mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
>mscorlib.dll!System.Threading.Tasks.Task.InternalWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
>mscorlib.dll!System.Threading.Tasks.Task.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken)	Unknown
>mscorlib.dll!System.Threading.Tasks.Task.Wait(System.TimeSpan timeout)	Unknown
>Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.WaitSynchronouslyCore(System.Threading.Tasks.Task task)	Unknown
>Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTaskFactory.WaitSynchronously(System.Threading.Tasks.Task task)	Unknown
>Microsoft.VisualStudio.Threading.dll!Microsoft.VisualStudio.Threading.JoinableTask.CompleteOnCurrentThread()	Unknown
>Merq.Async.Core.dll!Merq.AsyncManager.Run(System.Func<System.Threading.Tasks.Task> asyncMethod)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.Models.Model.OnPropertyChanged(string propertyName, bool onUIThread)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.Models.Model.OnPropertyChanged<bool>(System.Linq.Expressions.Expression<System.Func<bool>> propertyExpresion, bool onUIThread)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.Models.ServerData.IsDiscovered.set(bool value)	Unknown
>Xamarin.Messaging.Integration.dll!Xamarin.Messaging.Integration.ServerManager.OnServiceFound(object sender, Xamarin.Messaging.Integration.ServerDiscoveryInfo args)	Unknown
>Xamarin.Messaging.Windows.dll!Xamarin.Messaging.Windows.Bonjour.BonjourDiscoveryService.OnResolved(System.Collections.Generic.IReadOnlyList<Zeroconf.IZeroconfHost> hosts)	Unknown
>Xamarin.Messaging.Windows.dll!Xamarin.Messaging.Windows.Bonjour.BonjourDiscoveryService.DiscoverHosts(object state)	Unknown\

and a bunch more that look like the Main Thread stack

## Mini dumps

I will follow up with Mini dump attachements from both VS2017 and VS2015

## Environment

Here is my env info from VS2015 - however this can be reproduced in both VS2015 and VS2017 -

Microsoft Visual Studio Enterprise 2015
Version 14.0.25431.01 Update 3
Microsoft .NET Framework
Version 4.7.02046

Xamarin   4.7.0.820 (bb372d0)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android   7.5.0.0 (bbf378a)
Visual Studio extension to enable development for Xamarin.Android.

Xamarin.iOS   10.14.0.0 (38d3a2b)
Visual Studio extension to enable development for Xamarin.iOS.
Comment 1 Kyle White 2017-07-24 21:30:24 UTC
Created attachment 23763 [details]
Mini Dump (without heap) from VS2017.

This dump was capture while VS was hanging during a Android app archive. Screenshot: https://www.screencast.com/t/iCbMxlRNTTx
Comment 2 Kyle White 2017-07-24 21:34:33 UTC
Created attachment 23764 [details]
Mini Dump (without heap) from VS2015.

This dump was captured while VS was finishing creating/opening a new tvOS app.
Screenshot: https://www.screencast.com/t/c2xs32mq8
Comment 3 xamarin-release-manager 2017-07-26 17:27:16 UTC
Fixed in version 4.8.0.72 (master)

Author: Mauro Agnoletti
Commit: 75b9ca1559503032493b208be3f0c4c6de8ba8ee (xamarin/XamarinVS)
Comment 7 Kyle White 2017-08-07 16:13:38 UTC
I wouldn't say it's limited to UWP projects - I have the hang here which occurred while going through the motions to connect to the XMA agent:

The main thread call stack is in Xamarin.Messaging.Integration again which leads me to believe it's the same issue:
https://gist.github.com/kdubau/463a4c86645cc6ddb2bd6d83f441312d

XVS version 4.7.0.872 (a327999)
Comment 8 Kyle White 2017-08-07 16:14:49 UTC
Created attachment 24067 [details]
Minidump without heap

Minidump from hang in XVS 4.7.0.872 (a327999)
Comment 9 Kyle White 2017-08-08 18:30:52 UTC
Encountered this again in 4.7.0.874 (b4d02a0) - same call stack, let me know if you want more minidumps.
Comment 10 mag@xamarin.com 2017-08-11 21:28:58 UTC
2 more commits has been introduced, after the re opening, that fixes two other potential deadlocks found, besides the one fixed first in this bug. All of them were related to the Tools > Options > iOS Dialog, Bonjour Discovery and/or XMA connection flow.

I mark this issue as Resolved > Fixed since that from the following commits in master and d15-4, we covered all the stack traces and locks attached in this thread.

Please feel free to re open it if you still experience related issues:

master => cfc0b3333c875f420b1f7deafb783955502327d7 => https://wrench.internalx.com/Wrench/ViewLane.aspx?lane_id=3012&host_id=406&revision_id=908768

d15-4 => 13c2781a91a30990762625c9c826785c390dba3a => https://wrench.internalx.com/Wrench/ViewLane.aspx?lane_id=5156&host_id=406&revision_id=908770