Bug 20031 - Hang (potential deadlock) when running unit tests on ASP.NET vNext projects
Summary: Hang (potential deadlock) when running unit tests on ASP.NET vNext projects
Status: RESOLVED FIXED
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: unspecified
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2014-05-25 15:55 UTC by David Fowler
Modified: 2014-06-04 19:41 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description David Fowler 2014-05-25 15:55:17 UTC
Using mono (master) from github:

mono --version

Mono JIT compiler version 3.4.1 (master/e1f9b1e Sat May 24 01:06:07 PDT 2014)


1. Install kvm.sh via the following command

curl https://raw.githubusercontent.com/graemechristie/Home/KvmShellImplementation/kvmsetup.sh | sh && source ~/.kre/kvm/kvm.sh && kvm upgrade

This should make k and kpm available on the shell

2. Clone the HttpAbstractions repository https://github.com/aspnet/HttpAbstractions

 git clone https://github.com/aspnet/HttpAbstractions.git

3. Navigate to the repository that was just cloned and type kpm restore. This should restore all of the nuget packages required.

4. Navigate to the test project for Microsoft.AspNet.FeatureModel.Tests:

    cd test/Microsoft.AspNet.FeatureModel.Tests

5. Run the "k test" command on that folder

   k test

This will cause the terminal to hang, even ctrl +  won't work.

To see more output, run export KRE_TRACE=1 before running k test. It will show a debug spew. It seems to hang while emitting the in memory assmembly for Microsoft.AspNet.FeatureModel:

 [RoslynAssemblyLoader]: Emitting assembly for Microsoft.AspNet.FeatureModel

It never returns from that point.
Comment 1 Ian Battersby 2014-05-25 19:21:20 UTC
Just to add I am experiencing a similar problem git aspnet/kruntime when executing test runs. Did a little digging with lldb but couldn't spot the issue, output/notes are in a GitHub gist here: https://gist.github.com/ianbattersby/f065e38a2a35e4d43697
Comment 2 Ian Battersby 2014-05-27 07:31:31 UTC
FYI; Trace file on Dropbox (too big too upload), might be of help: https://dl.dropboxusercontent.com/u/795371/k-trace.log.zip
Comment 3 David Fowler 2014-05-29 13:21:25 UTC
Here's another trace where it hangs:

https://gist.github.com/ianbattersby/17e8bb9019fc3d4e9db3

After doing some debugging last night we found the minimal set of things that makes it hang and it seems like it's something to do with closures + roslyn + mono + xunit (not sure if unit is relevant).
Comment 4 Ian Battersby 2014-05-29 15:34:07 UTC
Just to add to the comment above, here is the thread backtrace on another example (on a different test - https://github.com/aspnet/HttpAbstractions/blob/dev/test/Microsoft.AspNet.FeatureModel.Tests/InterfaceDictionaryTests.cs#L41) that shows the problem:

https://gist.github.com/ianbattersby/59285faaa33c164628a5

As David mentions there seems to be some issue with lexical scoping that exhibits itself in xUnit from within an Assert.Throws<>(() ..). For example this causes deadlock;

// Where FeatureObject is an empty class **in a different assembly**
var f = new FeatureObject();
Assert.Throws<ArgumentException>(() => { int x = f.GetHashCode(); });

Interestingly, in the case of InterfaceDictionaryTests.cs#L41, when you wrap everything in the closure it seems to work:

              Assert.Throws<ArgumentException>(() => {
                      var interfaces = new FeatureCollection();
                      var thing = new Thing();

                      interfaces.Add(typeof(IThing), thing);

                      interfaces.Add(typeof(IThing), thing);
                      });
Comment 5 Ian Battersby 2014-05-29 15:37:49 UTC
Should have made it clear, given the simplest example above, with everything captured inside the closure, it doesn't hang (albeit the test fails):

Hangs:
var f = new FeatureObject();
Assert.Throws<ArgumentException>(() => { int x = f.GetHashCode(); });

Doesn't Hang:
Assert.Throws<ArgumentException>(() => { int x = new FeatureObject().GetHashCode(); });
Comment 6 Miguel de Icaza [MSFT] 2014-05-30 16:03:24 UTC
It seems that this bug happens when Parallel.For is used.

David, I take it you have disabled Parallel.For by default?   It would be good if we can control this with some environment variable so you can let the users try things out now, but let us fix the bug without spending hours finding where to enable it again.
Comment 7 Rodrigo Kumpera 2014-06-04 15:41:03 UTC
Hey guys,

Could you provide a more dummy proof test case? Having so much plumbing happening behind the scenes make it very hard to troubleshoot.


Right now I just tried "kpm restore" and it's failing with the follow exception.
I have no idea if this is a mono regression of a problem with NuGet.


System.AggregateException:  ---> System.Xml.XmlException: Text node cannot appear in this state.  Line 1, position 1.
  at Mono.Xml2.XmlTextReader.ReadText (Boolean notWhitespace) [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.ReadContent () [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at Mono.Xml.XmlFilterReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.Linq.XDocument.ReadContent (System.Xml.XmlReader reader, LoadOptions options) [0x00000] in <filename unknown>:0 
  at System.Xml.Linq.XDocument.LoadCore (System.Xml.XmlReader reader, LoadOptions options) [0x00000] in <filename unknown>:0 
  at System.Xml.Linq.XDocument.Load (System.IO.TextReader textReader, LoadOptions options) [0x00000] in <filename unknown>:0 
  at System.Xml.Linq.XDocument.Load (System.IO.Stream stream) [0x00000] in <filename unknown>:0 
  at Microsoft.Framework.PackageManager.Restore.NuGet.PackageFeed+<_FindPackagesByIdAsync>d__1.MoveNext () [0x00000] in <filename unknown>:0 


Would it be possible to provide an archive with all files required to run the failing program?

In the meanwhile I'll keep poking at the above, I'll assume it's a regression on our end.
Comment 8 Rodrigo Kumpera 2014-06-04 15:55:46 UTC
Quick update, issue with one of my nuget sources.

It would be great if nuget was resilient to broken repos.
Comment 9 Rodrigo Kumpera 2014-06-04 16:12:00 UTC
So testing this with 3.4, it fails flat.

With our 3.6 internal preview it works perfectly fine for a hundred runs.

Marking this bug as fixed for now.
Comment 10 Ian Battersby 2014-06-04 19:41:12 UTC
NB: Having built and run against branch of 3.6 this seems fixed for HttpAbstractions, but kruntime is still hanging, with or without David's plinq disable on mono. Will speak with David but perhaps open a new bugzilla.