Bug 13390 - TestRunner Does Not Execute TestFixtureSetUp/TearDown When Running Individual Tests
Summary: TestRunner Does Not Execute TestFixtureSetUp/TearDown When Running Individual...
Status: VERIFIED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: Tools ()
Version: 6.2.x
Hardware: All Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Sebastien Pouliot
URL:
Depends on:
Blocks:
 
Reported: 2013-07-22 17:09 UTC by Ben Rogers
Modified: 2014-06-04 10:10 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Ben Rogers 2013-07-22 17:09:35 UTC
The Xamarin.iOS TestRunner allows developers to execute tests individually. However, when doing so, the TestRunner does not invoke the TestFixtureSetUp/TearDown methods. The TestFixtureSetUp/TearDown methods are invoked, however, when executing all tests (i.e. when clicking "Run Everything" button).

As the TestRunner uses NUnitLite, and NUniteLite is based on NUnit, I believe this page describes the expected behavior:

  http://www.nunit.org/index.php?p=fixtureSetup&r=2.2.10

Specifically:

> This attribute is used inside a TestFixture to provide a single
> set of functions that are performed once prior to executing any
> of the tests in the fixture.

Note the use of the word "any" (as opposed to "all").

The following code demonstrates the bug. When running "everything", the test passes; when running the individual test, the test fails.

> using System;
> using NUnit.Framework;
> 
> namespace TouchRunnerTest
> {
>     [TestFixture]
>     public class Tests
>     {
>         private bool _setUp = false;
> 
>         [TestFixtureSetUp]
>         public void SetUp()
>         {
>             _setUp = true;
>         }
> 
>         [TestFixtureTearDown]
>         public void TearDown()
>         {
>             _setUp = false;
>         }
> 
>         [Test]
>         public void Test()
>         {
>             Assert.IsTrue(_setUp);
>         }
>     }
> }

The test SetUp/TearDown attributes can be used as a workaround. However, this is less than ideal in some scenarios, particularly when performing expensive integration tests.
Comment 1 mlpstingray 2013-09-12 11:35:05 UTC
Same problem on Android, if I position myself inside a TextFixture (in the Xamarin.Android.NUnitLite UI) and click on Run Tests, it is not run making everything harder to debug and isolate. Very annoying.
Comment 2 Sebastien Pouliot 2013-10-05 10:13:59 UTC
Fixed in Touch.Unit / master 6cbc618515fcdce461c0a3904bd0014b35d230c6

@mlpstingray you need to file a separate bug report, the iOS and Android runners are different applications.
Comment 3 Atsushi Eno 2013-12-11 07:36:17 UTC
Copy of my comment on #16621:

Have you asked about the issue to NUnitLite developers? If this is the expected
behavior, it should rather be NUnitLite itself but not Touch.Unit or
Xamarin.Android.NUnitLite that should "fix" the issue. Otherwise it brings
*inconsistency* between uses of NUnitLite in other environments and that in
Xamarin.

According to the committed changes, it seems that the fix is rather "hacked" in
Touch.Unit. Upgrading to NUnitLite 1.0.0 did not fix anything on Android.

I think NUnitLite developers rather defined the behavior of [TestFixtureSetup]
and [TestFixtureTearDown] to work like that, because their implementation
explictly creates their "OneTimeSetup" internal test runner commands in
TestSuite.
http://bazaar.launchpad.net/~nunitlite-dev/nunitlite/trunk/view/head:/src/framework/Internal/Tests/TestSuite.cs#L180
Comment 4 Sebastien Pouliot 2013-12-11 09:08:32 UTC
> Have you asked about the issue to NUnitLite developers? 

No I took the lazy approach and tried it myself.

* There's no "official" NUnitLite runner GUI, except the one we provided with XI and XA, that allow you to run a single test. 

* The console runner (for the full nunit) does allow "single test execution". 

> it should rather be NUnitLite itself

No. The fact is that Touch.Unit, not NUnitLite, support this "single test execution" feature so I don't expect the library to take care of this for me. If you copied Touch.Unit "hacked" feature then you're in the same situation.

> Otherwise it brings *inconsistency*

Here's how NUnit behave for consistency purpose:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using NUnit.Framework;

namespace nunit3test {
	[TestFixture]
	public class Class1 {

		bool setup;

		[TestFixtureSetUp]
		public void Setup ()
		{
			setup = true;
		}

		[TestFixtureTearDown]
		public void TearDown ()
		{
			Assert.False (setup);
		}

		[Test]
		public void Test1 ()
		{
			Assert.True (setup);
			setup = false;
		}

		[Test]
		public void Test2 ()
		{
			Assert.True (setup);
			setup = false;
		}
	}
}

compile (w/csc) and execute like this:

> nunit-console.exe -framework=net-4.5 Class1.dll -test=nunit3test.Class1.Test1

and you'll get only one test has executed with no failure.

> Selected test(s):
>    nunit3test.Class1.Test1
>
> Test Run Summary -
>    Overall result: Passed
>    Tests run: 1, Errors: 0, Failures: 0, Inconclusive: 0
>      Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
>         Time: 0 seconds

TestResult.xml will show that Test1 passed successfully - which can only happens if the Setup method was called when a single test was selected.
Comment 5 Ram Chandra 2014-06-04 10:10:10 UTC
I have checked this issue and I observed that the "TestRunner" executes "TestFixtureSetUp/TearDown" when I run the individual test.

Screencast: http://www.screencast.com/t/aYveCR6GY407

This issue has been fixed. Hence closing this issue.

Environment Info:

=== Xamarin Studio ===

Version 5.0 (build 878)
Installation UUID: 6ea47b0d-1852-4aaf-808d-373ff0a5002b
Runtime:
	Mono 3.4.0 ((no/63569a7)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 304000205

=== Xamarin.Android ===

Version: 4.12.4 (Trial Edition)
Android SDK: /Users/jatin66/Desktop/Backup/android-sdk-macosx
	Supported Android versions:
		1.6   (API level 4)
		2.1   (API level 7)
		2.2   (API level 8)
		2.3   (API level 10)
		3.1   (API level 12)
		3.2   (API level 13)
		4.0   (API level 14)
		4.0.3 (API level 15)
		4.1   (API level 16)
		4.2   (API level 17)
		4.3   (API level 18)
		4.4   (API level 19)
Java SDK: /usr
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

=== Apple Developer Tools ===

Xcode 5.1 (5084)
Build 5B130a

=== Xamarin.iOS ===

Version: 7.2.2.2 (Business Edition)
Hash: db4427f
Branch: 
Build date: 2014-04-22 12:49:14-0400

=== Xamarin.Mac ===

Xamarin.Mac: 1.8.0.8

=== Build Information ===

Release ID: 500000878
Git revision: bcd66920d963483e7d638a2339c4022fe035b529
Build date: 2014-05-27 17:36:26-04
Xamarin addins: da9064ce55b0fa90930a7c437a4cc1ae0e5c778c

=== Operating System ===

Mac OS X 10.9.2
Darwin Jatin66s-iMac.local 13.1.0 Darwin Kernel Version 13.1.0
    Thu Jan 16 19:40:37 PST 2014
    root:xnu-2422.90.20~2/RELEASE_X86_64 x86_64