Bug 42717 - MSBuild orphaned
Summary: MSBuild orphaned
Status: VERIFIED FIXED
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: iOS ()
Version: 4.1.0 (C7)
Hardware: PC Windows
: Normal normal
Target Milestone: 15.4
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-07-21 16:33 UTC by Dan Siegel
Modified: 2018-02-13 15:38 UTC (History)
15 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
MSBuild processes running amok (18.17 KB, image/png)
2017-03-28 05:55 UTC, Kent
Details
CPU usage (33.28 KB, image/png)
2017-03-28 05:55 UTC, Kent
Details
Attached IDE logs (7.14 KB, application/x-zip-compressed)
2017-08-02 12:32 UTC, Aman Dharwal
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Dan Siegel 2016-07-21 16:33:30 UTC
When building my Xamarin projects in Visual Studio I am noticing a number of instances of MSBuild that are orphaned in the background. Even closing Visual Studio does not solve the issue. I must manually kill the MSBuild processes that are orphaned or just reboot.

This creates a particular problem when the MSBuild process still has file locks on certain files. The easiest to duplicate is with Xamarin Forms. After building a couple of times try to upgrade to a new Xamarin Forms package, it won't be able to update the package for your solution even after restarting Visual Studio, until all MSBuild processes have been killed.
Comment 2 Daniel Cazzulino 2016-08-10 20:11:06 UTC
We have some additional checks in the current alpha drop (4.2.x) which you can try. 

Please do let us know if it solves the issue. Otherwise, we can investigate further.

Thanks!
Comment 3 xamarin-release-manager 2016-08-11 12:20:11 UTC
Fixed in version 99.0.0.3334 (master)

Author: Daniel Cazzulino
Commit: e2b40db0139bd1424bc4408af1f924250c98b276 (xamarin/XamarinVS)
Comment 4 Dan Siegel 2016-08-11 15:08:49 UTC
Sounds great. I will install that and test this weekend.
Comment 5 xamarin-release-manager 2016-08-18 15:32:04 UTC
Fixed in version 4.2.0.541 (cycle8)

Author: Daniel Cazzulino
Commit: d72202ea6c9bc44ae36ee8b3b00611998db63851 (xamarin/XamarinVS)
Comment 6 Ben Beckley 2016-08-25 16:22:17 UTC
I am still hitting this after a run of about 50 tests - a mixture of Android, iOS, and CrossPlatform template building tests. Each test opens a new instance of Visual Studio. The VS instances are closed via File>Exit in the tear down, and the end of the will kill the devenv process in the case that File>Exit doesn't work.

This was using 4.2.0.584 (c8 beta 2)

Here is a portion of the orphaned processes
http://www.screencast.com/t/HUF9w3ZO
Comment 7 Chase Long 2017-03-28 04:50:11 UTC
I have this problem when trying to launch a Xamarin.Forms project on an Android emulator. The solution builds, but then the status at the bottom of the Visual Studio window changes to "Deploying..." and never progresses. If I cancel the build and kill any orphaned MSBuild processes, then it works. I believe the orphaned processes are from a previous debug run that I quit using the red square stop button in VS.

This is on Visual Studio 2015 with Xamarin 4.3.0.789 (the VS extension) and Xamarin.Android 7.1.0.43 (the VS extension). I don't have a sample solution. At least one other developer at my company has this problem.
Comment 8 Kent 2017-03-28 05:07:07 UTC
I get this all the time, latest stable (cycle 9). My CPU ends up nearly pegged, mostly "system" usage, not user.

I've been relying on this in my PS profile for quite some time:

    # Kill all MSBuild processes (because Xamarin)
    Function KillMSBuild
    {
        Get-Process | Where-Object { $_.Name -eq "msbuild" } | Stop-Process
    }

I just run this every now and then (obviously when there's no actual build taking place) and my CPU usage drops back to nothing.

What info can I provide to help get this fixed? It's driving me nuts.

It especially seems to be exacerbated by switching startup projects between iOS and Android in my solution.
Comment 9 Kent 2017-03-28 05:55:21 UTC
Created attachment 20927 [details]
MSBuild processes running amok

Here is what I typically see in procexp when my CPU is burning.
Comment 10 Kent 2017-03-28 05:55:43 UTC
Created attachment 20928 [details]
CPU usage

Notice the CPU usage is mostly system.
Comment 11 mag@xamarin.com 2017-03-28 15:12:22 UTC
It would be helpful to determine if the MSBuild issue manifests with iOS builds, Android builds, or both. Previous comments mentioned that it was an issue with Android on a Forms app, and new comments that it mostly happens when switching from iOS to Android.
This kind of information would be helpful to detect faster on which targets we could be still letting instances alive.
Comment 12 Kent 2017-03-29 00:04:28 UTC
All I have to do is open my solution and problems begin. I get a single MSBuild process, but it chews up more than 10% CPU and never stops doing so.

I'll keep an eye on when more processes spawn throughout my day today and report back.
Comment 13 Kent 2017-03-29 08:43:34 UTC
A couple more MSBuild processes spawn on first build (Android, device). The new processes bounce around with CPU usage during the build, then drop back to nothing once done. The original process continues to use 10-15% CPU throughout build and once build completes. This pattern continued for several Android builds.

Switching startup project to iOS had no immediate effect on MSBuild processes. However, upon building and deploying (simulator), the original process seems to consume even more CPU (consistenly at 15% now) and one of the processes spawned during the Android build has now also shot up to 15% consumption.

Had to close VS to switch branches. Re-opened with iOS as startup project. No MSBuilds started until I built. This time I ran into a (possibly related) problem where two of three MSBuild processes were using ~15% and VS stopped responding with "Launching Application for debugging..." in the output panel. I had to kill devenv and MSBuild processes and start again.

This time it worked. However, two of three MSBuild processes were then sitting at ~15%.

Then after another build, all three were consuming ~18%. It's around this point I kinda need to kill all MSBuild processes because my total CPU consumption across all cores is ~75%, more than half of which is system, not user.
Comment 14 Kent 2017-04-06 04:53:58 UTC
Can I have an update on this issue please? It's absolutely killing me on iOS in particular. Every build I have to kill all MSBuild processes.
Comment 15 John Miller [MSFT] 2017-04-10 17:41:04 UTC
@Kent,


Non-engineering view note:

I tried to reproduce this in VS 2017. I'll attach my version info below. I was unable to get even 1 additional MSBuild.exe process to spawn. I tried building iOS/Android projects and deploying them. I always ended up with 1 MSBuild.exe process and closing VS caused it to go away. I'll continue to try and reproduce this. Can you share your version information?

Microsoft Visual Studio Enterprise 2017
Version 15.1 (26403.0) Release
VisualStudio.15.Release/15.1.0+26403.0
Microsoft .NET Framework
Version 4.6.01586

Installed Version: Enterprise

Xamarin   4.4.0.34 (3f99c5a)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android SDK   7.2.0.7 (b16fb82)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK   10.8.0.174 (7656cc6)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
Comment 16 Kent 2017-04-11 00:42:10 UTC
Thanks John. Yeah, I figured this wasn't going to be easily reproducible. If it was, you'd have a million people jumping up and down about this.

Here's my version info:

Microsoft Visual Studio Enterprise 2017
Version 15.1 (26403.0) Release
VisualStudio.15.Release/15.1.0+26403.0
Microsoft .NET Framework
Version 4.6.01586

Installed Version: Enterprise

Visual Basic 2017   00369-60000-00001-AA208
Microsoft Visual Basic 2017

Visual C# 2017   00369-60000-00001-AA208
Microsoft Visual C# 2017

Visual C++ 2017   00369-60000-00001-AA208
Microsoft Visual C++ 2017

Visual F# 4.1   00369-60000-00001-AA208
Microsoft Visual F# 4.1

Add New File   3.5
The fastest and easiest way to add new files to any project - including files that start with a dot

ASP.NET and Web Tools 2017   15.0.30320.0
ASP.NET and Web Tools 2017

ASP.NET Web Frameworks and Tools 2017   5.2.50303.0
For additional information, visit https://www.asp.net/

Azure App Service Tools v3.0.0   15.0.30209.0
Azure App Service Tools v3.0.0

Common Azure Tools   1.9
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

File Icons   2.6
Adds icons for files that are not recognized by Solution Explorer

HideMenu   1.0
Hides the Visual Studio main menu, similar to Windows Explorer and Internet Explorer

ILSpy.AddIn   1.0
Integration of the ILSpy Decompiler into Visual Studio.

JavaScript Language Service   2.0
JavaScript Language Service

JavaScript Project System   2.0
JavaScript Project System

JavaScript UWP Project System   2.0
JavaScript UWP Project System

KofePackagePackage Extension   1.0
KofePackagePackage Visual Studio Extension Detailed Info

Merq   1.1.17-rc (cba4571)
Command Bus, Event Stream and Async Manager for Visual Studio extensions.

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual Studio VC Package   1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio   Mono.Debugging.VisualStudio
Support for debugging Mono processes with Visual Studio.

NCrunch   
Continuous Testing Tool for .NET
Copyright © 2010-2016 Remco Software Ltd

NuGet Package Manager   4.1.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

Rebracer   1.0
Saves editor formatting settings as part of each solution.

SQL Server Data Tools   15.1.61702.140
Microsoft SQL Server Data Tools

Trailing Whitespace Visualizer   2.5.83
Keeps your code files clean by making it easier than ever to identify and remove any trailing whitespace

TypeScript   2.1.5.0
TypeScript tools for Visual Studio

Visual Studio Tools for Universal Windows Apps   15.0.26403.00
The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.

VSColorOutput   2.5
Color output for build and debug windows - http://mike-ward.net/vscoloroutput

Xamarin   4.4.0.34 (3f99c5a)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android SDK   7.2.0.7 (b16fb82)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK   10.8.0.174 (7656cc6)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
Comment 17 Kent 2017-04-20 01:32:42 UTC
ping. Still no movement on this?
Comment 18 Vlad 2017-04-26 04:49:21 UTC
Same here on few laptops when iOS project is build, and its very easy on our side to reproduce.

Environment:
  -Windows 10
  -VS 2015
  -Xamarin latest stable version

Every time when iOS project is build (is not reproducible with Android projects) a new instance or multiple instances of MSBuild.exe are created and aren't closed until manually are killed, even with one MSBuild instance my CPU gets crazy going up to 100% and stays there until manually is not killed. This issue kills our desire to develop anything on iOS and its there for months.
Comment 19 Chase Long 2017-05-10 18:55:49 UTC
This seemed to be fixed for me (I posted earlier in this thread) after switching to VS2017 about two months ago, but I am seeing it again as of today's VS2017 update, "Version 15.2 (26430.4) Release". The MSBuild.exe processes close when I close VS2017, but I have to manually kill them via Task manager after building so that they don't occupy .dlls from the previous build's output.

Windows 10
Xamarin 4.5.0.443 (c871575)
Comment 20 Nico Vermeir 2017-05-11 08:34:39 UTC
seeing the same thing after this morning's update with a classic Xamarin (ios / android) solution. build solution throws the error, rebuild solution seems to work
Comment 21 Kent 2017-05-22 10:36:53 UTC
Wow. How is this issue languishing so?
Comment 22 ilya.golovach 2017-06-01 06:44:09 UTC
Same issue here. Latest VS2017 and Xamarin, Win 10. Valid at least for Xamarin.Forms. The amount of leftover msbuild process is random, but typically 2-3, sometimes 4. This may cause another issue (though I am not absolutely sure) when you just hit the Run button to run the app, but the build process will not finish with the error message that one of the .dll (typically the dll for my app) cannot be moved/copied. Sometimes, explicitly triggering rebuild fixes this, sometimes you need to do clean and than rebuild, in worse cases - you need to restart the VS.
Comment 23 mag@xamarin.com 2017-06-27 15:51:49 UTC
This bug report includes many comments and also different type of projects. Some refers only to iOS, others to Android and others to Forms or a combination.

Respect to Android builds, we have been testing different applications with different configurations and environments, and we can't observe any orphan MSBuild after closing the corresponding VS instance.

On the other hand, related to iOS, we have identified an issue within our build connection layer, which was causing CPU overhead and also some issues on the disposing and closing chain of the resources.

This is the bug related to that fix, that would also fix the remaining things for this bug: https://bugzilla.xamarin.com/show_bug.cgi?id=57339

This is the last commit of the series of fixes we did for this case and the bug above: b586e5bc2a59e01f5e14ff32a6e6995238bae433

After applying these changes, we didn't observe any orphan MSBuild in iOS projects, or even Forms projects with a combination of builds for the different platforms.

One important thing to comment is that the MSBuild instance will be alive from the first time you build until you close VS, because we maintain that instance and connection (in case of iOS) alive during the life of the parent process, which is VS in this case. Also, only one connection is maintained and not one per build.

So, the right way of testing this fix is that after you close all the VS instances you don't see any orphan MSBuild.

The latest mentioned fixes should be available for the next 15.3 refresh.

Given that we don't have more evidence of the existence of this issue, I mark this as Resolved.
Comment 24 Kent 2017-06-27 23:14:49 UTC
THANK YOU! Such a relief to hear this is fixed - it's been killing my productivity. Looking forward to 15.3.
Comment 25 Aman Dharwal 2017-08-02 12:32:57 UTC
Created attachment 23977 [details]
Attached IDE logs

Tried to reproduce the issue by Android, iOS and Forms project, but unable to get multiple instances of MS build, the project is build by only single instance of MS build, hence this issue doesn't exist anymore.

Environment info :

https://gist.github.com/aman-daharwal/322abc08e89e49354b431568dafae4ed

screencast link : https://www.screencast.com/t/0PQTchwPpCX
Comment 26 philip.o 2018-02-13 15:38:38 UTC
I see this on android builds:
https://forums.xamarin.com/discussion/comment/319882/#Comment_319882