Bug 40959 - [XVS 4.0] "The application ... needs to be rebuilt due to an inconsistency between the connected Mac and the local app." due to "Renci.SshNet.Common.SshAuthenticationException: Permission denied (publickey)" during consistency check
Summary: [XVS 4.0] "The application ... needs to be rebuilt due to an inconsistency be...
Status: VERIFIED FIXED
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: iOS ()
Version: 4.1.0 (C7)
Hardware: PC Windows
: High major
Target Milestone: 4.2.0 (C8)
Assignee: Brendan Zagaeski (Xamarin Team, assistant)
URL:
: 39421 ()
Depends on: 43565
Blocks: 36059
  Show dependency tree
 
Reported: 2016-05-09 06:03 UTC by Brendan Zagaeski (Xamarin Team, assistant)
Modified: 2016-09-02 01:23 UTC (History)
12 users (show)

Tags: BZRC6S1_C5SR5S1
Is this bug a regression?: Yes
Last known good build: XVS 3.11


Attachments
Zip Xamarin Logs (20.94 KB, application/zip)
2016-07-21 17:59 UTC, Mohit Kheterpal
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Brendan Zagaeski (Xamarin Team, assistant) 2016-05-09 06:03:28 UTC
[XVS 4.0] "The application ... needs to be rebuilt due to an inconsistency between the connected Mac and the local app." due to "Renci.SshNet.Common.SshAuthenticationException: Permission denied (publickey)" during consistency check




## Comparison to other bugs about the "inconsistency" error

As discussed on other bugs, the "inconsistency" error can be _correct_ if the iOS project was intentionally built while disconnected from the Mac build host, but in this bug the error is _not_ expected. Any users who wish to confirm whether they are seeing this particular issue will need to install Cycle 7 (XamarinVS 4.1) or higher: earlier versions of Xamarin do not log the additional diagnostic error message from Renci.SshNet that is essential to identify this problem.




## No steps to replicate locally so far

The user reports of this problem so far suggest that the problem does not occur with a new template project, so the problem might have a tricky timing dependency or similar.

I am filing this initial report _without_ a known way for the Xamarin team to replicate the problem because we have had several user reports of the same issue.


### To the XamarinVS engineers

If it will be crucial to get access to a user project, you can indicate that on this bug report. The Support Team will then aim to coordinate with any users who are seeing this issue to obtain a project that consistently demonstrates the problem (at least on the user's machine). Thanks!




## Regression status: probably not an immediate regression

Based on the information available, it seems this issue was most likely introduced by the new SSH-based build host in the first Cycle 6 release but was obscured until now by various other causes of the "inconsistency" error message.




## Steps affected users have followed to collect the log files

Attempt to build and deploy certain iPhone applications in the "Debug|iPhone" configuration.




## Results

The build succeeds, but the deployment fails because the consistency check fails. The consistency check fails because SSH.NET claims that the private/public key authentication has failed. What is particularly strange is that the private/public key authentication works without error during the _build_, and the log messages from `sshd` show _no_ authentication failures during either the _build_ or the _deployment_.



### Message in the "Error List" and the "Output > Debug" window

"The application UnifiedSingleViewIphone1 needs to be rebuilt due to an inconsistency between the connected Mac and the local app. Please rebuild the application and try again."



### Corresponding excerpt of `*Ide.log` from "Help > Xamarin > Zip Logs"

(With the project name anonymized.)

> Xamarin.VisualStudio.IOS.MonoTouchProjectProperties Error: 0 : [2016-03-22 10:11:53.6346] There was an error checking the consistency between the local build and the remote build state on the Mac:
> 	Local signature path:  'C:\source\UnifiedSingleViewIphone1\UnifiedSingleViewIphone1\obj\iPhone\Debug\build-signature\signature'
> 	Remote signature path: 'C:\source\UnifiedSingleViewIphone1\UnifiedSingleViewIphone1\obj\iPhone\Debug\build-signature\signature-remote'
> 	Mac Server connected: True
> 	Build agent status: 'Running'
> 	AppName: 'UnifiedSingleViewIphone1'
> 	SessionId: 'ffbf46b165ee8291f1a8ac7c1bf1ee51'
> 	Server Signature Path: 'obj\iPhone\Debug\build-signature\signature'
> System.AggregateException: One or more errors occurred. ---> Renci.SshNet.Common.SshAuthenticationException: Permission denied (publickey).
>    at Renci.SshNet.ClientAuthentication.Authenticate(IConnectionInfoInternal connectionInfo, ISession session)
>    at Renci.SshNet.ConnectionInfo.Authenticate(ISession session)
>    at Renci.SshNet.Session.Connect()
>    at Renci.SshNet.BaseClient.Connect()
>    at Xamarin.Messaging.VisualStudio.MessagingService.<DownloadFileAsync>d__81.MoveNext()


### `/private/var/log/system.log` on the Mac shows _no_ authentication failures

> Mar 22 10:11:52 MacBuildHost sshd[62004]: Accepted publickey for ...
Comment 2 Brendan Zagaeski (Xamarin Team, assistant) 2016-05-10 21:19:57 UTC
Thanks to a user's test case, I have now been able to replicate this problem locally. I am working on minimizing the test case. I will then provide it to the engineers for further investigation.
Comment 5 Mathias Hartner 2016-06-14 09:32:41 UTC
Hi Brendan,

I have exactly the same problem. Building always succeeds but if I want to debug I always get the error message: "The application iOSApp needs to be rebuilt due to an inconsistency between
the connected Mac and the local app. Please rebuild the application and try
again.". The "system.log" also shows no authentication problems.

Excerpt of the "*Ide.log":

Xamarin.Messaging.Client.Ssh.MessagingAuthenticationMethod Warning: 0 : [2016-06-14 10:53:12.3672] Unable to authenticate using the private key authentication method
Renci.SshNet.Common.SshException: Message type 80 is not valid.
   at Renci.SshNet.Session.WaitOnHandle(WaitHandle waitHandle, TimeSpan timeout)
   at Renci.SshNet.PrivateKeyAuthenticationMethod.Authenticate(Session session)
   at Xamarin.Messaging.Client.Ssh.MessagingAuthenticationMethod.Authenticate(Session session)
Xamarin.VisualStudio.IOS.MonoTouchProjectProperties Error: 0 : [2016-06-14 10:53:13.1172] There was an error checking the consistency between the local build and the remote build state on the Mac:
	Local signature path:  'D:\Sources\Collection\Branches\Mobile\Mobile.iOS\obj\iPhoneSimulator\Debug\build-signature\signature'
	Remote signature path: 'D:\Sources\Collection\Branches\Mobile\Mobile.iOS\obj\iPhoneSimulator\Debug\build-signature\signature-remote'
	Mac Server connected: True
	Build agent status: 'Running'
	AppName: 'MobileiOS'
	SessionId: '29fdbcdc35d7932aecb00341521aa3c4'
	Server Signature Path: 'obj\iPhoneSimulator\Debug\build-signature\signature'
System.AggregateException: One or more errors occurred. ---> Renci.SshNet.Common.SshAuthenticationException: Permission denied (publickey).
   at Renci.SshNet.ClientAuthentication.Authenticate(IConnectionInfoInternal connectionInfo, ISession session)
   at Renci.SshNet.ConnectionInfo.Authenticate(ISession session)
   at Renci.SshNet.Session.Connect()
   at Renci.SshNet.BaseClient.Connect()
   at Xamarin.Messaging.VisualStudio.MessagingService.<DownloadFileAsync>d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Xamarin.Messaging.VisualStudio.MessagingService.<DownloadFileAsync>d__84.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Xamarin.VisualStudio.IOS.MonoTouchProjectProperties.MatchesBuildSignature() in C:\Users\builder\data\lanes\3345\2e397405\source\XamarinVS\src\Core\VisualStudio.IOS\PropertyProviders\MonoTouchProjectProperties.cs:line 91
---> (Inner Exception #0) Renci.SshNet.Common.SshAuthenticationException: Permission denied (publickey).
   at Renci.SshNet.ClientAuthentication.Authenticate(IConnectionInfoInternal connectionInfo, ISession session)
   at Renci.SshNet.ConnectionInfo.Authenticate(ISession session)
   at Renci.SshNet.Session.Connect()
   at Renci.SshNet.BaseClient.Connect()
   at Xamarin.Messaging.VisualStudio.MessagingService.<DownloadFileAsync>d__82.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Xamarin.Messaging.VisualStudio.MessagingService.<DownloadFileAsync>d__84.MoveNext()<---
Comment 6 aer127 2016-06-14 15:07:26 UTC
Same problem here!
Comment 7 Brendan Zagaeski (Xamarin Team, assistant) 2016-06-22 20:06:29 UTC
## Brief summary of the details I have provided to the engineering team (for anyone watching this bug who is wondering)

I have been able to replicate the problem reliably, but the ease of replicating the problem seems to depend quite heavily on the performance characteristics of the test systems.  For example:


- I was able to hit the problem fairly easily when I was testing with a Visual Studio instance running inside a virtual machine on the Mac build host itself.

- I had a little harder time hitting the problem with a Visual Studio instance running in a Hyper-V VM limited to 30% "Resource Control > Virtual machine limit" on a Windows 10 physical machine.

- I was not able to hit the problem at all with Visual Studio running in a Hyper-V VM set to 100% "Resource Control > Virtual machine limit" on a Windows 10 physical machine.  (Note: This doesn't mean it's not _possible_ to hit the problem in this setup.  I have a suspicion that the likelihood of hitting the problem also varies based on the exact solution being tested.  I used exactly one minimal test case across all of my tests so far to eliminate one potential source of variability.)


I have a suspicion that the resource consumption on the Mac build host might be involved in this problem too, but I have not yet had a chance to run a controlled experiment to test that idea.




## Only known partial workaround at this time

Since this problem is intermittent, most users have reported that if they repeat the deployment step enough times, the deployment will eventually succeed.

(Note: To minimize the time needed for each retry attempt, it is important to ensure that the project is not _building_ on each deployment attempt.  For example, if any files are set to "Copy to Output Directory > Copy Always" in item properties, that will force a build on every deployment attempt.  If needed, you can disable the build step temporarily by un-checking the "Build" checkbox for the project in "Build > Configuration Manager...".  Just be sure to re-enable that checkbox when you start making changes to the project.)




## Untested ideas that might reduce the chances of hitting the issue

Minimizing the number of NuGet packages and source files used by the solution or the iOS app project _might_ decrease the chances of hitting the issue.  I haven't had a chance to run a controlled experiment to test this idea either.
Comment 8 Brendan Zagaeski (Xamarin Team, assistant) 2016-07-14 20:49:52 UTC
## Note to the Xamarin team

For cross-referencing, there is a chance that updating XamarinVS to use the latest version of the SSH.NET library (as mentioned in Bug 42411) might help with this issue too.
Comment 12 Mohit Kheterpal 2016-07-21 17:59:12 UTC
Created attachment 16748 [details]
Zip Xamarin Logs
Comment 16 Brendan Zagaeski (Xamarin Team, assistant) 2016-07-22 19:00:40 UTC
*** Bug 39421 has been marked as a duplicate of this bug. ***
Comment 17 mag@xamarin.com 2016-08-01 20:21:56 UTC
After the feedback of all the repro steps to reproduce this bug in different ways, a problem was found related to thread safety on Debug & Launch process on iOS apps. For this reason, when multiple Start (F5) or Start Without Debugging (Ctrl + F5) actions are performed, then errors related to SSH commands, MLaunch or Debugging Session could be thrown.

Some improvements related to concurrency has been introduced on this Debug & Launch process, so I believe that this problem could be fixed for the most of the cases.

As mentioned in previous comments, build consistency checks between local build and remote build could throw errors that are expected, because it's the idea of the build signature check. However, there are some bugs that could affect on this check (directly or indirectly), and that's the idea of the past commits with fixes and improvements, an the latest ones that this comment covers. Commits applied:

- master: 20f4547f1ce3f997f7a065071ff808620c9a2f4e

- cycle8: 556b899297392c179fd813e86bfb62a4c7693823

As I said, because it's an issue with problems to get a consistent repro and also for the given repro, the latest commits takes care of it, I'm marking this issue as Resolved -> Fixed and waiting for testing feedback and verification.
Comment 18 Mohit Kheterpal 2016-08-04 15:29:10 UTC
As I have reproduce this issue in comment 11, Today I have again checked this issue with XVS 4.2.0.413 and able to reproduce this issue as shown in screencast : http://screencast.com/t/MdJFcsy4eTnT

Now, to verify this issue with fixed build mentioned in comment 17, I have tried to reproduce it with fixed build but the issue is not reproducible with fixed builds, so it seems that this issue has been fixed with today's Cycle 8 builds : Xamarin.VisualStudio_4.2.0.429 : cabdcc64eb28450d740b26967e5194e4d1465453 and master build : Xamarin.VisualStudio_99.0.0.3298 : 65f4ae866d745ef63464c354bd442279f38ab2dd

Hence closing this issue by marking it as Verified fixed.

thanks
Comment 19 Brendan Zagaeski (Xamarin Team, assistant) 2016-08-04 16:31:40 UTC
Because the full stack trace in non-public Comment 11 is different from the original stack trace from Comment 0, I will reset the bug status to "RESOLVED" until I get a chance to verify the behavior in the environment where I have consistently observed the original stack trace.  Thanks!
Comment 20 Brendan Zagaeski (Xamarin Team, assistant) 2016-09-02 01:23:22 UTC
Thanks for the candidate fixes!




## Verification status: verified fixed in Cycle 8 Beta 3

(To be precise, it is verified fixed as far as I can determine based on the various test procedures and environments I have used to date: I was no longer able to replicate the problem despite trying several different scenarios where I _did_ see the problem on earlier versions.)


As usual, I double-checked that I could still replicate the problem on Cycle 7 XamarinVS 4.1.0 before attempting the verification. (I was indeed still able to replicate the problem quite reliably on the old versions.)



### GOOD results Cycle 8 Beta 3 (as tested with the minimized test case)

0/7 trials



### Bad results for the minimized from non-public Comment 4, for comparison

6/7 trials 




## Version info (brief)

XamarinVS 4.2.0.628 (f4d4600)
Windows 8.1 (64-bit)

Xamarin.iOS 9.99.5.54 (cycle8: 974ea0b)
Mono 4.6.0 (mono-4.6.0-branch/3ed2bba)

Xcode 7.3 (10183.3), Build 7D175
Mac OS X 10.11.6