Bug 34932 - [XMA] Build host connection can fail if Mac build host is connected to a router that is not connected to the internet, even if other SSH connections succeed
Summary: [XMA] Build host connection can fail if Mac build host is connected to a rout...
Status: RESOLVED FIXED
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: XMA ()
Version: 4.0.0 (C6)
Hardware: Macintosh Mac OS
: Normal normal
Target Milestone: Future Cycle
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2015-10-16 04:53 UTC by Brendan Zagaeski (Xamarin Team, assistant)
Modified: 2017-06-30 03:14 UTC (History)
8 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
LogsXMAConnectionFailureRouterNoInternet.zip (5.48 KB, application/zip)
2015-10-16 04:53 UTC, Brendan Zagaeski (Xamarin Team, assistant)
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Brendan Zagaeski (Xamarin Team, assistant) 2015-10-16 04:53:06 UTC
Created attachment 13364 [details]
LogsXMAConnectionFailureRouterNoInternet.zip

[XMA] Build host connection can fail if Mac build host is connected to a router that is not connected to the internet, even if other SSH connections succeed



Based on the results I have gathered so far on this bug, one key consideration seems to be the various different behaviors of routers with respect to DNS lookups when they are not connected to the internet.


DNS lookups are apparently used by the OpenSSH server for reverse DNS lookup as part of the default security process [1].

[1] http://ubuntuforums.org/showthread.php?t=1411957&p=8857054#post8857054



Note: Rebooting Windows and disabling and re-enabling "Remote Login" in the System Preferences on the Mac were both important steps to ensure more consistent results when switching between various testing configurations. I should caution that even with those measures in place, I still had some trouble getting perfectly consistent results. Consequently the descriptions below might not be 100% accurate, but hopefully they should at least provide a good "feel" for the problem.




## Regression status: I suspect that the old HTTPS build server behaved differently

This is the trickiest aspect of this bug. On the one hand, some of the problematic behaviors are caused by `ssh` itself, and affect _any_ `ssh` client, not just XamarinVS. On the other hand, if the old HTTPS build server had less strict requirements for the DNS behavior of the router between the Windows PC and the Mac, then these complications could break existing customer environments.

At the least, we might need to explicitly test and polish up the "internet disconnected, 'UseDNS no' added to `/etc/sshd_config`" condition so that we will be able to offer that workaround to any customer who needs it.




## Router 1: D-Link WBR-1310



### Internet connected, default SSH server settings

Everything works smoothly.



### Internet disconnected, default SSH server settings


- Raw SSH connections (for example from a Linux machine, or using a simple SSH.NET test program [2]) succeed _eventually_ but it takes 30 seconds or longer before server prompts for the password.

[2] https://gist.github.com/brendanzagaeski/d963d1e031dbaffb5fe9/raw/Program.cs



- XamarinVS _eventually_ retrieves the SSH fingerprint, but cannot log in:

#### In "Output -> Xamarin" window

Couldn't connect to XSU-39A.local. Please try again.
Disconnected from Mac XSU-39A.local (172.16.5.1)



#### Behavior after entering the Mac build host IP address by hand in the "Xamarin Mac Agent" dialog

- Retrieving the SSH fingerprint takes a _long_ time (on the order of 30 seconds).

- The dialog shows "Trying to connect..." for approximately 11-30 seconds.

- The dialog displays "Couldn't connect to XSU-39A.local. Please try again."



### Internet disconnected, "UseDNS no" added to `/etc/sshd_config`

- XamarinVS connects successfully, but "abandons" the connection apparently due to a timeout while contacting the activation server on the Mac:

> Activating the Mac...
> Unable to activate the Mac. See the logs for more details (Help->Xamarin->Open Logs...)
> Failed to update iOS license: The request timed out
> Disconnected from Mac 172.16.5.1 (172.16.5.1)


(I was able to reproduce this second problem in at least one alternate way unrelated to the SSH configuration. I think it is not directly tied to the primary topic of this bug report, so I will file a second bug for it.)




## Router 2: Asus RT-N16



### Internet connected, default SSH server settings

Everything works smoothly.



### Internet disconnected, default SSH server settings

- [Different from Router 1] Raw SSH connections (for example from a Linux machine, or using a simple SSH.NET test program [2]) take a long time on the first attempt (on the order of 30 seconds), but succeed quickly on subsequent attempts.

[2] https://gist.github.com/brendanzagaeski/d963d1e031dbaffb5fe9/raw/Program.cs


- [Same as Router 1] XamarinVS _eventually_ retrieves the SSH fingerprint, but cannot log in. Subsequent attempts are _not_ any faster.



### Internet disconnected, "UseDNS no" added to `/etc/sshd_config`

- [Similar to Router 1] If you get lucky [3], XamarinVS will sometimes retrieve the fingerprint quickly and connect successfully, but then "abandon" the connection due to a timeout contacting the activation server on the Mac (as with Router 1).


[3] Some very quick observation seems to suggest that if you have the VM connected in "Shared Networking" mode, then the Mac build host has 2 IP addresses you can use. If you use the IP address for the Mac build host that is _not_ currently being displayed by the Bonjour list, then the fingerprint returns quickly.



## Environment info

Windows 8.1 (64-bit) VM in VMWare Fusion 6.0.6 on the same machine.

I primarily used the "Shared Networking" mode, but I also tried a few tests with the "Bridged Networking", so I suspect this problem would also affect a physical Windows machine attempting to connect to the Mac build host over the LAN.
Comment 1 Brendan Zagaeski (Xamarin Team, assistant) 2015-10-16 16:33:07 UTC
I have now filed the "Unable to activate the Mac" problem in its own non-public Bug 34978 so that this bug (Bug 34932) can focus on _just_ the SSH reverse-DNS issue.
Comment 7 Jose Gallardo 2016-01-05 20:31:36 UTC
We can confirm that changing /etc/sshd_config on the Mac with:

UseDNS no


resolves the issue, but there is no programmatical way to identify the scenario accurately from VS. We've already spent some cycles investigating if that's possible without success.

This is something that needs to be tackled from the docs, and there are no more action items on the XVS team.

Looping Amy who is working on XMA docs.
Comment 9 Brendan Zagaeski (Xamarin Team, assistant) 2016-03-01 02:15:02 UTC
> Should the path not be: /etc/ssh/sshd_config

The path is different on OS X 10.10 vs. OS X 10.11.
Comment 14 Jon Goldberger [MSFT] 2016-03-04 20:19:45 UTC
For anyone not familiar with UNIX, here is how you can edit sshd_config to use the noted workaround in comment 7:

1. Open the Terminal app (Applications/Utilities/Terminal.app)

2. Use Nano text editor to edit the file by entering the following command:
sudo nano /etc/ssh/sshd_config

3. Enter your admin password

4. Search for "UseDNS by pressing ctrl-w and entering:
UseDNS

5. Cursor should now be on the relevant line. On my system the line is commented out with a leading pound (#) sign, it may or may not be on yours (likely is), but whatever that line looks like, edit it to make it exactly:
UseDNS no

6. Press ctrl-o to write the file

7. press ctrl-x to exit the nano editor
Comment 15 mag@xamarin.com 2017-06-30 03:14:39 UTC
As Joe mentioned in Comment 7, we have no way identify this specific problem within the connection exception that we have on VS and the SSH layer.

Also, I could verify that the troubleshooting docs already have the necessary information about this possible issue and how to workaround it: https://developer.xamarin.com/guides/ios/getting_started/installation/windows/connecting-to-mac/troubleshooting/

For this reason, I consider that there is not much to do for this problem and I consider this as Resolved.