The situation: I was trying to export (backup) a Cisco FXOS configuration to an SSH server using Secure Copy (scp), which is one of the methods supported by Cisco FXOS’s configuration export feature.
The SSH server is OpenSSH running in a Cygwin environment on Windows.
The issue is that the configuration export fails and the FXOS GUI just generates a generic and vague “End point timed out. Check for IP, port, password, disk space or network access related issues” error message.
What sheds some light is what the sshd process sends to the Windows Event Log:
sshd: PID 9676: fatal: seteuid 187611: Operation not permitted
Running sshd with -d (for debug), one can see that sshd does not handle that failure gracefully, and instead, terminates immediately. The client (FXOS, which is trying to use SSH to perform a secure copy), sees this as an authentication failure. This can be seen if one gets a fprm tech-support — in some file in the tech-support bundle one will see how an scp is spawned with the right arguments to perform the file copy but after the command runs one sees “Authentication failure” in the log.
After comparing good (from outside FXOS) and bad (from FXOS) scp transfers I realized that the difference is that FXOS is attempting to perform public key authentication. I have no idea where the key it is proposing comes from because I did not configure any public keys, but the fact of the matter is that it proposes a key and tries to authenticate using pubkey.
Normally if pubkey authentication is proposed and there is no matching key on the server, the client moves on to the next authentication method. However, because the SSH server is terminating abnormally because of the seteuid() error, the client cannot proceed with the next authentication method and everything dies there.
So, the main issue is the Cygwin SSH daemon’s handling of the seteuid() error, although one could argue that the real problem is that seteuid() fails. This could be the result of misconfiguration on the Cygwin SSH daemon, but whatever — on a Unix server this does not happen, and it is happening on Windows because of how complicated it is to handle POSIX accounts, permissions, and security — just read the following to get an idea of how complicated this is:
https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-setuid-overview
Now to the workaround — because pubkey authentication is essentially not working at all and is even preventing SSH clients proposing publey authentication to move to the next preferred authentication method, e.g. “password”, the workaround is to just disable pubkey authentication. On the Cygwin server where I ran into this problem this was accomplished by editing /etc/sshd_config, changing this line:
#PubkeyAuthentication yes
to:
PubkeyAuthentication no
and then restarting the sshd service.
So, if you run into some strange scp issue trying to backup (export) the FXOS configuration, try disabling pubkey authentication on the Cygwin SSH server; you might get lucky and you might get things to work.
Some other good references:
Cygwin FAQ: http://cygwin.com/faq.html#faq.using.sshd-in-domain
Somebody else running into a similar problem: http://cygwin.1069669.n5.nabble.com/seteuid-1019-Operation-not-permitted-td102924.html
Good blog post on configuring Cygwin’s SSHD: https://techtorials.me/cygwin/sshd-configuration/