SSH host identification changed for master.gnome.org?

Infrastructure
1

I came to upload a tarball today, and got:

$ scp /opt/gnome/build/glib/meson-dist/glib-2.77.3.tar.xz master.gnome.org:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:uLLU4nGpQeDPqLM28eG7dMiZy2ReDKk+LxbM9CGNTQ4.
Please contact your system administrator.
Update the SSHFP RR in DNS with the new host key to get rid of this message.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:uLLU4nGpQeDPqLM28eG7dMiZy2ReDKk+LxbM9CGNTQ4.
Please contact your system administrator.
Add correct host key in /home/philip/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /home/philip/.ssh/known_hosts:160
Host key for master.gnome.org has changed and you have requested strict checking.
Host key verification failed.
scp: Connection closed

Is the host identification expected to have changed recently for master.gnome.org?

1 Like
2

That’s an unexpected change from yesterday’s OCP upgrade, the master.g.o VM is part of our OCP Virt cluster, for some reason the upgrade of the virtualization operator also triggered cloud-init which regenerated the SSH keys for the host, that means it’s actually expected for that message to appear now, please remove the former keys from known_hosts and you should be good, I’ll update SSHFP entries accordingly.

4 Likes
3

Is it possible to arrange that the host SSH keys are stored in a persistent way such that they will survive across a routine redeploy?

Ideally, key rotation should be an exceptional event in the event of a security incident, with a lot of communication outbound via trusted platforms so that people know to re-verify the keys. e.g. We updated our RSA SSH host key - The GitHub Blog

FOSS infrastructure is a legitimate attack target and GNOME’s code runs on many millions of systems. Desensitising contributors/participants to the key just randomly changing as a routine event increases the risk here that an individual may one day not check or make the wrong choice, which makes GNOME more vulnerable to attacks.

For comparison - Endless uses Hashicorp vault to store host secrets which are used during deployment automation so that existing host keys can be re-used in the common redeployment case. We use AWS EC2/ECP so it’s packer+terraform+ansible, which has various options to access vault running in the administrator context, but there might be different ways to make stored host secrets available during the early bringup with cloud-init.

3 Likes
4

Robert,

I’ve landed backups for master.g.o so that this will be prevented in the future. Regarding your comment about Endless, I agree using HC Vault could be a viable solution but I believe you also understand I’m helping out in my free time and Bart is very busy as well, I don’t think having to maintain yet another service is something we’re looking forward to right now, that’s also why we’ve been focusing a lot into retiring old (unused) services including (hopefully soon) the Wiki, which is a major pain on its own due to the poor RBD OCP support.

1 Like