Possible re-appearance of the AWS public key loading bug and what to do if you see it
Posted 29 Nov, 2017 by Daniil Baturin
There were some reports of re-appearence of the infamous bug with SSH public key loading in Amazon EC2. None of those were confirmed so far and one of them was traced to a user error, and we and our community members alike created dozens instances in different regions but could not reproduce the bug, so chances are it's a false alarm.
It's too early to become concerned, but we cannot rule out anything just yet because its previous appearence also started as intermittent and then became permanent and propagated to all regions.
To prevent a wide scale disaster, we are watching the situation closely. Past time when we could reproduce the bug ourselves it has already propagated too all regions and made the old AMI unusable for all users, so we definitely wouldn't want to repeat that. I suppose we should deploy some kind of automated test for it as well.
Meanwhile, if you have any issues logging in to a newly deployed AMI:
- Check carefully if the permissions of the private key file are set to 600 (rw-------). OpenSSH will say permission denied if the permissions are not restrictive enough, after telling you about permissions (http://lpaste.net/360350), but it has nothing to do with the key itself, it's just that it ignores the key with wrong permissions
- If you still cannot login to a newly deployed instance, please do not terminate it!. Snapshot it and contact us so that we can investigate it together.
Since marketplace AMI updates can take up to two weeks, if the bug does re-appear, we will provide a community AMI with all platform checks disabled to make the key load by default as an interim solution.
We are not ready to say if 1.1.9 will include the code that checks the platform by verifying the digital signature of the environment data, but 1.2.0 definitely will.
Right now the marketplace page of VyOS (https://aws.amazon.com/marketplace/pp/B074KJK4WC) has the sole negative review about the original SSH key issue that was written (or at least published) after the issue was resolved and the update was announced on the blog, which gives our AMI overall rating of one star. It feels rather unfair that this is all we get for working hard to fix the issue that wasn't even our fault as soon as possible and communicate it to our users.
If you are using VyOS on AWS, please consider leaving a customer review there to give it a more balanced rating.