SSH is an undeniably popular protocol that we use for remote access to computer instances that we own (or maybe not?). It’s secure, simple to configure, and it comes built in with the majority of modern OS that we have right now for both client and server. It supports a couple methods of authentication, most notably password and public key based authentication. Granting access to a user is as simple as creating a user account on the instance and whitelist the user’s public key by adding it as a new line on
However like everything else, it’s getting complicated with scale. How do we grant access for hundreds of users to hundreds or thousands of computing instances? And yet that amount is completely normal now with most organizations that have systems serving thousands of users per day.
The simplest way would be creating a script to update
authorized_keys file to the machines. We can use automation platforms such as Ansible, Puppeteer, Chef, etc. Heck, we can even update that file periodically by setting a CRON job running
rsync. There are countless ways to get around this limitation.
But well, while these solutions work, they lack elegance and efficiency. For most cases, what we truly need is a way to somehow make SSH query our database system to determine if a user can pass or not. Then we can of course build a complex system on top of that database to define the flow that we need. Either it’s to add some kind of approval pipeline, implementing RBAC (Role Based Access) for users based on their groups/departments/etc, we can do that easily on the application. The good news is, we can do this without hassle, thanks to the flexibility of OpenSSH.
Authorized Keys Command to The Rescue
By default, OpenSSH will lookup for authorized keys in a file defined by
AuthorizedKeysFile in the server config
sshd_config. However, OpenSSH got an alternative called
AuthorizedKeysCommand which provide the same function in different way. Instead of reading the file, OpenSSH will execute the executable provided in
AuthorizedKeysCommand with arguments of your choice. Then the executable is expected to return a list of public keys, one per line, to the
stdout. That is your regular
That means we can do anything in the executable, whether it’s a query to database, HTTP call, FTP, anything you can implement in your favorite programming language. But of course, it would be better if the authentication method doesn’t slow down the process by a huge margin and doesn’t break for your scale.
Redis: Fast enough for most requirements
You don’t need to use Redis. But for my case, I think Redis suit my need best. It’s a fast key value storage, which means you can store a key and associate a value to it. The value can be anything serializable and the key is a string. OpenSSH will just provide the username during the authentication process anyway, so we don’t really need a robust and complex query system that other database solution provides.
(Also read: How fast is redis?)
First of all, let’s implement the authentication script. I will be using Python 3 for this.
Simple enough right? We can breakdown what the script does into these steps:
- Get the username from the execution argument
- Get the computer hostname
- Connect to redis and get members of the key
- Print out the keys
Note: this implementation is laughably not secure. Anyone would be able to change the value in redis and grant access to anyone to any server. Make sure to wrap the redis access into another service. For example, you can wrap it into an HTTP service and simply call the endpoint in the script.
So to test if our script is working properly, we will add a dummy public key on the redis instance.
127.0.0.1:6379> sadd Gabriel-PC_ubuntu public_key
Give the executable permission to the script with
chmod and make
root user own it with
ubuntu@Gabriel-PC:~$ sudo chmod 755 /etc/ssh/auth.py
ubuntu@Gabriel-PC:~$ sudo chown root:root /etc/ssh/auth.py
Install the dependencies (redis library)
python3 -m pip install redis
And fire it!
ubuntu@Gabriel-PC:~$ /etc/ssh/auth.py ubuntu
Looking good! Now let’s move on to the SSH server config.
SSH Server Configuration
The configuration is also simple, what you need to do is delete the
AuthorizedKeysFile directive and replace it with
AuthorizedKeysCommand. By default the configuration file should be located in
#MaxSessions 10PubkeyAuthentication yes# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2#AuthorizedPrincipalsFile noneAuthorizedKeysCommand /etc/ssh/auth.py %u AuthorizedKeysCommandUser root
As you can see, from my default configuration, I did two things:
- Uncomment the
- Uncomment the
AuthorizedKeysCommandand add the executable path +
%uas the argument (token for user argument)
- Uncomment the
AuthorizedKeysCommandUserand add a username. This would be the executor for the command.
- Another optional step is to add
LogLevel INFOto make debugging easier
After saving the configuration above, you can restart/reload the ssh service. Restart will well.. restart the SSH service while reload will just load the latest config. I’m using Ubuntu 18.04 and the command to do so is
sudo service ssh restart
Or alternatively if you want to debug, you can stop the service and start it manually. Don’t forget to start the service again after you’re done debugging.
ubuntu@Gabriel-PC:~$ sudo service ssh stop
* Stopping OpenBSD Secure Shell server sshd [ OK ]
ubuntu@Gabriel-PC:~$ sudo /usr/sbin/sshd -d
debug1: sshd version OpenSSH_7.6, OpenSSL 1.0.2n 7 Dec 2017
debug1: private host key #0: ssh-rsa SHA256:xygmXSTGgwKX46CG42khhP7dO8vRk4oGtdIRFBXrcFI
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:UsEHbhhZUr79/9rjY+T6DpsVEx8kdUlaeb9yXN1pq9w
debug1: private host key #2: ssh-ed25519 SHA256:odASZpd3w/x/KsZdPh4/wBAzTJbRO6vrNnZ9ZVLBp5A
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
And done! Test out the authentication with an SSH client of your choice. Play around with the value in redis to ensure that the script works properly.
Customizing your SSH server authentication flow is easy with the
AuthorizedKeysCommand directive. The next step would be for example, building an interface for access request and approval, making a role system to enable only certain roles for certain servers, etc.