Combining an existing git repo with Keybase encrypted git
When developing software, particularly with larger cloud-based distributed systems, there are often a few secrets that need to be disseminated in a secure manner amongst the developers and to the services that use them to unlock credentials.
In systems with many credentials, services, and instances, storage of these secrets is often delegated to a “vault” like AWS Key Management Service or HashiCorp Vault. But you still need a credential to open the vault to pull out a credential for the specific principal and scope you want.
In the past I’ve seen people deposit secrets into source code files and include them in their source repository. Although access to those repositories is often private and data is typically transmitted over SSH, it is not a good practice as those files are stored in plain text in the repository and their contents are scattered throughout the repository’s history (generally forever).
Other mechanisms like LastPass and 1Password are good for teams to share these secrets, but the management of them is out-of-band of the normal development workflow and requires everyone to manually copy/paste or save the secure note to the right local path when there’s a change (which should be reasonably often).
Recently Keybase (who have been democratizing encryption for a few years now) have added encrypted git. This provides a nice balance between convenience and adequate security. Given these files are long-lived and persisted on shared infrastructure (“in the cloud”), there is still the risk of a 3rd-party gaining access to the data and using cryptanalytic techniques to eventually derive your secrets (which is one of the reasons why you should change them reasonably often).
I decided to try the facility using a real-world scenario of a git repository with a file containing a secret that I want to manage. I only want to store the secrets in the Keybase repository server, and I want to keep using the team’s usual repository server (e.g. Bitbucket or GitHub).
I thought of three different mechanisms I could use to bring together the unencrypted repository and the encrypted one:
- Separate local directories, with a symbolic link to join them
- Git subtree
- Git submodule
Of the three, the symbolic link is definitely the simplest (on reasonable file systems) but is still leaving it to each developer to work out that they need to clone a separate repository. It also assumes a particular directory structure on each developer’s machine (that the two repositories are siblings). The symlinks are stored fine in git and the missing targets are a good clue, but it didn’t feel quite good enough. The biggest kicker is obviously that the symlinks won’t work inside a Docker container (unless you also mount the target directory with the same relativity within the container).
While the git subtree tool makes merging multiple repositories into one project directory tree fairly easy, the problem is it is doing a form of merging. Thus, the encrypted part of the tree is actually decrypted and included in the unencrypted part even on the remote, so git subtree is not the solution we need in this case.
Good ol’ git submodule got a bum rap at some point, but I’ve found them to be highly effective from time to time. Indeed, the properties of submodules are perfect for this scenario. It means that principals (like the CI server) without access to the encrypted repository can still work with the rest of the source code in the unencrypted repository.
After installing the Keybase toolchain locally, creating an account, and asserting identities, go ahead and create an encrypted repository. If you’re doing this with a team, the team members will also need Keybase logins and be part of a Keybase team that you also create.
In the following snippets, substitute the following as appropriate to your machine: ${CODE_HOME}, ${KEYBASE_USER}, ${BITBUCKET_USER}. Relevant output is commented in italic and errors are commented in bold italic.
First, let’s create the encrypted repository:
cd ${CODE_HOME}keybase login # pop-up dialogkeybase git create my-secretsgit clone keybase://private/${KEYBASE_USER}/my-secretscd my-secretsecho "The Somerton Man was 'ere" > passphrase.txtgit add passphrase.txtgit commit -m "Initial commit"git push
Now there’s an encrypted version of our passphrase file stored in a git repository on the cloud. It remains in plain text on our local repository. You really should use an encrypted file system in case your machine’s storage is stolen.
Did you notice the keybase: protocol in the git remote URL? That indicates the data being sent and received by the git client is encrypted and the Keybase client is handling the de/cryption of the git payload.
Note: you’ll probably be using this in a team, so the Keybase URL will be more like keybase://team/{KEYBASE_TEAM}/our-secrets
To make this integrate with a normal git repository, create a private repository in Bitbucket (or GitHub). I’ve called mine my-service. For the purpose of example, I’ve created a shell script that says it’s going to authenticate with the contents of the passphrase file. If the passphrase file is not found in the subdirectory mapped to the submodule, the secret is not known.
cd ..git clone ${BITBUCKET_USER}@bitbucket.org:${BITBUCKET_USER}/my-service.gitcd my-servicecat <<-_END_ >login.sh
#!/usr/bin/env bash
echo "Authenticating with [\$(cat my-secrets/passphrase.txt)]"
_END_chmod +x login.sh./login.sh
# cat: my-secrets/passphrase.txt: No such file or directory
# Authenticating with []
At this point, the file containing the passphrase doesn’t exist in this directory tree. Adding the encrypted repository as a submodule will do that for us.
git submodule add keybase://private/${KEYBASE_USER}/my-secrets
# Cloning into ‘${CODE_HOME}/my-service/my-secrets’…
# fatal: transport ‘keybase’ not allowed
# fatal: clone of ‘keybase://private/${KEYBASE_USER}/my-secrets’ into submodule path ‘${CODE_HOME}/my-service/my-secrets’ failedIt looks like that keybase: protocol can’t be used for submodules! This is a default security setting in git to limit certain commands to known protocols. It can be done ephemerally with a command line variable, however, as we’re expecting to interact with this Keybase repository on a regular basis, we’ll configure git to allow it:
git config --global --add protocol.keybase.allow alwaysgit submodule add keybase://private/${KEYBASE_USER}/my-secrets
# Cloning into ‘${CODE_HOME}/my-service/my-secrets’…
# Initializing Keybase… done.
# Syncing with Keybase… done.
# Counting: 230 bytes… done.
# Cryptographic cloning: (100.00%) 230/230 bytes… done../login.sh
# Authenticating with [The Somerton Man was 'ere]
Excellent, we now have nicely integrated encrypted source code with unencrypted source code!
Let’s make sure that someone with access only to the unencrypted repository can’t get the encrypted repository contents due to the submodule. Remember, you need to have already created the empty repository on Bitbucket:
git add .git commit -m "Initial commit"git remote add origin ${BITBUCKET_USER}@bitbucket.org:${BITBUCKET_USER}/my-service.gitgit push --set-upstream origin mastercd ..rm -rf my-servicekeybase logoutgit clone ${BITBUCKET_USER}@bitbucket.org:${BITBUCKET_USER}/my-service.gitcd my-service./login.sh
# cat: my-secrets/passphrase.txt: No such file or directory
# Authenticating with []git submodule update --init --recursive
# Submodule ‘my-secrets’ (keybase://private/${KEYBASE_USER}/my-secrets) registered for path ‘my-secrets’
# Cloning into ‘${CODE_HOME}/my-service/my-secrets’…
# Initializing Keybase… done.
# git-remote-keybase error: (1) You are not logged into Keybase. Try `keybase login`.
# fatal: clone of ‘keybase://private/${KEYBASE_USER}/my-secrets’ into submodule path ‘${CODE_HOME}/my-service/my-secrets’ failed
# Failed to clone ‘my-secrets’. Retry scheduled
# Cloning into ‘${CODE_HOME}/my-service/my-secrets’…
# Initializing Keybase… done.
# git-remote-keybase error: (1) You are not logged into Keybase. Try `keybase login`.
# fatal: clone of ‘keybase://private/${KEYBASE_USER}/my-secrets’ into submodule path ‘${CODE_HOME}/my-service/my-secrets’ failed
# Failed to clone ‘my-secrets’ a second time, aborting./login.sh
# cat: my-secrets/passphrase.txt: No such file or directory
# Authenticating with []
Good, so someone who doesn’t have access to the encrypted repository can’t see the secrets. To access them, they need to log in to Keybase:
keybase login # pop-up dialoggit submodule update --init --recursive
# Cloning into ‘${CODE_HOME}/my-service/my-secrets’…
# Initializing Keybase… done.
# Syncing with Keybase… done.
# Counting: done.
# Cryptographic cloning: done.
# Submodule path ‘my-secrets’: checked out./login.sh
# Authenticating with [The Somerton Man was 'ere]# For future reference, the straight-forward way to update submodules from origin
git submodule update --remote --rebase
#Initializing Keybase... done.
#Syncing with Keybase... done.
The inclusion of encrypted git by Keybase is an superb feature and a nice addition to the kit bag of high performance technology teams who care about security. Give it a try, and let me know if you use it to improve your team’s process around sharing secrets.
UPDATE: This doesn’t tell the story of sharing those secrets with a CI/CD server. Given that Keybase requires each device to also be cryptographically identified, that makes life quite difficult in an age of infrastructure-as-code where your CI/CD system is possibly automatically created and configured.
One option is to simply duplicate those secrets needed for CI/CD into the CI/CD server’s configuration. Another alternative is to containerize the CI/CD server and supply a previous snapshot of the Keybase client state on that server each time it is provisioned. I will go into this in a subsequent article.
UPDATE 2: While some hackery with host volumes does solve the problem, the Keybase team are addressing it directly with “oneshot” commands using a paper key. Caveats about how to manage and deliver that paper key remain, however https://github.com/keybase/client/pull/11500 looks good.
