Large browser vendors have quite some time ago established model of "trusted" web sites based on SSL certificates signed by "trusted" CAs. This trend increasingly spreads to other applications and protocols. This article gives exact steps needed for obtaining Let's Encrypt's wildcard certificates from FreeBSD host, using DNS validation through dynamic updates of a BIND server's zones, using EFF's ACME client, certbot.

Before Let's Encrypt There Was StartSSL

Before I switched to Let's Encrypt, I have been using StartSSL's free service, which was offering free "trusted" certificates that could be used to "secure" single second-level domain (eg. mimar.rs) and its single arbitrary subdomain (eg. www.mimar.rs). In order to obtain these, one would need to register account on StartSSL's website and do some 15 minutes of clicking around. But, less because they lasted whole year, contrary to Let's Encrypt's 90-day lifetime certificates, and more because they happened to be the only free "trusted" certificates around I was aware of, I was satisfied with their service. However, in autumn of 2016, StartSSL got themselves into a scandal which resulted in loss of "trust" by major players like Apple, Google, Microsft, Mozilla and others, and ultimately closed their business on 1st of January, 2018.

Subdomain Killed A Subfolder Star

In summer of 2016, FreeBSD ports tree has already been offering a choice of various ACME clients, of which security/dehydrated caught my eye, mostly because of short list of its dependencies. It took me no more than an hour to set .well-known/acme-challenge aliases in apache and configure dehydrated for http-based validation, after which I looked in awe how an one-liner gets me my certificates in a matter of seconds. But it didn't take long before I started to realize there is much more to Let's Encrypt than just shortening time needed to get my certificates. The newly obtained power to efortlessly get single certificate for a myriad of subdomains got me to rethink my design of placing various web services, such as Nextcloud, phpMyAdmin and Roundcube, into aliased subfolders (e.g. https://www.mimar.rs/roundcube), and I started to give them their own subdomains by means of apache name-based virtual hosts (e.g https://roundcube.mimar.rs). Furthermore, configuration of those apache vhosts could be greatly simplified by pointing all of the subdomain vhosts to the same set of certificates. With setup being so simple, it didn't take long before I started to obtain Let's Encrypt certificates for dozens of my clients' domains and subdomains in a completely automated way, with single crontab line.

Enter Wildcard

So, as far as web was concerned, things regarding "trusted" certificates went really well. But I wanted to push things further. Or have I been pushed to push them? I guess those two don't have to be mutually exclusive. Either way, there's much more to the Internet than just web, isn't it? What about other protocols like XMPP? SMTP? IMAP? LDAP? They all suppport secure communication which make use of SSL certificates, either via STARTTLS, or protocol over SSL, and I have been serving them securely for years, using self-signed certificates. However, developers of these services' client software, perhaps unfortunately, followed the path of big browser vendors, and started to warn users about self-signed certificates as "untrusted" as well. With Let's Encrypt around, it was not too hard for me to obtain certificates for subdomains offering these services, but not as easy as single crontab line, either. I had been considering setting up temporary apache vhosts for subdomains which otherwise didn't offer web services, lasting for the duration of domain validation and certificate signing. But then another game changer came into play - as of March 2018, Let's Encrypt started to offer wildcard certificates, eliminating the need to validate each and every subdomain. There's a catch though - they can be only obtained through dns validation via ACME v2, while my solution relied on http validation via ACME v1 protocol.

Tooling Switch

Now, although dehydrated supports both ACME v2 protocol and DNS validation, and also provides multiple hook examples which should provide convenient way to validate domains over DNS, I had a hard time figuring out underlying principles from its documentation and configure it for DNS validation. At one point I gave up and started to look for alternatives. Searching the web for related keywords brought my attention to EFF's ACME client which recently changed its name to certbot. Well structured user guide looked promising, but after I saw that the plugin dns-rfc2136, which was supposed to suit my needs, also has good documentation, and that FreeBSD ports tree already includes security/py-certbot-dns-rfc2136, I knew I had to give it a spin. To my amazement, I got my wildcard certificates with everything configured in a matter of minutes (as opposed to futile hours misspent with dehydrated's hooks, but that's is my fault, not dehydrated's), adding just a few lines to my BIND's configuration prior to runing a oneliner. Here's how that looked in a shell.

Oh, one more thing. I run all my services in separate jails. In order to minimize hassle, I run certbot directly on jail host, and mount certs directory via nullfs to all the jails needing it. As jail users will have access to sensitive files, not only world-readable private keys obtained from Let's Encrypt, but also secret needed for BIND zone updates, and Let's Encrypt account credentials, it is assumed that only trusted people log into them.

Now let's get our hands dirty.

Boring Stuff Part I - BIND

On BIND jail, use dnssec-keygen to generate new SHA512 TSIG key. This can be done in home directory as a non-privileged user.

dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST keyname.

The command created two files: Kkeyname.+165+XXXXX.key and Kkeyname.+165+XXXXX.private, 165 being the type of hashing algorithm and X's being seemingly random digits. Both files are readable with cat, so it's easy to extract secret key from .private one:

grep -e Key Kkeyname.+165+XXXXX.private | cut -d' ' -f2-

We will add this key to our named.conf:

key "keyname." {
  algorithm hmac-sha512;
  secret "i/KJzuKiYGbWlZ7x3qR6OxXjEX3L8XdeOy90F3aFYtRaGJp/8Ig4HmP/WfG6WsFqoe+a31emvJWeapxmNbnDmA==";
};

We also need to configure appropriate update-policy in configuration of the zone we want to have validated:

zone "mimar.rs" {
  type master;
  file "/usr/local/etc/namedb/master/mimar.rs.db";
  update-policy {
    grant keyname. name _acme-challenge.mimar.rs. txt;
  };
  allow-transfer { SEC.ON.DA.RY; };
};

BIND's master zone directory on FreeBSD is by default owned by root:wheel, with permissions set to 755, meaning only root can create and modify containing files. However, we have just configured our BIND server to allow dynamic updates on our mimar.rs zone, which will require creation of .jnl journal files in it. BIND on FreeBSD starts as root, but drops privileges to bind service account, and it is bind who creates journal files. Therefore we need to change master zone directory so that it is group-owned and group-writable by bind:

chown root:bind /usr/local/etc/namedb/master
chmod 775 /usr/local/etc/namedb/master

After service named restart our BIND DNS server is all set up - we can move on to installing and configuring certbot.

Boring Stuff Part II - Certbot

On jail host, install security/py-certbot-dns-rfc2136:

pkg install security/py-certbot-dns-rfc2136

Certbot does not need elevated privileges in order to obtain certificates. We could use personal unprivileged account to run it, but I prefer to create dedicated account just for this purpose, after OpenBSD's founder, Theo de Raadt, advised me so. I believe he would know a thing or two about security :)

pw groupadd certbot -g 20001
pw useradd certbot -u 20001 -c "Certbot unprivileged user" -d /nonexistent -g certbot -s /usr/sbin/nologin

We also need to create certbot's config dir and change its ownership accordingly:

mkdir /usr/local/etc/letsencrypt
chown certbot:certbot /usr/local/etc/letsencrypt/

A file named rfc2136.ini (or any other name which will be passed as a switch to certbot command later on) needs to be created with the following contents, placed into above config dir, chowned to certbot:certbot and chmodded to 600:

dns_rfc2136_server = IP.ADD.RE.SS
dns_rfc2136_name = keyname.
dns_rfc2136_secret = i/KJzuKiYGbWlZ7x3qR6OxXjEX3L8XdeOy90F3aFYtRaGJp/8Ig4HmP/WfG6WsFqoe+a31emvJWeapxmNbnDmA==
dns_rfc2136_algorithm = HMAC-SHA512

Default paths in current certbot version in FreeBSD (0.22.2 at the time I'm writing this) respect FreeBSD's standards, but they are not created on install. Let's create them, and give them appropriate ownership and permissions:

mkdir /var/log/letsencrypt
mkdir /var/db/letsencrypt
chown certbot:certbot /var/log/letsencrypt/
chown certbot:certbot /var/db/letsencrypt/

Time to obtain our certificates (assuming appropriate sudo privileges are in place):

sudo -u certbot certbot certonly --dns-rfc2136 \
  --dns-rfc2136-credentials /usr/local/etc/letsencrypt/rfc2136.ini \
  --server https://acme-v02.api.letsencrypt.org/directory \
  --email hostmaster@mimar.rs \
  --agree-tos --no-eff-email \
  --domain 'mimar.rs' --domain '*.mimar.rs'

If everything went well, we should get the following message:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for mimar.rs
dns-01 challenge for mimar.rs
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /usr/local/etc/letsencrypt/live/mimar.rs/fullchain.pem
   Your key file has been saved at:
   /usr/local/etc/letsencrypt/live/mimar.rs/privkey.pem
   Your cert will expire on 2018-07-18. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

If something went wrong, we can check logs in /var/log/letsencrypt/.

Boring Stuff Part III - Smuggling Stuff To Jails

Services on jail host using newly obtained certificates can now be pointed to certs in /usr/local/etc/letsencrypt/live/mimar.rs. What remains to be done is to provide read-only access to them from jails as well. In order to do this, we will add something like this to jail host's fstab (/etc/ssl/certs/certbot directory should already exist in jail):

/usr/local/etc/letsencrypt /usr/jails/somejail/etc/ssl/certs/certbot nullfs ro 0 0

All that remains to be done is to actually mount it...

mount /usr/jails/somejail/etc/ssl/certs/certbot

...and point jail's services using certificates to appropriate paths in /etc/ssl/certs/certbot/live/mimar.rs.

Obtaining wildcard certificates for multitude of domains could be configured with trivial adjustments to described setup, namely configuring their appropriate DNS zones for dynamic updates and passing these domains to certbot command.

Boring Stuff Part IV - Certificate Renewal

This is as easy as typing sudo -u certbot certbot renew. This will renew all the certificates we have previously requested. If you made it this far, than setting up certbot's unprivileged user's crontab should be a no-brainer. Restarting services in jails so that they load fresh certificates can be done with a bit of scripting where we will be checking if new certificates were issued, and doing something along the lines of sudo jexec -U root somejail service servicename restart. Perhaps I will add exact lines in this section sometime later on, at this moment I just want to finish the damn article, I'm writing it for days already :)

Afterword

You have probably noticed that throughout the article I sometimes quoted words security and trusted. In my personal opinion, Let's Encrypt certificates don't offer security, at least no more than self-signed certificates do. Furthermore, on contemporary Internet, where trust is increasingly being built only for the purpose of its later cash-in (I'm not going to call any names here but go figure), I don't trust freshly arrived, heavily promoted, Internet companies. Actually, as we say here in Serbia, I don't trust anyone until we eat a bag of salt together (It takes a lot of time, and a lot of physical contact, to eat a bag of salt with someone). I don't have enough knowledge in the area of cryptography to be able to know for sure there aren't any technical weaknesses that can be exploited, but I believe that power to secure communications should not be entrusted exclusively to corporations. What I got with above setup is compliance with requirements dictated by corporations, and maybe trust from non-technical people using my services, but not trust in general, as I don't trust these certificates, signed by what comes down to "some people on the Internet" more than I would trust those I self-signed, or those which I know were signed by people I trust. And security? Time will tell.

Next Post Previous Post