DANE for SMTP and TLS certificate agility

[Carsten Strotmann]

Below a little text about TLS certificate agility and some pitfalls
herein (thanks to Viktor and Patrick for input and review): 

The Transport Layer Protocol (TLS) supports the introduction of new key
algorithms for x509 certificates. For a long time, keys based on the RSA
algorithm were the only available choice. Nowadays, TLS also permits
keys based on elliptic curve algorithms such as ECDSA. Other, new
algorithms might follow in the future.

Modern algorithms promise same or better security with less CPU usage
and smaller key sizes. This makes the new algoritms interesting for
operators of TLS secured services. However older software (on client or
peer server side) might not support and understand the newer algorithms,
so both the new and the old need be configured for some transition
time. Ideally, both communicating endpoints will negotiate the best
algorithm (in terms of security and performance) supported by both
sides. The scheme of supporting multiple certificate key algorithms is
called certificate agility.

Care must be taken when multiple key algorithms are used in a TLS setup
together with DNS-based Authentication of Named-Entities (DANE) as
documented in RFC 7672
([https://tools.ietf.org/html/rfc7672.html]). With SMTP-DANE, data
authenticating the TLS certificates (the hashes of TLS certificate,
hashes of the public key, or the full certificates) are stored in DNS,
secured by DNSSEC. Upon connection, a delivering mail-server (MTA) will
recieve the certificate chain from the destination MTA and will try to
validate this certificate chain via the data stored in DNS as one or
more TLSA resource records. The TLSA records contain either information
on the leaf-node certificates (called DANE-EE(3)) or information about
the certification authority (DATE-TA(2)).

With DANE-EE(3) and TLS certificate agility in use, each distinct
certificate that is configured on the server side does require it's own
TLSA record containing either the hash of the certificate, hash of the
public key of the certificate or the certificate itself. With
DANE-TA(2), multiple TLSA-Records are only needed if the certificates
are issued by different certification authorities (care must be taken to
compare the CA keys in the certificate chain, as certain CA companies or
"brands" are using different certificate authorities to sign x509
certificates).

The postfix mail server supports certificate agility, in the example
below two certificates are configured, one based on RSA keys, and a 2nd
certificate using ECDSA keys:

,----
| # RSA certificate configuration
| smtpd_tls_key_file=/etc/postfix/tls/rsa-key.pem
| smtpd_tls_cert_file=/etc/postfix/tls/rsa-cert.pem
| # ECDSA certificate configuration
| smtpd_tls_eckey_file=/etc/postfix/tls/ecdsa-key.pem
| smtpd_tls_eccert_file=/etc/postfix/tls/ecdsa-cert.pem
`----

This configuration requires two separate DANE-EE(3) TLSA records in DNS,
one for RSA and one for the ECDSA certificate. If both certifcates are
issued from the same CA, only one DANE-TA(2) TLSA record is required.

OpenSSL 1.0.x does not handle multiple chain files gracefully, and so
both certificate chain files need to contain all the issuer CA
certificates needed to validate either end-entity (leaf) certificate.
This is fixed in OpenSSL 1.1.0.

The selection of which certificate algorithm to use is based on the TLS
handshake. Older TLS software prefers RSA over ECDSA, while some more
modern software will have a preference for ECDSA. Today most DANE
commandline tools (ldns-dane, gnutls-cli ...), as well as online DANE
services such as the DANE SMTP Validator ([https://dane.sys4.de]), are
only testing one certificate (based on the preference or configuration
of the underlying TLS library used to implement the tools). A missing
TLSA record can go unnoticed but can lead to rejection of mail. The DANE
SMTP Validator will be updated to check all possible TLS certificate key
algorithms.

---

Carsten Strotmann