From ietf-dane at dukhovni.org Sat Sep 1 04:52:41 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Fri, 31 Aug 2018 22:52:41 -0400 Subject: Update on stats 2018-08 Message-ID: <20180901025241.GD47864@straasha.imrryr.org> Credits: The coverage of DNSSEC domains continues to improve with ongoing data support from Paul Vixie of Farsight Security. Credits also due to ICANN for the gTLD data via CZDS, Afilias for access to .INFO zone data, data contributions from the TLD registries for .CH, .DK, .INFO, .LI, .NL and .ORG and open access for .FR, .NU and .SE. More data sources of ccTLD signed delegations welcome. Summary: The DANE domain count is now 314,472 The number DNSSEC domains in the survey stands at 8,908,099. Thus DANE TLSA is deployed on 3.52% of domains with DNSSEC. All DNSSEC denial of existence issues are resolved at mijnhostingparner.nl and tse.jus.br. As of today I count 314,472 domains with correct SMTP DANE TLSA records at every primary MX host that accepts connections[1]. As expected the bulk of the DANE domains are hosted by the DNS/email hosting providers who've enabled DANE support for the customer domains they host. The top 15 MX host providers by domain count are: 112848 transip.nl 96132 domeneshop.no 34452 active24.com 23638 udmedia.de 10651 bhosted.nl 3720 interconnect.nl 2491 provalue.nl 2456 nederhost.nl 1666 yourdomainprovider.net 1297 xcellerate.nl 1177 hi7.de 1093 surfmailfilter.nl 724 omc-mail.com 608 core-networks.de 567 mailbox.org The real numbers are surely larger, because I don't have access to the full zone data for most ccTLDs, especially .no/.cz/.de/.eu/.be. Speaking of countries, the IPv4 GeoIP distribution of DANE-enabled MX hosts shows the below top 10 countries (each unique IP address is counted, so multi-homed MX hosts are perhaps somewhat over-represented): 4263 TOTAL 1453 DE, Germany 922 US, United States 541 NL, Netherlands 346 FR, France 161 GB, United Kingdom 124 CZ, Czech Republic 100 CA, Canada 58 SE, Sweden 57 SG, Singapore 56 CH, Switzerland IPv6 is still comparatively rare for MX hosts, and the top 10 countries by DANE MX host IPv6 GeoIP are (same top 6). 2066 TOTAL 813 DE, Germany 435 US, United States 309 NL, Netherlands 194 FR, France 66 GB, United Kingdom 66 CZ, Czech Republic 37 SE, Sweden 28 SG, Singapore 21 CH, Switzerland 18 AT, Austria There are 3563 unique zones in which the underlying MX hosts are found, this counts each of the above providers as just one zone, so is a measure of the breadth of adoption in terms of servers deployed. The number of published MX host TLSA RRsets found is 5015. These cover 5376 distinct MX hosts (some MX hosts share the same TLSA records through CNAMEs). The number of domains that at some point were listed in Gmail's email transparency report is 165 (this is my ad-hoc criterion for a domain being a large-enough actively used email domain). Of these, 91 are in recent reports: gmx.at jpberlin.de ouderportaal.nl transip.be lrz.de overheid.nl nic.br mail.de pathe.nl registro.br posteo.de politie.nl gmx.ch ruhr-uni-bochum.de transip.nl open.ch tum.de truetickets.nl anubisnetworks.com uni-erlangen.de utwente.nl fmc-na.com unitybox.de uvt.nl gmx.com unitymedia.de xs4all.nl mail.com web.de domeneshop.no societe.com dk-hostmaster.dk rushtrondheim.no solvinity.com egmontpublishing.dk webcruitermail.no t-2.com netic.dk aegee.org trashmail.com tilburguniversity.edu debian.org xfinity.com insee.fr freebsd.org xfinityhomesecurity.com octopuce.fr gentoo.org xfinitymobile.com comcast.net ietf.org active24.cz dd24.net isc.org cuni.cz gmx.net netbsd.org destroystores.cz hr-manager.net openssl.org klubpevnehozdravi.cz inexio.net samba.org knizni-magazin.cz mpssec.net torproject.org nic.cz t-2.net asf.com.pt optimail.cz transip.net handelsbanken.se smtp.cz xs4all.net iis.se bayern.de bhosted.nl minmyndighetspost.se bund.de boozyshop.nl skatteverket.se elster.de deltion.nl t-2.si fau.de hierinloggen.nl govtrack.us freenet.de interconnect.nl gmx.de intermax.nl Of the ~314000 domains, 1193 have "partial" TLSA records, that cover only a subset of the MX hosts. While this protects traffic to some of the MX hosts, such domains are still vulnerable to the usual active attacks via the remaining MX hosts. The number of domains with incorrect TLSA records or failure to advertise STARTTLS (even though TLSA records are published) stands today at 287. Some of these have additional MX hosts that don't have broken TLSA records, so mail can still arrive via the remaining MX hosts. A partial list is available at: https://github.com/danefail/list To avoid getting listed, please make sure to monitor the validity of your own TLSA records, and implement a reliable key rotation procedure. See: https://dane.sys4.de/common_mistakes http://imrryr.org/~viktor/ICANN61-viktor.pdf http://imrryr.org/~viktor/icann61-viktor.mp3 http://tools.ietf.org/html/rfc7671#section-8.1 http://tools.ietf.org/html/rfc7671#section-8.4 The DNSSEC denial of existence breakage is lower this month, as a result of a complete resolution of all issues at mijnhostingpartner.nl and tse.jus.br. After eliminating parked domains that do not accept email of any kind, the number of "real" email domains with bad DNSSEC support stands at 708. The top 20 name server operators with problem domains are: 112 webspacecontrol.com (reportedly working on a solution) 64 is.nl 51 dotserv.com 39 tiscomhosting.nl 33 sylconia.net 33 metaregistrar.nl 29 nrdns.nl 25 active24.cz (some broken wildcard cnames) 21 host-redirect.com 15 nazwa.pl (some broken wildcard NS RRs) 13 psb1.org 12 ultratek.com 11 blauwblaatje.nl 11 army.mil 9 dnscluster.nl 8 vultr.com 8 pcextreme.nl 8 glbns.com 7 forpsi.net 6 domdom.hu If anyone has good contacts at one of these providers, please encourage them to remediate not only the broken domains (I can send them a list), but also the root cause that makes the breakage possible. None of the domains all whose nameservers have broken denial of existence appear in historical Google reports. -- Viktor. [1] Some domains deliberately include MX hosts that are always down, presumably as a hurdle to botnet SMTP code that gives up where real MTAs might persist. I am not a fan of this type of defence (it can also impose undue latency on legitimate email). However, provided the dead hosts still have TLSA records, (which don't need to match anything, just need to exist and be well-formed) there's no loss of security. From johnpc at xs4all.net Mon Sep 3 12:57:07 2018 From: johnpc at xs4all.net (Jan-Pieter Cornet) Date: Mon, 3 Sep 2018 12:57:07 +0200 Subject: xs4all enabled DANE outgoing verification Message-ID: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> Hi, On smtp.xs4all.nl we enabled DANE outgoing verification, but currently only with a "soft fail": if DANE fails, we fallback to non-DANE delivery... for now. Except for a few hardcoded domains (currently only our own, and havedane.net). If anyone feels confident about their own DANE setup, feel free to send me your domain (or domains), and I'll add it to the list of hardfails. ... but in a few weeks we'll disable the softfail anyway, if we don't see any problems (other than the ones in the danefail list, which I'm not yet using, but the softfail is hitting some of the domains on that list already). We don't do smtp-tlsrpt reporting (yet?), but I can make some stats on demand for your domain, if you'd like that. As a word of caution to other would-be DANE implementers: we also had problems with a domain that was not on the dane-fail list. This domain had DNSSEC and TLSA records for the MX host, but did not offer STARTTLS. That would be a huge red flag, and fortunately we had the softfail fallback so mail kept on being delivered. After investigating, it turned out that this domain had *my* IPs in an exception list of "do not offer TLS", because a few years ago we hit some sort of timeout bug that caused hanging connections. The hanging connection bug has since been solved, but the IP exception was still there... So the moral of the story is: next to the domains in the dane-fail list, there might be local exceptions that might apply, so keep an eye on your logfiles. -- Jan-Pieter Cornet Systeembeheer XS4ALL Internet bv www.xs4all.nl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: OpenPGP digital signature URL: From ietf-dane at dukhovni.org Mon Sep 3 13:58:03 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Mon, 3 Sep 2018 07:58:03 -0400 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> Message-ID: <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> > On Sep 3, 2018, at 6:57 AM, Jan-Pieter Cornet wrote: > > On smtp.xs4all.nl we enabled DANE outgoing verification, but currently only with a "soft fail": if DANE fails, we fallback to non-DANE delivery... for now. Except for a few hardcoded domains (currently only our own, and havedane.net). Great news, welcome to the club, and thanks! When you do enable a default hardfail, you might consider exempting a particular sender address or subject tag, or perhaps a custom header: https://tools.ietf.org/html/draft-ietf-uta-smtp-require-tls-03#section-3 so that you can still send email to the contacts of any domains that are failing, prior to disabling DANE for the domain for all senders. You might find that some domains have intermittent outages as a result of poorly executed key/cert rollovers where the TLSA records are updated *after* they first become invalid. For any of those, soft fail may make sense until your logs show no failures for a year or more. > As a word of caution to other would-be DANE implementers: we also had problems with a domain that was not on the dane-fail list. This domain had DNSSEC and TLSA records for the MX host, but did not offer STARTTLS. That would be a huge red flag, and fortunately we had the softfail fallback so mail kept on being delivered. After investigating, it turned out that this domain had *my* IPs in an exception list of "do not offer TLS", because a few years ago we hit some sort of timeout bug that caused hanging connections. The hanging connection bug has since been solved, but the IP exception was still there... > > So the moral of the story is: next to the domains in the dane-fail list, there might be local exceptions that might apply, so keep an eye on your logfiles. Indeed the danefail list is not expected to be "complete". If any of you run into domains for which you need to make an exception, please open an issue or pull request on github if unable to resolve with the remote domain. For those publishing TLSA records for inbound DANE, please make *sure* that you're offering STARTTLS *unconditionally*, to all SMTP clients with no restrictions by client IP address or reputation. Configurations that restrict STARTTLS to a set of "good" IPs are not compatible with DANE. If STARTTLS was disabled with some client IPs for interoperability reasons, resolve those first. -- Viktor. From bjorn at mork.no Mon Sep 3 14:25:32 2018 From: bjorn at mork.no (=?utf-8?Q?Bj=C3=B8rn_Mork?=) Date: Mon, 03 Sep 2018 14:25:32 +0200 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> (Viktor Dukhovni's message of "Mon, 3 Sep 2018 07:58:03 -0400") References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> Message-ID: <87o9de3gzn.fsf@miraculix.mork.no> Viktor Dukhovni writes: > For those publishing TLSA records for inbound DANE, please make *sure* that > you're offering STARTTLS *unconditionally*, to all SMTP clients with no > restrictions by client IP address or reputation. Configurations that > restrict STARTTLS to a set of "good" IPs are not compatible with DANE. This is indeed an important point to consider. Never thought of the possibility that the same client would first fail TLS and then start using DANE at some later point in time. > If STARTTLS was disabled with some client IPs for interoperability reasons, > resolve those first. In a perfect world, yes. But in practice: How do you do that? I don't think it is realistic to offer STARTTLS without some local exception list. There are just too many buggy clients and ignorant sysadmins. Bj?rn From ietf-dane at dukhovni.org Mon Sep 3 14:52:23 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Mon, 3 Sep 2018 08:52:23 -0400 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <87o9de3gzn.fsf@miraculix.mork.no> References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> <87o9de3gzn.fsf@miraculix.mork.no> Message-ID: > On Sep 3, 2018, at 8:25 AM, Bj?rn Mork wrote: > >> If STARTTLS was disabled with some client IPs for interoperability reasons, >> resolve those first. > > In a perfect world, yes. But in practice: How do you do that? > > I don't think it is realistic to offer STARTTLS without some local > exception list. There are just too many buggy clients and ignorant > sysadmins. AFAIK, things aren't as dire as you describe. I'd expect MTA-to-MTA opportunistic TLS to be working reliably with the vast majority of senders, with the need for exceptions very rare. Yes, some Yahoo systems (still!) insist on shooting themselves in the foot and dropping to cleartext when WebPKI certificate verification fails, but they do then succeed in the clear: http://postfix.1071664.n5.nabble.com/Another-yahoo-problem-td89756.html#a89769 with no exceptions needed in their case. If a sender has buggy client-side STARTTLS, they'll typically send in the clear when STARTTLS fails, in which case you still don't need to make exceptions, let them retry. If you do find the need to make an exception, I'd recommend applying any STARTTLS exceptions on just a subset of your primary MX hosts. This has two benefits: 1. They can still get through if/when they fix their software turn on DANE. 2. Their logs and your logs continue to record intermittent failures while they are still broken, with the mail getting through via the MX hosts with the exception list. That way, you can prune your lists when failures cease, and they may take action sooner if they see ongoing issues. -- Viktor. From andreas.schulze at datev.de Mon Sep 3 15:07:55 2018 From: andreas.schulze at datev.de (Andreas Schulze) Date: Mon, 3 Sep 2018 15:07:55 +0200 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> Message-ID: Am 03.09.2018 um 12:57 schrieb Jan-Pieter Cornet: > On smtp.xs4all.nl we enabled DANE outgoing verification, but currently only with a "soft fail": if DANE fails, we fallback to non-DANE delivery... for now. Except for a few hardcoded domains (currently only our own, and havedane.net). If anyone feels confident about their own DANE setup, feel free to send me your domain (or domains), and I'll add it to the list of hardfails. Hello, cool stuff! I assume you use postfix. Could you be more verbose on how you implement what you name as "soft fail"? I'm currently not aware how to configure the "log dane failures but deliver anyway" Our domain (datev.de) could be a candidate for your "hardcoded domains". But I expect there is virtually no traffic between your and my users :-/ Andreas -- A. Schulze DATEV eG From bart.knubben at forumstandaardisatie.nl Mon Sep 3 15:30:27 2018 From: bart.knubben at forumstandaardisatie.nl (Knubben, B.S.J. (Bart) - Forum Standaardisatie) Date: Mon, 3 Sep 2018 13:30:27 +0000 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> Message-ID: Hi Jan-Pieter, > On smtp.xs4all.nl we enabled DANE outgoing verification, but currently only with a > "soft fail": if DANE fails, we fallback to non-DANE delivery... for now. Except for a few > hardcoded domains (currently only our own, and havedane.net). If anyone feels > confident about their own DANE setup, feel free to send me your domain (or > domains), and I'll add it to the list of hardfails. Very nice! Please add internet.nl and forumstandaardisatie.nl to your list of hardfails. -- Best regards, Bart Knubben Dutch Standardisation Forum https://www.forumstandaardisatie.nl/content/english Dit bericht kan informatie bevatten die niet voor u is bestemd. Indien u niet de geadresseerde bent of dit bericht abusievelijk aan u is toegezonden, wordt u verzocht dat aan de afzender te melden en het bericht te verwijderen. De Staat aanvaardt geen aansprakelijkheid voor schade, van welke aard ook, die verband houdt met risico's verbonden aan het elektronisch verzenden van berichten. This message may contain information that is not intended for you. If you are not the addressee or if this message was sent to you by mistake, you are requested to inform the sender and delete the message. The State accepts no liability for damage of any kind resulting from the risks inherent in the electronic transmission of messages. From ietf-dane at dukhovni.org Mon Sep 3 15:31:32 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Mon, 3 Sep 2018 09:31:32 -0400 Subject: xs4all enabled DANE outgoing verification In-Reply-To: References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> Message-ID: <305E0FBF-5436-4FEB-97DC-BDB19D1F2BB4@dukhovni.org> > On Sep 3, 2018, at 9:07 AM, Andreas Schulze wrote: > > I assume you use postfix. Could you be more verbose on how you implement what you name as "soft fail"? > I'm currently not aware how to configure the "log dane failures but deliver anyway" Postfix does not presently have a 'soft-fail' mode for DANE. So either xs4all is not using Postfix, or the code has been modified to support soft-fail. -- Viktor. From johnpc at xs4all.net Mon Sep 3 16:28:39 2018 From: johnpc at xs4all.net (Jan-Pieter Cornet) Date: Mon, 3 Sep 2018 16:28:39 +0200 Subject: xs4all enabled DANE outgoing verification In-Reply-To: References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> Message-ID: <351e949b-6337-f755-3b21-28d56bc055d4@xs4all.net> Hi Andreas, On 3-9-18 15:07, Andreas Schulze wrote: > Am 03.09.2018 um 12:57 schrieb Jan-Pieter Cornet: >> On smtp.xs4all.nl we enabled DANE outgoing verification, but currently only with a "soft fail": if DANE fails, we fallback to non-DANE delivery... for now. Except for a few hardcoded domains (currently only our own, and havedane.net). If anyone feels confident about their own DANE setup, feel free to send me your domain (or domains), and I'll add it to the list of hardfails. > > cool stuff! > I assume you use postfix. Could you be more verbose on how you implement what you name as "soft fail"? No, we use Cloudmark Gateway (version 5.5.2 at the moment). > I'm currently not aware how to configure the "log dane failures but deliver anyway" This MTA is fairly programmable. The way we implemented 'soft fail' is by inspecting the error in the "temp fail" phase. If that indicates a DANE problem (either bad TLSA records, bad certificate, or no TLS at all), then we re-queue the message for delivery within a few seconds, but marked as "no DANE". So what that effectively does is connect to the remote MX, notice that there's a problem and close the connection again. Then a few seconds later you get another connection, but this time we do not check DANE, and delivery proceeds with only opportunistic TLS. I'll use it to see if there are any more domains that need to be put on the dane-fail list. > Our domain (datev.de) could be a candidate for your "hardcoded domains". But I expect there is virtually no traffic between your and my users :-/ Actually, there is some traffic, a few mails a day it seems :). I've added your domain to the list. (Check out connections from 194.109.24.0/26. smtp.xs4all.nl is a cluster, and the frontend IP address is never going to make outgoing connections, but the cluster members are, which are in that network). > Andreas -- Jan-Pieter Cornet Systeembeheer XS4ALL Internet bv www.xs4all.nl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: OpenPGP digital signature URL: From johnpc at xs4all.net Mon Sep 3 23:26:33 2018 From: johnpc at xs4all.net (Jan-Pieter Cornet) Date: Mon, 3 Sep 2018 23:26:33 +0200 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> Message-ID: <60806b3a-41d9-b9db-4625-a201a082778f@xs4all.net> On 3-9-18 13:58, Viktor Dukhovni wrote: >> On smtp.xs4all.nl we enabled DANE outgoing verification[...] > > Great news, welcome to the club, and thanks! When you do enable a default > hardfail, you might consider exempting a particular sender address or subject > tag, or perhaps a custom header: > > https://tools.ietf.org/html/draft-ietf-uta-smtp-require-tls-03#section-3 > > so that you can still send email to the contacts of any domains that are > failing, prior to disabling DANE for the domain for all senders. Oh, that's a good idea. Implemented the "RequireTLS: NO" header now :) > You might find that some domains have intermittent outages as a result of poorly > executed key/cert rollovers where the TLSA records are updated *after* they first > become invalid. For any of those, soft fail may make sense until your logs show > no failures for a year or more. Hm, that requires quite a bit of state-keeping. Can such domains be added to the dane-fail list, or should those domains be put on another list? (dane-transient-failures?) > Indeed the danefail list is not expected to be "complete". If any of you run > into domains for which you need to make an exception, please open an issue or > pull request on github if unable to resolve with the remote domain. I'll go over all the DANE logs in a few days, and see if any domains not on the dane-fail list show any errors. If I find any, I'll contact the domains and if necessary create a github pull request on the danefail list. > For those publishing TLSA records for inbound DANE, please make *sure* that > you're offering STARTTLS *unconditionally*, to all SMTP clients with no > restrictions by client IP address or reputation. Configurations that > restrict STARTTLS to a set of "good" IPs are not compatible with DANE. > If STARTTLS was disabled with some client IPs for interoperability reasons, > resolve those first. Thanks, I'll use your message to persuade any other domains that don't send me STARTTLS, if I find any :). If they do not respond, can I add those domains to the dane-fail list too? I understand that selectively offering STARTTLS is a lot harder to test for other people... -- Jan-Pieter Cornet Systeembeheer XS4ALL Internet bv www.xs4all.nl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: OpenPGP digital signature URL: From johnpc at xs4all.net Tue Sep 4 09:33:17 2018 From: johnpc at xs4all.net (Jan-Pieter Cornet) Date: Tue, 4 Sep 2018 09:33:17 +0200 Subject: xs4all enabled DANE outgoing verification In-Reply-To: <87o9de3gzn.fsf@miraculix.mork.no> References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> <87o9de3gzn.fsf@miraculix.mork.no> Message-ID: On 3-9-18 14:25, Bj?rn Mork wrote: >> For those publishing TLSA records for inbound DANE, please make *sure* that >> you're offering STARTTLS *unconditionally*, to all SMTP clients with no >> restrictions by client IP address or reputation. Configurations that >> restrict STARTTLS to a set of "good" IPs are not compatible with DANE. > > This is indeed an important point to consider. Never thought of the > possibility that the same client would first fail TLS and then start > using DANE at some later point in time. In our case, it wasn't that we failed TLS, but a previous software version had a limitation that annoyed some receivers: the outgoing connection would be kept open for as long as the "max command timeout" value, which was set at 5 minutes. So after every mail delivery, the connection would stay open for another 5 minutes (the idea being: existing connections are cheap, making a TLS connection isnt, so if another mail arrives this connection can be reused). But not everyone agrees that connections are cheap, especially if your software forks for every incoming SMTP connection. So, most operators would, when looking at their systems at some point in time, just see a few of our hosts keeping open a connection to their system, usually doing nothing. This got combined with flaky NAT-alike loadbalancers that would expire idle TCP connections, and you'd get stuck with half-open TCP/SSL connections that took a while to get rid of (sometimes hours). I know at least in one case that this happened. For some reason, disabling TLS made these symptoms less painful, so that approach was taken (this was a few years ago, DANE wasn't on anyone's radar). Our current software no longer has that limitation, and we now close the connection 30 seconds after delivery (unless new mail arrives that can reuse the connection). >> If STARTTLS was disabled with some client IPs for interoperability reasons, >> resolve those first. > > In a perfect world, yes. But in practice: How do you do that? > > I don't think it is realistic to offer STARTTLS without some local > exception list. There are just too many buggy clients and ignorant > sysadmins. I never had a need for STARTTLS exceptions. Just make sure your software has appropriate timeouts and connection limits, and closes idle connections. And make sure that any loadbalancer you have in front of your systems, keeps track of TCP connections at least as long as the maximum timeout on your mail platform (and preferably a lot longer). If any software misbehaves (eg doesn't respect timeouts), complain to the vendor. When you really do get abusive clients, just block them completely instead of only blocking TLS. That draws their attention a lot sooner and puts much less of a load on your system. If this "client" is too big to be blocked completely, then either they likely have a well functioning abuse department and can resolve the issue, or you are doing something wrong in the first place, see timeouts and connection limits above. -- Jan-Pieter Cornet Systeembeheer XS4ALL Internet bv www.xs4all.nl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: OpenPGP digital signature URL: From andreas.schulze at datev.de Tue Sep 4 09:54:27 2018 From: andreas.schulze at datev.de (Andreas Schulze) Date: Tue, 4 Sep 2018 09:54:27 +0200 Subject: validation problem Message-ID: Hello, unsure it that topic may better discussed on postfix-users.... I'm messages for <*@sushi-circle.de> in my outbound deferred queue. Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: 422V1C0rjyzNkGv: to=<***@sushi-circle.de>, relay=none, delay=88795, delays=88794/0.24/0.61/0, dsn=4.7.5, status=deferred (TLSA lookup error for login.enterprise-email.com:25) Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: warning: TLS policy lookup for sushi-circle.de/login.enterprise-email.com: TLSA lookup error for login.enterprise-email.com:25 Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.login.enterprise-email.com type=TLSA: Host not found, try again Sep 2 11:15:37 idvmailout02 postfix/smtp[73811]: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.login.enterprise-email.com type=TLSA: Host not found, try again I wonder, why postfix try to lookup TLSA Records for the MX at all. The destination domain sushi-circle.de is unsigned and so any TLSA for the MX don't matter. Is there any switch to influence that behavior? mail_version = 3.2.5 -- A. Schulze DATEV eG From bjorn at mork.no Tue Sep 4 11:46:10 2018 From: bjorn at mork.no (=?utf-8?Q?Bj=C3=B8rn_Mork?=) Date: Tue, 04 Sep 2018 11:46:10 +0200 Subject: xs4all enabled DANE outgoing verification In-Reply-To: (Viktor Dukhovni's message of "Mon, 3 Sep 2018 08:52:23 -0400") References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> <87o9de3gzn.fsf@miraculix.mork.no> Message-ID: <87k1o11tp9.fsf@miraculix.mork.no> Viktor Dukhovni writes: > If you do find the need to make an exception, I'd recommend applying > any STARTTLS exceptions on just a subset of your primary MX hosts. > This has two benefits: > > 1. They can still get through if/when they fix their software > turn on DANE. > > 2. Their logs and your logs continue to record intermittent > failures while they are still broken, with the mail getting > through via the MX hosts with the exception list. That way, > you can prune your lists when failures cease, and they may > take action sooner if they see ongoing issues. Smart! That is certainly a good solution. Thanks Bj?rn From ietf-dane at dukhovni.org Tue Sep 4 16:09:24 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Tue, 4 Sep 2018 10:09:24 -0400 Subject: xs4all enabled DANE outgoing verification In-Reply-To: References: <33eaf11d-65a0-a82f-8e14-ac294d461573@xs4all.net> <6FCD0CB6-FD9E-41E3-81A7-ED4BFB2B91B7@dukhovni.org> <87o9de3gzn.fsf@miraculix.mork.no> Message-ID: > On Sep 4, 2018, at 3:33 AM, Jan-Pieter Cornet wrote: > > Our current software no longer has that limitation, and we now close the connection 30 seconds after delivery (unless new mail arrives that can reuse the connection). One off-topic comment on this subject. Postfix also has support for connection caching, but uses the feature much more carefully (on demand): 1. Connection caching to a destination is only enabled when the queue has more messages to that destination than the destination concurrency limit. In other words, only when there are messages waiting for a delivery slot queued behind the current message, and so the cached connection is likely to get used. 2. Cached connections are closed after 2s of idle time, only sustained traffic keeps cached connections open. 3. Cached connections are closed after ~300s of use. This amortizes connection setup latency when some MX hosts are slow. Sadly some receiving sites limit the number of messages per connection (rather than connection duration). That's unfortunate, re-use limits by message count don't overcome slow MX connection "attractors". See http://www.postfix.org/CONNECTION_CACHE_README.html#safety These ensure that connection caching is never seen as "aggressive" by receiving systems. -- Viktor. From ietf-dane at dukhovni.org Tue Sep 4 16:39:43 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Tue, 4 Sep 2018 10:39:43 -0400 Subject: validation problem In-Reply-To: References: Message-ID: <7210AC12-860B-4A73-BE17-271C1C9F74A4@dukhovni.org> > On Sep 4, 2018, at 3:54 AM, Andreas Schulze wrote: > > unsure it that topic may better discussed on postfix-users.... This probably belongs on the Postfix-users list, but perhaps not a bad idea if other DANE users also know. > I'm messages for <*@sushi-circle.de> in my outbound deferred queue. > Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: 422V1C0rjyzNkGv: to=<***@sushi-circle.de>, relay=none, delay=88795, delays=88794/0.24/0.61/0, dsn=4.7.5, status=deferred (TLSA lookup error for login.enterprise-email.com:25) > Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: warning: TLS policy lookup for sushi-circle.de/login.enterprise-email.com:TLSA lookup error for login.enterprise-email.com:25 > Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.login.enterprise-email.com type=TLSA: Host not found, try again > Sep 2 11:15:37 idvmailout02 postfix/smtp[73811]: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.login.enterprise-email.com type=TLSA: Host not found, try again > > I wonder, why postfix try to lookup TLSA Records for the MX at all. The destination domain sushi-circle.de is unsigned and so any TLSA for the MX don't matter. > Is there any switch to influence that behavior? http://www.postfix.org/postconf.5.html#smtp_tls_dane_insecure_mx_policy. Bottom line, DNSSEC-signed domains need to have working DNSSEC. You can disable DNSSEC for enterprise-email.com in your resolver and fix all the domains (~40 according to viewdns.info) that happen to use its MX hosts. In unbound that would be: /etc/unbound/unbound.conf: server: domain-insecure: "enterprise-email.com" However, in this case the real issue is that the nameservers for enterprise-email.com don't handle empty non-terminals correctly: http://dnsviz.net/d/_25._tcp.login.enterprise-email.com/dnssec/ An alternative fix is to disable qname-minimization (which does run into interop issues in such cases): server: qname-minimisation: no Then you'll find that the TLSA records actually exist! And mail to this domain will be partly protected by DANE (barring forged MX records, which leave forensic evidence in your logs). -- Viktor. From ietf-dane at dukhovni.org Tue Sep 4 17:00:05 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Tue, 4 Sep 2018 11:00:05 -0400 Subject: validation problem In-Reply-To: <7210AC12-860B-4A73-BE17-271C1C9F74A4@dukhovni.org> References: <7210AC12-860B-4A73-BE17-271C1C9F74A4@dukhovni.org> Message-ID: <163F4753-666E-4242-936B-89202481CED2@dukhovni.org> > On Sep 4, 2018, at 10:39 AM, Viktor Dukhovni wrote: > > An alternative fix is to disable qname-minimization (which does > run into interop issues in such cases): > > server: > qname-minimisation: no > > Then you'll find that the TLSA records actually exist! And mail > to this domain will be partly protected by DANE (barring forged > MX records, which leave forensic evidence in your logs). I should mention that at least four domains with the MX host in question are also DNSSEC-signed, so disabling DNSSEC would disable DANE for those four domains: enterprise-email.com. IN MX 10 login.enterprise-email.com. ; NoError AD=1 marcriemer.de. IN MX 10 login.enterprise-email.com. ; NoError AD=1 marcriemer.de. IN MX 20 smtp-in20.enterprise-email.com. ; NoError AD=1 weliano.com. IN MX 10 login.enterprise-email.com. ; NoError AD=1 weliano.com. IN MX 20 smtp-in20.enterprise-email.com. ; NoError AD=1 flexiconf.com. IN MX 10 login.enterprise-email.com. ; NoError AD=1 flexiconf.com. IN MX 20 smtp-in20.enterprise-email.com. ; NoError AD=1 login.enterprise-email.com[95.128.200.159]: pass: TLSA match: depth = 0, name = login.enterprise-email.com cert sha256 [matched] <- 3 0 1 ebb423a21d60370e9f9df7e5fdef08518748142c4411749758e386c560f05eba smtp-in20.enterprise-email.com[46.235.201.57]: pass: TLSA match: depth = 0, name = smtp-in20.enterprise-email.com cert sha256 [matched] <- 3 0 1 3f82f164796edead461434a60f13bf21416e6fba5f15c9a08c6483b644f81009 smtp-in20.enterprise-email.com[2a00:1200:0:9::65b]: pass: TLSA match: depth = 0, name = smtp-in20.enterprise-email.com cert sha256 [matched] <- 3 0 1 3f82f164796edead461434a60f13bf21416e6fba5f15c9a08c6483b644f81009 So you can have either qname-minimization, or unimpeded delivery to this and similar domains. You might reach out the tech-support team at enterprise-email.com and ask them to fix their nameservers, the mishandling of empty non-terminals needs to be fixed. -- Viktor. From andreas.schulze at datev.de Thu Sep 6 16:04:48 2018 From: andreas.schulze at datev.de (Andreas Schulze) Date: Thu, 6 Sep 2018 16:04:48 +0200 Subject: validation problem In-Reply-To: References: Message-ID: <3835d6fb-e237-1048-670b-369f2600d4e0@datev.de> Am 04.09.2018 um 09:54 schrieb Andreas Schulze: > Hello, > > unsure it that topic may better discussed on postfix-users.... > > I'm messages for <*@sushi-circle.de> in my outbound deferred queue. > Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: 422V1C0rjyzNkGv: to=<***@sushi-circle.de>, relay=none, delay=88795, delays=88794/0.24/0.61/0, dsn=4.7.5, status=deferred (TLSA lookup error for login.enterprise-email.com:25) > Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: warning: TLS policy lookup for sushi-circle.de/login.enterprise-email.com: TLSA lookup error for login.enterprise-email.com:25 > Sep 2 11:15:38 idvmailout02 postfix/smtp[73811]: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.login.enterprise-email.com type=TLSA: Host not found, try again > Sep 2 11:15:37 idvmailout02 postfix/smtp[73811]: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.login.enterprise-email.com type=TLSA: Host not found, try again > > I wonder, why postfix try to lookup TLSA Records for the MX at all. The destination domain sushi-circle.de is unsigned and so any TLSA for the MX don't matter. > Is there any switch to influence that behavior? > > mail_version = 3.2.5 > Update: the very responsive people @iptox.net fixed that. No issue anymore. Thanks! -- A. Schulze DATEV eG From ietf-dane at dukhovni.org Sat Sep 8 21:39:56 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Sat, 8 Sep 2018 15:39:56 -0400 Subject: Don't set the NSEC3 opt-out bit in your own zones Message-ID: <20180908193956.GC97812@straasha.imrryr.org> The NSEC3 opt-out bit allows DNS registry operators of public suffix domains (TLDs like .com, 2LDs like .co.uk, ...) where most delegations are unsigned to save CPU time and space by including only the signed delegations in their NSEC3 chain. While this is handy for such domains with lots of insecure delegations, it is otherwise a bad idea. Unless you're a TLD/2LD operator of that sort, DO NOT set the opt-bit in your NSEC3PARAM record. Here's one example of what can go wrong: http://dnsviz.net/d/_25._tcp.mail.ormanns.net/dnssec/ Until today, the ormanns.net domain used to have working secure TLSA records for its MX host: ormanns.net. IN MX 0 mail.ormanns.net. ; NoError AD=1 mail.ormanns.net. IN A 37.120.182.194 ; NoError AD=1 mail.ormanns.net. IN AAAA 2a03:4000:15:7e:: ; NoError AD=1 but then the TLSA record: _25._tcp.mail.ormanns.net. IN TLSA 3 1 1 c95ab7f0b49c1d762408d0089133d20d78c992e6e28b28c165d18e043c574ba8 was replaced with a wildcard: *._tcp.mail.ormanns.net. IN TLSA 3 1 1 c95ab7f0b49c1d762408d0089133d20d78c992e6e28b28c165d18e043c574ba8 unfortunately, while the zone is DNSSEC-signed, and the wildcard recurd is "secure", denial of existence of the removed TLSA RR for _25._tcp.mail.ormanns.net is "insecure" because of the NSEC3 opt-out bit. And so the wild-card synthesized TLSA is also "insecure" and the domain is no longer DANE-protected. One could keep the NSEC3 opt-bit bit and carefully add desired CNAMEs to share a TLSA record across services: _dane.mail.ormanns.net. IN TLSA 3 1 1 c95ab7f0b49c1d762408d0089133d20d78c992e6e28b28c165d18e043c574ba8 _25._tcp.mail.ormanns.net. IN CNAME _dane.mail.ormanns.net. _587._tcp.mail.ormanns.net. IN CNAME _dane.mail.ormanns.net. _465._tcp.mail.ormanns.net. IN CNAME _dane.mail.ormanns.net. _443._tcp.mail.ormanns.net. IN CNAME _dane.mail.ormanns.net. But, whether that's a good idea or not, it is far better to avoid NSEC3 opt-out and have secure denial of existence for all names in the zone, and then wildcards don't bring any security surprises. -- Viktor. From ietf-dane at dukhovni.org Sun Sep 30 20:39:04 2018 From: ietf-dane at dukhovni.org (Viktor Dukhovni) Date: Sun, 30 Sep 2018 14:39:04 -0400 Subject: Update on stats 2018-09 Message-ID: <20180930183904.GB3594@straasha.imrryr.org> Credits: The coverage of DNSSEC domains continues to improve with ongoing data support from Paul Vixie of Farsight Security. Credits also due to ICANN for gTLD data via CZDS, and to the TLD registries for .CH, .COM, .DK, .FR, .INFO, .LI, .NAME, .NL, .NU, .ORG and .SE. More data sources of ccTLD signed delegations welcome. Summary: The DANE domain count is now 316,920 The number DNSSEC domains in the survey stands at 8,986,410. Thus DANE TLSA is deployed on 3.52% of domains with DNSSEC. This month DNSSEC denial of existence issues were resolved at KPN Internedservices (internedservices.nl or is.nl) and dotroll.com (also known as webspacecontrol.com). My thanks to both for taking action to significantly reduce the residual barriers to DANE adoption. As of today I count 316,920 domains with correct SMTP DANE TLSA records at every primary MX host that accepts connections[1]. As expected the bulk of the DANE domains are hosted by the DNS/email hosting providers who've enabled DANE support for the customer domains they host. The top 15 MX host providers by domain count are: 114384 transip.nl 96340 domeneshop.no 34676 active24.com 23670 udmedia.de 10761 bhosted.nl 3721 interconnect.nl 2533 provalue.nl 2451 nederhost.nl 1521 yourdomainprovider.net 1299 xcellerate.nl 1189 hi7.de 1062 surfmailfilter.nl 753 omc-mail.com 622 core-networks.de 591 mailbox.org The real numbers are surely larger, because I don't have access to the full zone data for most ccTLDs, especially .no/.cz/.de/.eu/.be. Speaking of countries, the IPv4 GeoIP distribution of DANE-enabled MX hosts shows the below top 10 countries (each unique IP address is counted, so multi-homed MX hosts are perhaps somewhat over-represented): 4251 TOTAL 1449 DE, Germany 910 US, United States 549 NL, Netherlands 330 FR, France 158 GB, United Kingdom 128 CZ, Czech Republic 110 CA, Canada 57 SE, Sweden 56 SG, Singapore 55 CH, Switzerland IPv6 is still comparatively rare for MX hosts, and the top 10 countries by DANE MX host IPv6 GeoIP are (same top 6). 2126 TOTAL 816 DE, Germany 426 US, United States 317 NL, Netherlands 191 FR, France 70 GB, United Kingdom 70 CZ, Czech Republic 37 SE, Sweden 27 SG, Singapore 19 CH, Switzerland 17 AT, Austria There are 3571 unique zones in which the underlying MX hosts are found, this counts each of the above providers as just one zone, so is a measure of the breadth of adoption in terms of servers deployed. The number of published MX host TLSA RRsets found is 5087. These cover 5449 distinct MX hosts (some MX hosts share the same TLSA records through CNAMEs). The number of domains that at some point were listed in Gmail's email transparency report is 168 (this is my ad-hoc criterion for a domain being a large-enough actively used email domain). Of these, 86 are in recent (last 90 days of) reports: gmx.at lrz.de ouderportaal.nl transip.be mail.de overheid.nl nic.br posteo.de pathe.nl registro.br ruhr-uni-bochum.de politie.nl gmx.ch tum.de transip.nl open.ch uni-erlangen.de truetickets.nl anubisnetworks.com unitybox.de uvt.nl gmx.com unitymedia.de xs4all.nl mail.com web.de domeneshop.no societe.com dk-hostmaster.dk handelsbanken.no solvinity.com egmontpublishing.dk rushtrondheim.no t-2.com netic.dk webcruitermail.no trashmail.com tilburguniversity.edu aegee.org xfinity.com insee.fr debian.org xfinityhomesecurity.com octopuce.fr freebsd.org xfinitymobile.com comcast.net gentoo.org active24.cz gmx.net ietf.org cuni.cz hr-manager.net isc.org destroystores.cz inexio.net netbsd.org klubpevnehozdravi.cz mpssec.net openssl.org optimail.cz t-2.net samba.org smtp.cz transip.net torproject.org bayern.de xs4all.net asf.com.pt bund.de bhosted.nl handelsbanken.se elster.de boozyshop.nl minmyndighetspost.se fau.de deltion.nl skatteverket.se freenet.de hierinloggen.nl t-2.si gmx.de interconnect.nl govtrack.us jpberlin.de intermax.nl Of the ~317000 domains, 1390 have "partial" TLSA records, that cover only a subset of the MX hosts. While this protects traffic to some of the MX hosts, such domains are still vulnerable to the usual active attacks via the remaining MX hosts. The number of domains with incorrect TLSA records or failure to advertise STARTTLS (even though TLSA records are published) stands today at 258. Some of these have additional MX hosts that don't have broken TLSA records, so mail can still arrive via the remaining MX hosts. A partial list is available at: https://github.com/danefail/list To avoid getting listed, please make sure to monitor the validity of your own TLSA records, and implement a reliable key rotation procedure. See: https://dane.sys4.de/common_mistakes http://imrryr.org/~viktor/ICANN61-viktor.pdf http://imrryr.org/~viktor/icann61-viktor.mp3 http://tools.ietf.org/html/rfc7671#section-8.1 http://tools.ietf.org/html/rfc7671#section-8.4 The DNSSEC denial of existence breakage is lower this month, as a result of a complete resolution of all issues at is.nl and dotroll.com. After eliminating parked domains that do not accept email, the number of "real" email domains with bad DNSSEC support stands at 531. The top 20 name server operators with problem domains are: 51 dotserv.com 39 tiscomhosting.nl 35 metaregistrar.nl 33 sylconia.net 31 nrdns.nl 25 active24.cz (some broken wildcard cnames) 20 host-redirect.com 19 nazwa.pl (some broken wildcard NS RRs) 12 psb1.org 11 blauwblaatje.nl 10 eth-services.de 10 army.mil 9 vultr.com 9 dnscluster.nl 8 pcextreme.nl 8 forpsi.net 7 ovh.net 6 loopia.se 6 domdom.hu 5 1cocomo.com If anyone has good contacts at some of these providers, please encourage them to remediate not only the broken domains (I can send them a list), but also the root cause that makes the breakage possible. None of the domains all whose nameservers have broken denial of existence appear in historical Google reports. So it is likely that the DNSSEC denial of existence problems are not felt by most email senders. -- Viktor. [1] Some domains deliberately include MX hosts that are always down, presumably as a hurdle to botnet SMTP code that gives up where real MTAs might persist. I am not a fan of this type of defence (it can also impose undue latency on legitimate email). However, provided the dead hosts still have TLSA records, (which don't need to match anything, just need to exist and be well-formed) there's no loss of security.