Support Encrypted Client Hello (formerly known as ESNI)

[Gunni]

Please add support for TLS Encrypted Client Hello

• Championed by the EFF: https://www.eff.org/deeplinks/2018/09/esni-privacy-protecting-upgrade-https
• Deployed by Cloudflare: https://blog.cloudflare.com/esni/
• Cloudflare's technical details post: https://blog.cloudflare.com/encrypted-sni/
• Supported by Firefox: https://blog.mozilla.org/security/2018/10/18/encrypted-sni-comes-to-firefox-nightly/
• Supported by NSS: https://bugzilla.mozilla.org/show_bug.cgi?id=1495120
• Work in Progress in picotls: Encrypted SNI h2o/picotls#155

Previous ticket for ESNI was not recreated after being closed, so here it is.

List taken from the openssl issue.

[francislavoie]

https://blog.cloudflare.com/encrypted-client-hello/

I don't think ESNI has a future.

And this isn't actionable for the Caddy project until Golang implements it. And there seems to be no plans to unless there's suddenly wider adoption.

[mholt]

ECH is its successor though. If Go implements it we should try to support it.

[francislavoie]

Oh sorry, I totally misread the issue title. My bad! 🤦‍♂️

[joshxyzhimself]

Related

• https://guardianproject.info/2021/11/30/implementing-tls-encrypted-client-hello/
• https://news.ycombinator.com/item?id=29530811

[diyism]

Any updates? all of nginx,apache,lighttpd have support ECH, how about caddy?

[francislavoie]

@diyism I can't find any evidence that they do support it. Where did you see that? Please share links.

[diyism]

@diyism I can't find any evidence that they do support it. Where did you see that? Please share links.

I've test it in https://defo.ie/ with msedge dev(https://www.thewindowsclub.com/enable-encrypted-client-hello-in-microsoft-edge):

nginx: https://draft-13.esni.defo.ie:10413
apache2: https://draft-13.esni.defo.ie:11413
both of the upper two pages show "SSL_ECH_STATUS: success" and a green tick mark.
Can it prove they hava a ECH-ready nginx and apache2 ?

[francislavoie]

Those are experiments using a custom build of OpenSSL. That's not really "support".

Either way, Caddy doesn't use OpenSSL, it uses Go's stdlib for TLS. So this work doesn't apply to us.

[mholt]

Yeah, that would be like us building with a custom fork of the Go standard library to support ECH. We could do that (Cloudflare has one), but given the sensitive nature of the crypto/tls package, I would only recommend that for experimental or low-risk environments for now. (CF is surely using it in production, but we don't know how much of the diff is tailored to just their use cases.)

So I guess we can answer "How about Caddy?" with "Sure, go for it" -- just build it with a forked tls package and it is possible. (I'm uncertain if any code changes to Caddy would need to take place, but since you're building from source already, might as well.)

As a side-note, the ECH draft spec seems to have expired: https://datatracker.ietf.org/doc/draft-ietf-tls-esni/

[ViViDboarder]

I think this doesn’t change anything as Caddy’s stance appears to be “when it’s supported in Go stdlib,” which makes total sense to me, but it looks like ECH has legs and is now supported in mainline Firefox: https://blog.mozilla.org/en/products/firefox/encrypted-hello/

[mholt]

Yeah; to clarify, we like ECH, and will support it someday assuming Go's standard lib implements it. (Because we're wary about switching to a fork of crypto/tls for our standard distribution of Caddy.)

However, I should note that ECH has limited practical use currently as DNS lookups still reveal hostnames in plaintext most of the time, and packets still have IP addresses that aren't encrypted.

It's just really, really hard to solve the problem people are hoping to solve, and ECH is not the whole solution, and without the whole solution, users are still at risk.

But it's cool I guess. A step in the right direction.

[ViViDboarder]

Firefox also defaults to DNS over TLS as well, so this is another step to close the gaps.

IP addresses is a big gap. Though it’s one that can be mitigated, depending on trade offs one is willing to make. I like to self host, so I run Caddy on my own server with my IP. So my connection is going to be known that way, but others, who may not care about their own end to end control, can run their server behind a CDN, like Cloudflare. In that case, the IP revealed is one that gives an interloper very little information.

I’m glad to have seen more and more progress over the last few years and excited to see what’s to come!

[dennisjackson]

@mholt - Just to clarify, ECH is the last part of the solution. There were four places where website names were leaking in plaintext:

• In the TLS Server Certificate (fixed by TLS 1.3)
• In the DNS lookup (fixed by DNS-over-HTTPS - which has already shipped in Firefox)
• In the certificate revocation checking (fixed by OCSP Stapling and CRLite)
• In the SNI field (fixed by ECH).

On the topic of IPs, they are far less unique than people realise. Cloudflare have already experimented with serving all of their websites from a single IP address [1]. In their measurement study, over 50% of the entire .com, .net, .info and .org namespace (255 million websites) is available from only ~100 IPs. This is obviously not great from a centralisation perspective, but that seems like more a reason for Caddy to support privacy features like ECH, not less!

[mholt]

Firefox also defaults to DNS over TLS as well, so this is another step to close the gaps.

I'm curious how widespread this is; if it's mostly us tech geeks who actually use this feature -- I know, for example, my parents don't benefit from this. I'd still wager the vast majority of DNS lookups are cleartext.

And yeah, IP addresses in some cases don't reveal much, inasmuch as we centralize to a single hosting provider.

but that seems like more a reason for Caddy to support privacy features like ECH, not less!

To AGAIN clarify, we like ECH. I just want to make sure that people understand that it is only a part of the solution and the solution is still largely incomplete. Your sites and web browsing experience may already have the whole solution, but (without seeing stats I can only guess) I'd wager that most of the Internet still has a long way to go, so I want to make sure we don't hype up something that isn't real for most people yet.

[dennisjackson]

I'm curious how widespread this is; if it's mostly us tech geeks who actually use this feature -- I know, for example, my parents don't benefit from this. I'd still wager the vast majority of DNS lookups are cleartext.

It's the default out of the box in North America, Russia and Ukraine at the moment. ECH is also enabled by default worldwide.

To AGAIN clarify, we like ECH.

That's great to hear!

I just want to make sure that people understand that it is only a part of the solution and the solution is still largely incomplete.

One step at a time :-).

[mholt]

(What I'm saying is that most people don't use Firefox, unfortunately. I know other browsers plan to roll it out too, but we're not there yet.)

[Harusakii]

(What I'm saying is that most people don't use Firefox, unfortunately. I know other browsers plan to roll it out too, but we're not there yet.)

I think we are there now, Chrome released the update 117.0.5938.150 which now enables ECH on default. Just saying though ^^

[networkException]

Where did you see that? Checking out chromium source at the 117.0.5938.150 tag declares kEncryptedClientHello as FEATURE_DISABLED_BY_DEFAULT in net/base/features.cc

[Harusakii]

Idk about Chromium but as of https://chromestatus.com/roadmap it is in Origin Trial/Prepare to Ship for 117 so it will come later, but either way Chrome did enable it by default (you may need to reset your chrome://flags) and then you can check by going to https://www.cloudflare.com/ssl/encrypted-sni to see it enabled

[mohammed90]

We're dependent on Go standard library to support this function. Watch golang/go#63369. It seems to be going in a positive direction.

[Helanzy]

Interesting; so, you're suggesting that some residential ISPs block all TLS handshakes with a non-empty SNI?

Direct TLS handshaking is not blocked. In fact occasional access to the web page is also tolerant, only that if hosting a long-lived web service and ISP detects it at a time, residential IP may get degraded accessibility, e.g. 1) block the port; 2) convert residential public IP to private NAT IP; 3) contact the owner for a fine or stop network service. SNI here is a proof for ISP that a web service exists.

That seems odd, since if they want to block all HTTP, they should be blocking based on ALPN and/or port, not SNI.

Port 80/443 has already been banned and no connection can be made using this port. ALPN is indeed another way to detect that it's HTTP connection, but SNI helps ISP to do a replay to see if there's really a web service.

Besides, in Cloudflare's blog, ALPN is encrypted into ClientHelloInner in ECH, and ClientHelloOuter need not to specify it:

It's not necessary that the client specify an ALPN list in the ClientHelloOuter, nor any other extension used to guide post-handshake behavior. All of these parameters are encapsulated by the encrypted ClientHelloInner.

Therefore, outer SNI might be the only issue for a non-HTTPs TLS handshake in this scenario.

The encrypted_client_hello extension (65037) is advertised.

Okay, then ECH may not be a permanently safe way for such scenario. But given that ClientHelloOuter is designed to look like normal TLS connection to live with existing network ossification, making ECH without outer SNI looks like non-HTTPs TLS connection also makes it fit into existing network ossification in the long-term, since it does not influence ISP's basic functionality, which otherwise would make them aware of ECH and prompt them to deal with it.

[christaikobo]

Sorry if I seem dismissive -- I want to be clear that I value your feedback and experience. I'm trying to understand it, and for me, that understanding comes by way of scrutiny. It's how I challenge my current view of the topic. So please forgive me if I break it down, I really am just trying to make sure I am on the same page, since I don't believe there's much published literature on this topic yet. (I haven't found it if there is.) So I feel like we're forging ahead here.

Sorry I was a bit impatient as I was a bit frustrated, not being able to get my perspective across.

I have said earlier that for small organization and individual, hiding private SNI with public SNI is pretty much useless, because you are revealing an SNI anyway and if you only have one domain, it doesn't really matter if you are revealing "a.domain.com" or "b.domain.com".

If the outer SNI is not that of the services in the anonymity set, then what is the harm in it?

The restriction that residential user shouldn't host anything the other user mentioned applies as well, SNI is an indication of the existence of HTTPS server. I didn't want to talk about it because it is quite niche.

But more importantly, the thing about privacy is sometimes not always about immediate and tangible harm, which is unlike security. One of the major ISP in my country which has over a billion users used offer this service, in which you could print out all the website you have visited this billing period, that they logged via SNI. It's thousands of entries if not 10s of thousands of entries. And I don't like that at all. I am going to make an analogy: what harm does it have if some random person sees me naked? But I certainly don't want to be seen naked, even if no harm comes my way.

Also, can you help me understand why "if you only have one domain" is a relevant constraint? I don't think there's any reason you have to only have 1 domain. Do you mean registered domain?Even if you do only have one of those, maybe for cost reasons, you could serve the actual service at service.example.com but have the public name be example.com -- something generic. Or maybe ech.example.com. What does that reveal that can't already be inferred from casual inspection (e.g. IP address etc)?

The thing is a lot of these automated system will never do "casual inspection", they only do automated logging and/or blocking based off certain predefined rules. And SNI is a VERY vital part of these automated systems. They don't have the resources to "casually inspect" every user. Nor would they want to deploy such a unreliable method which could cause massive side effects on a large scale, which I explained in the last comment.

About "if I only have one domain", if I have 10 domains, and I use ech.domain-a.com, I can at least hide the other 9 domains. Again, automated systems will not target me individually and "casually inspect" me, that is just not feasible.

[mholt]

Sure, but again that's what GREASE is for. Everyone should have GREASEd ECH in the handshake even if they're not using it, to mitigate that possibility

Good point, but I also don't think I will reasonably expect that all major clients will implement GREASE... maybe, and it'd be cool if basically every TLS handshake included it. But I dunno, we'll see.

But more importantly, the thing about privacy is sometimes not always about immediate and tangible harm, which is unlike security. One of the major ISP in my country which has over a billion users used offer this service, in which you could print out all the website you have visited this billing period, that they logged via SNI. It's thousands of entries if not 10s of thousands of entries. And I don't like that at all. I am going to make an analogy: what harm does it have if some random person sees me naked? But I certainly don't want to be seen naked, even if no harm comes my way.

And to clarify, I totally agree: there is a reason we wear clothes... nobody is disagreeing with that. We're just trying to figure out whether it's more private to wear a white shirt than a blue shirt or a graphic tee, etc.

The thing is a lot of these automated system will never do "casual inspection", they only do automated logging and/or blocking based off certain predefined rules. And SNI is a VERY vital part of these automated systems.

Right, but this doesn't explain why no SNI is better than an arbitrary non-empty one.

---

I feel like the argument for an empty outer SNI is still beating around the bush. What's the actual privacy benefit that is so substantial it is worth making client connections brittle? So far, the claim remains unsubstantiated. What am I missing?

[Gunni]

Having a default of outer SNI enabled with a domain already managed by caddy makes sense, it's the most compatible way.

I just think having the ability to have no outer SNI makes sense, even if it isn't the default. Just like being able to configure what TLS version are allowed, but TLS 1.2 (as configured by caddy by default) vs TLS 1.3 have no substantial privacy benefits, right?

[mholt]

@Gunni Gotcha -- although, to nit-pick, I disagree with:

Just like being able to configure what TLS version are allowed, but TLS 1.2 (as configured by caddy by default) vs TLS 1.3 have no substantial privacy benefits, right?

TLS 1.3 does add provable privacy benefits (post-quantum cryptography, for example), and at least at the time, TLS 1.2 was still required for a vast majority of clients to even be compatible.

I just think having the ability to have no outer SNI makes sense, even if it isn't the default.

To re-iterate what I said above, I hesitate to add this option without provable benefits (I added to my own quote in bold):

I'm opposed to just doing this without a really compelling reason is because I'm worried about introducing security/privacy theater into the application. If people think they have more privacy without a public name, and that's not provably true, then that's harmful because it makes the protocol and the whole ecosystem more brittle, [not to mention making people believe they have more privacy when they actually may not]. Clients will fail to connect entirely -- or worse, resort to insecure methods of connecting as a last resort -- without a public SNI.

It is provably harmful to add this without substantial reason. What hasn't been proven yet is that the reason to add it is, in fact, a substantial privacy benefit.

[Helanzy]

Right, but this doesn't explain why no SNI is better than an arbitrary non-empty one.

Per my understanding, for ClientHelloOuter to play the role to securely provide updated public key on decrypt failure, an encrypted TLS connection should be established using outer SNI. So that means a valid certificate shall be provided for outer SNI, which means we shall own the provided outer SNI domain, right?

Then, we cannot say it's an arbitrary non-empty domain. For home users, they may have to use their only domain ech.mydomain or whatever.mydomain as outer SNI. Then, the whois information, or regulated registrant for mydomain, or delegated subdomain sub.mydomain to regulated DNS provider, or personal page hosted at etld+1 mydomain may be enough to reveal their identity.

I feel like the argument for an empty outer SNI is still beating around the bush. What's the actual privacy benefit that is so substantial it is worth making client connections brittle? So far, the claim remains unsubstantiated. What am I missing?

Since I didn't see quote regarding my comment, I assume this is a response to my comment. I am seeking ECH as an opportunity to build a TLS connection with HTTPS information private:

1. Even ISP found ECH enabled in advertised extensions, they cannot tell whether it's HTTPS or not.
2. ISP cannot do a replay to see if there's a web service hosted on specific port.
3. ISP cannot trace domain owner's information using outer SNI.

ECH with outer SNI (with valid certificate) helps with 1 (ALPN is private now) and 2 (accessing ech.mydomain directly will be rejected), but 3 is not trace-proof (especially regarding owning the domain to provide a certificate). With empty outer SNI, 3 is ensured, while 1 is strengthened as there have been reports in my country's ISP using SNI as proof for existence of web service. On the contrary there are more ways to mitigate brittle client connection, e.g. using a private VPN channel to advertise public key in time.

[christaikobo]

I'm opposed to just doing this without a really compelling reason is because I'm worried about introducing security/privacy theater into the application. If people think they have more privacy without a public name, and that's not provably true, then that's harmful because it makes the protocol and the whole ecosystem more brittle, [not to mention making people believe they have more privacy when they actually may not]. Clients will fail to connect entirely -- or worse, resort to insecure methods of connecting as a last resort -- without a public SNI.

How about putting the option in an addon? Anyone who uses default binary or docker won't have the option.

And anyone who actively compile the addon into caddy should be aware what they are doing.

[mholt]

Per my understanding, for ClientHelloOuter to play the role to securely provide updated public key on decrypt failure, an encrypted TLS connection should be established using outer SNI. So that means a valid certificate shall be provided for outer SNI, which means we shall own the provided outer SNI domain, right?

Specifically, the server needs to be authoritative for the outer domain, yes.

But that should be no surprise. For example, of course a Cloudflare IP is going to own cloudflare-ech.com. Or you might own alsdkflksjdflksjdf.com at your residential IP. 🤷‍

Then, we cannot say it's an arbitrary non-empty domain.

Sure, but what I meant is that it doesn't have to be related to the services being hidden. It can be anything, even a random ASCII string. But yes you're right it can't just be any domain name. It can be random, though.

For home users, they may have to use their only domain ech.mydomain or whatever.mydomain as outer SNI.

Huh, I was not aware of this restriction. Can you tell me more?

Then, the whois information, or regulated registrant for mydomain, or delegated subdomain sub.mydomain to regulated DNS provider, or personal page hosted at etld+1 mydomain may be enough to reveal their identity.

So this is where I would say... don't do that.

(Also, is WHOIS information still publicly available? I wasn't aware it was still published. Just trying a few domains right now and I just get hidden/privacy masks in the output.)

But even so, that all misses the point.

ECH is not meant to hide associations with IP addresses. In fact, associations can be drawn even without domain names. Again, it is an orthogonal concern.

ECH with outer SNI (with valid certificate) helps with 1 (ALPN is private now) and 2 (accessing ech.mydomain directly will be rejected), but 3 is not trace-proof (especially regarding owning the domain to provide a certificate). With empty outer SNI, 3 is ensured, while 1 is strengthened as there have been reports in my country's ISP using SNI as proof for existence of web service. On the contrary there are more ways to mitigate brittle client connection, e.g. using a private VPN channel to advertise public key in time.

Thanks for elaborating. However I don't think I see things the same way:

they cannot tell whether it's HTTPS or not.

I believe that should be true, however, be aware that most ISPs / powers-that-be just use port, since the vast majority of HTTPS traffic is on port 443.

ISP cannot do a replay to see if there's a web service hosted on specific port.

Can you help me understand this better? The spec doesn't mention the word "replay" at all so I would be surprised if this was a design goal.

ISP cannot trace domain owner's information using outer SNI.

ISP does not need outer SNI to do this. :) IP address is even more informative.

How about putting the option in an addon?

Maybe, but enabling support for a module here would be akin to advocating the capability, I feel like.

[Helanzy]

But that should be no surprise. For example, of course a Cloudflare IP is going to own cloudflare-ech.com. Or you might own alsdkflksjdflksjdf.com at your residential IP. 🤷‍
It can be anything, even a random ASCII string. But yes you're right it can't just be any domain name. It can be random, though.

For home users, they may have to use their only domain ech.mydomain or whatever.mydomain as outer SNI.

Huh, I was not aware of this restriction. Can you tell me more?

The it can be random domain statement here is what I don't understand all along. The whole discussion is based on the statement Client will establish TLS connection using outer SNI if failed to decrypt ClientHelloInner, as stated in Cloudflare's Blog:

At a minimum, both ClientHellos must contain the handshake parameters that are required for a server-authenticated key-exchange. In particular, while the ClientHelloInner contains the real SNI, the ClientHelloOuter also contains an SNI value, which the client expects to verify in case of ECH decryption failure (i.e., the client-facing server). If the connection is established using the ClientHelloOuter, then the client is expected to immediately abort the connection and retry the handshake with the public key provided by the server.

Then, from my understanding we need a certificate signed by acknowledged CAs for outer SNI, to make the handshake success. If we provide a self-signed certificate for random ASCII string, won't the browser just reject to establish the connection?

For Cloudflare, they can prove their authority of ech-cloudflare.com and terminate the TLS connection for us, but for individuals with only single domain mydomain, isn't ech.mydomain or whatever.mydomain the only choice to get a widely-accepted certificate from providers like Let's Encrypt?

In this way, outer SNI ech.mydomain is co-related with hidden service service.mydomain according to etld mydomain, which is not random like alsdkflksjdflksjdf.com (outer) and service.mydomain (inner).

I believe that should be true, however, be aware that most ISPs / powers-that-be just use port, since the vast majority of HTTPS traffic is on port 443.

Since my ISP already blocked 80/443/8080/8443 port unconditionally, users have to use different port at high range like 12345 at the beginning. There are widely reported cases of my ISP that they detected web service in ports like 5000 (e.g. NAS's admin page) and contacted the user for regulation, so the port-443-only assumption is not relevant in my scenario.

Can you help me understand this better? The spec doesn't mention the word "replay" at all so I would be surprised if this was a design goal.

The "replay" detection of web service can work in this way:

1. ISP sniffed out SNI service.mydomain:port from HTTPS handshake;
2. ISP visits service.mydomain:port (path is encrypted in HTTPS, so can only view index page).
3. A welcome page is shown -> web service found.

For HTTP, everything including path and response HTML can be detected. For HTTPS, only domain can be detected from SNI and certificate CN in handshake. For HTTPS with ECH (with outer SNI), only ech.mydomain can be detected, and visiting this domain will be rejected. If HTTPS with ECH (without outer SNI), then nothing can be found (looks like TLS on top of non-HTTPS applications).

ISP does not need outer SNI to do this. :) IP address is even more informative.

You're right, so the point of ECH here is to completely hide the entrance to a web service (only ech.mydomain can be found), or in my expectation to completely behaves like non-HTTPS traffic.

---

Urrr, I found something contradictory in my expectation: If a certificate from recognized CA is always required, then how can we provide such a recognized certificate without outer SNI / with empty outer SNI? In RFC it seems authentication on outer SNI is required for client:

6.1.7. Authenticating for the Public Name
When the server rejects ECH, it continues with the handshake using the plaintext "server_name" extension instead (see Section 7). Clients that offer ECH then authenticate the connection with the public name, as follows:

The client MUST verify that the certificate is valid for ECHConfig.contents.public_name. If invalid, it MUST abort the connection with the appropriate alert.

If the server requests a client certificate, the client MUST respond with an empty Certificate message, denoting no client certificate.

In verifying the client-facing server certificate, the client MUST interpret the public name as a DNS-based reference identity. Clients that incorporate DNS names and IP addresses into the same syntax (e.g. [RFC3986], Section 7.4 and [WHATWG-IPV4]) MUST reject names that would be interpreted as IPv4 addresses. Clients that enforce this by checking ECHConfig.contents.public_name do not need to repeat the check at this layer.

Note that authenticating a connection for the public name does not authenticate it for the origin. The TLS implementation MUST NOT report such connections as successful to the application. It additionally MUST ignore all session tickets and session IDs presented by the server. These connections are only used to trigger retries, as described in Section 6.1.6. This may be implemented, for instance, by reporting a failed connection with a dedicated error code.

[christaikobo]

Maybe, but enabling support for a module here would be akin to advocating the capability, I feel like.

First of all, I see your concerns here. Caddy is a very popular tool and TLS is such a core component in its functionality that any change to it will have massive implications to the whole HTTP ecosystem.

On the other hand, you guys must have statistical metrics about the percentage of users who just use your docker image or download precompile binary from official website/github, as opposed to the percentage of users who use online xcaddy to build custom caddy with addons.

(Granted not all users use online build tool, but not all users download official docker image or precompiled binary either, so that kinda evens out)

And I would guess the ratio is quite extreme. My personal thought on this is if a very small percentage of users are willing to go out of their way to experiment on this with awareness of it being potentially breaking, the influence on the whole ecosystem should be limited. What is often harmful to the ecosystem is usually wrong defaults, which is not the case here.

[mholt]

@Helanzy

The "it can be random domain" statement here is what I don't understand all along. ... from my understanding we need a certificate signed by acknowledged CAs for outer SNI, to make the handshake success. If we provide a self-signed certificate for random ASCII string, won't the browser just reject to establish the connection?

Yes, so to clarify, it needs to be able to get a valid certificate for the retry mechanism to succeed -- you may use any random domain you own, I don't see why the contents of the domain are significant? For example, you quoted me saying: "Or you might own alsdkflksjdflksjdf.com" -- that is a perfectly fine domain you can use for your outer SNI if you want to. It doesn't have to be, like, helanzys-services.com. The domain can be a totally random string.

but for individuals with only single domain mydomain

And again, I'm stuck on why is this a requirement/restriction? Even if it is, what's the matter with that? Individual services are still masked behind a generic, not-service-specific domain.

The "replay" detection of web service can work in this way:

1. ISP sniffed out SNI service.mydomain:port from HTTPS handshake;
2. ISP visits service.mydomain:port (path is encrypted in HTTPS, so can only view index page).
3. A welcome page is shown -> web service found.

What? I follow step 1... and step 2... but step 3? The point of ECH isn't to hide web services. They are published publicly! The point of ECH is to make it difficult for wiretappers to know which service you are accessing.

@christaikobo Actually, we have no such information. We tried to count that once upon a time, but everyone lost their minds so we stopped.

---

Anyway, at this point, the claim that "ECH without an outer/public name is more private" has not been substantiated/proven, so I will not be implementing that at this time, to prevent potential harm to users and the ecosystem.

(Note I said, "at this time" -- if the claim is proven or demonstrated in a real world scenario, then we can revisit this, and see if it is worth the trade-off of making the system more brittle.)

[christaikobo]

Anyway, at this point, the claim that "ECH without an outer/public name is more private" has not been substantiated/proven, so I will not be implementing that at this time, to prevent potential harm to users and the ecosystem.

(Note I said, "at this time" -- if the claim is proven or demonstrated in a real world scenario, then we can revisit this, and see if it is worth the trade-off of making the system more brittle.)

I fully understand and respect the decision from the dev. Love all the work you guys have been doing. Although a consensus is not reached this time, caddy team's swift response to industry evolution and open attitude towards user input are much appreciated.

[gstrauss]

@mholt wrote:

I feel like the argument for an empty outer SNI is still beating around the bush. What's the actual privacy benefit that is so substantial it is worth making client connections brittle? So far, the claim remains unsubstantiated. What am I missing?

Perhaps those specifying "no outer SNI" are spitballing a "solution" without fully describing the problem they are trying to solve with that "solution"?

I understood only one attempt to explain "no outer SNI": I think @Helanzy is suggesting that a server configured for ECH which accepts TLS ClientHello without the TLS SNI extension, but with the TLS ECH extension might not be detected and blocked by an ISP as a web service. If that is what @Helanzy is suggesting, then I would like to see a documented example. That would be suggesting using TLS ECH to work around "all ISP-detected web services getting blocked" -- until the ISP wisened up -- even though that is not directly a privacy benefit of TLS ECH.

---

For others, you can have a wildcard cert for *.my-generic-domain.com as your public server name. Use random words prepended to .my-generic-domain.com and you can have lots and lots of different public SNI names coming in as requests, enlarging your anonymity set to lots of different public SNI names (though with shared domain suffix). The inner SNI can be anything else you like.

You can have a self-signed cert, or certs signed with a private CA (or private corporate CA), with lots of different SAN names, including spoofs of names of popular sites. If the CA signature does not validate or is not trusted by a given client, then the client should not trust the certificate, but is that a concern for you for your ECH public name? It could be a misconfiguration. It could be corporate malware producing spoofed certificates for all sites so that the corporate malware could proxy (and view) all HTTPS traffic for the specific corporate clients which trusts the corporate CA. Other than the situation where your ISP is state-owned and requires an explicit list of state-sanctioned CAs, most other ISPs are not able to enumerate all possible CAs.

---

If you want to have "secret" sites reachable only via ECH, and you do not want to expose retry configs, then that might provide an incremental level of privacy.

I am a lighttpd developer. lighttpd implements TLS ECH with the ability to (attempt) to hide hosts configured in lighttpd.conf to be explicitly accessed via ECH-only. I implemented the feature of ECH-only hosts in lighttpd as an experiment so that others could try it out and help substantiate whether or not it is measurably useful.

IF the ECH public key is distributed out-of-band instead of via public DNS, then an ECH-only service accessed could be "more private" than a typical service accessible via ECH and non-ECH connections. State-level actors with broad surveillance capabilities can still detect that you are accessing a "secret" service located at a given IP, but might not know specifically which service, and won't not be able to spider the "secret" service without the public key. (Requiring stronger authentication, which is not HTTP Basic Auth, to access the "secret" site is also strongly recommended.)

[Helanzy]

If that is what @Helanzy is suggesting, then I would like to see a documented example.

@gstrauss There are some discussion in forums [1] [2], and a blogged example that I'd like to specifically analyze: Blog (Translation):

The author's network got suspended by their ISP for hosting following unlicensed domain/port detected in ISP's feedback:

1. wow.wangwudi.com
2. https://wow.wangwudi.com:8443

Where:

1. wow.wangwudi.com is domain of a self-hosted World of Warcraft server resolved to author's residential IP, hosting no HTTP(s) service.
2. :8443 is a remote admin page of a router.

Therefore, wow.wangwudi.com:8443 is an unreasonable combination, which is reported to derive from scanning the ports on author's residential IP. wow.wangwudi.com is likely to be leaked from DNS requests generated in DDNS according to investigation.

Here we need to bring out the concept of ICP License. For domains to legitimately host services in China, they must be granted with ICP License. However ICP licenses for domains resolved to residential IPs are practically unobtainable.

So in summary, when a domain is detected to associate with a residential IP (either via SNI or DNS), and with port scanning a web service is found, then ISP would generally suspend the network for the user.

Interesting; so, you're suggesting that some residential ISPs block all TLS handshakes with a non-empty SNI? That seems odd, since if they want to block all HTTP, they should be blocking based on ALPN and/or port, not SNI. Are you sure that's correct?

@mholt Since ISP needs to check whether a domain is holding a license, SNI instead of ALPN is preferred.

Right, but this doesn't explain why no SNI is better than an arbitrary non-empty one.

Or you might own alsdkflksjdflksjdf.com at your residential IP. 🤷‍

but for individuals with only single domain mydomain

I'm stuck on why is this a requirement/restriction?

Burden to maintain one more domain aside, neither ech.mydomain nor alsdkflksjdflksjdf.com is a licensed domain, so once detected they plays no difference - you can reject any connection on these domains to not provide service, though. For ECH without outer SNI, then ISP cannot find any domain to check whether it's licensed or not.

---

A concern for it lies on implementation side. Since we cannot provide a valid license for empty SNI, server may send back HelloRetryRequest or reject it. But for client side, what will happen with empty SNI is not guaranteed. Clients are more likely to not omit the SNI extension in handshake, or just reject with error.

[gstrauss]

@Helanzy thank you for the details. While I encourage your research, the use case you presented is unrelated to "implementing TLS ECH according to the draft specification".

The present answer to @mholt's question: "What's the actual privacy benefit that is so substantial it is worth making client connections brittle?" is that there is no direct privacy benefit which has, as yet, been described.

[mholt]

@christaikobo

I fully understand and respect the decision from the dev. Love all the work you guys have been doing. Although a consensus is not reached this time, caddy team's swift response to industry evolution and open attitude towards user input are much appreciated.

Thanks for holding a respectful conversation, even though I'm sure it must be frustrating. :)

@Helanzy

Thanks for the detailed use case. That's very helpful, as I'm otherwise unaware how that works in China.

@gstrauss

I am a lighttpd developer.

First, thank you for chiming in! Feedback from another web server developer implementing ECH is as about as valuable as it gets for me right now. It's good to know I'm not alone. 😅 Thanks for bringing the focus onto the question at hand and summing things up for me with your knowledge.

So, at this point, my understanding is that:

• It can be desirable to keep which service(s) a host is operating private from parties who are able to observe all incoming and outgoing network activity as it happens (this includes, and is not limited to: origin and destination IPs, number of bytes, protocols used, any unencrypted data, and patterns of encrypted data).

• Having no outer SNI is not necessarily more private with respect to ECH, but if the desire is as stated in the last bullet point, omitting all plaintext server names would be beneficial.

• However, as a certificate is still needed for the inner SNI, a certificate must be procured by the server. This requires DNS records pointing to the machine, which eliminates pure anonymity, OR a DNS challenge record created to prove control over a domain. These records are (generally? always?) created in the clear and at specific times that can be observed on the network.

• Even if A and AAAA records don't exist for the inner domain, an observer of the network can correlate certificate issuance times with certificate transparency (CT) logs to ascertain which domain is being served at the observed host (even if timestamps aren't precise, it vastly narrows the search space).

• Thus, keeping the host's service(s) truly anonymous would require the use of a private/internal CA, that does not publish to CT logs, and whose root would have to be trusted/installed by any/all clients who wish to connect to such services. In other words, as the operator of the private service, you would have to: 1) distribute ECHConfigs in such a way that no adversary could discover, 2) create no public DNS records, 3) obtain no publicly-trusted certificate, 4) operate your own internal/private CA, and 5) ensure the CA's root is trusted by all clients of your service.

Clearly, this falls well beyond the scope of ECH, although ECH would be a crucial component of such a setup. This is the ONLY case I can think of where omitting an outer SNI would provide the privacy benefits being requested (I think); it would essentially eliminate all public traces of the service(s).

(Let me know if I have misunderstood or misrepresented anything!)

[Gunni]

You may be forgetting Wild Card DNS records with wild card certificates can completely mask which domain names are in use.

[gstrauss]

@mholt, I think you have expressed well that TLS ECH can be part of solutions making the web more private, but that there are many, many more details. Anyone looking at TLS ECH as "the solution" to a problem probably ought to first focus on VPNs, proxies, and other anonymity services before considering TLS ECH.

The Caddy feature for auto-procurement of certificates may leave fingerprints which make hiding the existence of TLS ECH-only hosts more difficult, ... if ECH-only services are even something that can be reasonably achieved (TBD).

---

Regarding TLS ECH implementation, I have not looked into the recently added Go support. Still, I want to highlight a potential privacy knob. There is a random byte in the ECHConfig which can help optimize selection of the key on the server used to decrypt the TLS ECH inner, and this feature is desirable to large CDNs. The RFC draft recognizes that this byte is a possible tracking vector. The defo.ie patches for OpenSSL include an option for ECHTrialDecrypt to enable trial decryption using a list of keys on the server. Whether or not to use the ECHConfig random byte -- which should be changed when ECHConfig keys are rotated -- is something to consider, as is whether or not to avoid using random byte (by always pinning to same value) and to force trial decryption to find matching keys.

lighttpd TLS ECH doc includes some notes on using OpenSSL and BoringSSL clients to generate ECHConfig:
https://wiki.lighttpd.net/TLS_ECH
OpenSSL and BoringSSL currently have slight differences in the precise format (wrapping) the generated ECHConfig.
The doc also includes an example of how to test TLS ECH using the cURL client and using private ECHConfig which are not published in DNS.

ECHConfig key rotation should be frequent (at least multiple times per day) and that adds work for servers, including Caddy, to retrieve and rotate keys, including possibly supporting recently-rotated keys for a short period of overlap.

Additional reference: https://github.com/defo-project
Stephen Farrell (@sftcd) has been one of the primary implementers developing and submitting patches with TLS ECH support to various projects. He is very approachable if you have questions about TLS ECH.

[sftcd]

Happy to help out with ECH here if I can - just ping if there's something I can help with.

IIRC the next golang release was supposed to include the ECH server code, so it should be there now or soon, and our experience is that it's not that hard to ECH-enable a server once you figure out where to put the few new ECH API calls. (But still happy to help with that if help's useful.)

[mholt]

@Gunni

You may be forgetting Wild Card DNS records with wild card certificates can completely mask which domain names are in use.

I thought about those, but they would still be subject to the same issuance-time correlation side-channel of any public certificates.

@gstrauss

Thanks, really great information there!

There is a random byte in the ECHConfig which can help optimize selection of the key on the server used to decrypt the TLS ECH inner, and this feature is desirable to large CDNs. The RFC draft recognizes that this byte is a possible tracking vector.

Are you referring to the config_id?

ECHConfig key rotation should be frequent (at least multiple times per day) and that adds work for servers, including Caddy, to retrieve and rotate keys, including possibly supporting recently-rotated keys for a short period of overlap.

That's the plan -- although, I don't know about multiple times per day 😅 I know the spec recommends "frequently" rotating keys, but we can probably manage either way.

@sftcd Nice to meet you, thanks for popping in!

IIRC the next golang release was supposed to include the ECH server code, so it should be there now or soon, and our experience is that it's not that hard to ECH-enable a server once you figure out where to put the few new ECH API calls.

Indeed, Go 1.24 supports ECH on the server-side, and its API is very simple. Basically just set this field in a tls.Config.

I'm currently designing the config API for enabling ECH ub Caddy, but then I'll surely have questions as I come to more ECH logistics, such as key rotation. I really appreciate your willingness to chime in 👍

I'll also hopefully have a WIP PR to share in the next week or two.

[gstrauss]

There is a random byte in the ECHConfig which can help optimize selection of the key on the server used to decrypt the TLS ECH inner, and this feature is desirable to large CDNs. The RFC draft recognizes that this byte is a possible tracking vector.

Are you referring to the config_id?

Yes.

[Gunni]

@mholt

I thought about those, but they would still be subject to the same issuance-time correlation side-channel of any public certificates.

Yes but you might f.ex get a wildcard cert for example.com, and then only allow specific subdomains to connect, then close the connection for others.

[sftcd]

On 20/02/2025 18:29, 古大羊 wrote: In my view, the ECH RFC standard is one aspect, while client-side support (such as in browsers) is another. Therefore, the claim that ECH can hide external domain names or use third-party domain names that do not belong to it is unrealistic. Server-side implementation must not only adhere to the ECH specification but also consider the actual compatibility of clients (browsers); otherwise, it is merely empty talk.

Not sure I'm following the above quite, but FWIW, browsers (FF and chromium-based) have supported ECH for quite a while, and cloudflare turned it on in their services (again) last October. So in terms of protocol level interop, things work just fine. There are some deployment wrinkles still but I'd expect those to improve over time. We did an interop report recently [1] calling out some of those. (BTW [1] didn't include a golang server, but we'll likely add one when we revise that in future.) [1] https://test.defo.ie/ech-interop-report.pdf In terms of how much of a privacy benefit one gets from ECH, yes that'll depend on which outer-sni (public_name) is used, how that maps to a set of names usable as inner-sni via that, how IP addresses are mapped to those names and how identifiable DNS queries may be. My argument would be that even if all of those involve unknowns, we're still doing the right thing if we make it harder to abuse the SNI value by encrypting that (via ECH). It's an incremental improvement in privacy, but a worthwhile one that should enable others in future. Cheers, S.

[mholt]

Thanks @sftcd -- great info!

I'm making progress on this, but recently hit a snag... the crypto/tls package expects us to hard-code the ECH keys into a TLS server config, rather than provide a callback function like it lets us do for certificates. That means the keys can't change during the TLS server lifetime, meaning they can't be rotated.

I hesitate to proceed with a design that doesn't support key rotation, because then if it ever is possible in the future, I may have to refactor a lot of code. (Or not; but it's uncertain.)

I might be totally doing something wrong or overlooking something obvious, but I opened an issue upstream just in case: golang/go#71920

[gstrauss]

@mholt, I do not know enough of the Go language to comment on the bug report, but want to highlight the importance of having the ability to support multiple ECH keys for trial decryption. Whether multiple keys are due to key rotation of the same host, or if multiple keys are due to multiple different hosts, sometimes trial decryption is necessary to decrypt the TLS ECH inner, and you will not be able to read SNI from the TLS ECH inner until after TLS ECH inner has been decrypted, which is after (potentially multiple different) decryption keys have been tried. Does Go provide a mechanism to configure a list of multiple decryption keys? Your bug report notes that Go does not provide server side support for dynamic reconfiguration of TLS ECH keys, and if what I am trying to describe is another limitation of the Go TLS ECH server-side implementation, it would be nice to provide that feedback so that the Go devs can suggest a solution to both issues.

[mholt]

@gstrauss Thanks -- Go does allow configuring multiple keys: https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys

The 'EncryptedClientHelloKeys' field lets us specify multiple, which we will be doing in Caddy.

[gstrauss]

https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys

	// If EncryptedClientHelloKeys is set, MinVersion, if set, must be
	// VersionTLS13.

Does that mean that servers on top of Go will require TLSv1.3 as minimum protocol -- and will not be able to support TLSv1.2 -- or does it mean that TLS ECH is available only with TLSv1.3, as TLS ECH is an extension which requires TLSv1.3?

[sftcd]

I might be totally doing something wrong or overlooking something obvious, but I opened an issue upstream just in case: golang/go#71920

The bug report looks valid to me (modulo my relative golang ignorance:-). In 'C'/OpenSSL you have the difference between the SSL_CTX context structure and the per-connection SSL structure so can handle ECH key rotation in the SSL_CTX, but I don't see a similar thing at https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys In any case, handling ECH key rotation similarly to TLS server private key/cert re-load seems like a good idea, esp given that cloudflare are rotating ECH keys hourly, its not unlikely others will emulate that.

As an aside, that https://pkg.go.dev/crypto/tls#Config thing looks like it might grow up to be like some of the internal OpenSSL structures (i.e. many too many fields) - avoiding that might be a good thing, were it possible:-)

And just for completeness - I very much doubt you'd want to, but haproxy has yet another way to handle key rotation for TLS server keys (and, in my PoC integration ECH keys) - you connect to a stats port and use socat to load in new private/public values. IIUC their logic in doing that is they don't want to depend on reading the disk any more once haproxy has started (so they continue in the face of some errors).

[mholt]

If anyone would like to try my WIP implementation, here it is: #6862