Last post of 2025, yay. ٩(◕‿◕)۶

A reverse proxy like Caddy is handy for automatically generating HTTPS certificates with auto-renewal, but it does have one problem.

endpoint.of.your.s3.domain.tld {
  reverse_proxy garage:3900 
}

The scenario

I have a DNS record A *.foo.bar.com so I can add or remove subdomains without touching DNS every time.
I also don’t want to expose every hostname in DNS and make scanning even easier.

Once I set up something like https://whoops.foo.bar.com, I am happy to use the service through the internet.
The weird thing is that, even though I don’t expose A whoops.foo.bar.com as a DNS record,

These bots can find the service almost instantly.
bot-instant-find

Meet Certificate Transparency

After some rough Googling, one phrase kept popping up in the discussion:

  • Certificate Transparency

Certificate Transparency is a mechanism for auditing certificate mis-issuance or CA fraud by publishing the record so everyone can audit and verify it.
For each certificate issuance, a CT log entry is appended and made publicly available.

As a result, certificate issuance on your domain becomes public information.
Tools like crt.sh can easily get the list of entries under your domain.
crt-sh-find

As HTTPS is basically a modern-day requirement, and HTTP-01 is widely adopted as a default challenge (because it only accepts explicit hostnames),
it feels a bit like sending an invitation letter to these bots.

That explains why, even if I don’t create a DNS record explicitly, the bots can still magically find my domain.
The trick they use is watching CT logs. When they find a new certificate, they scan it for vulnerabilities right away.
They don’t even need a DNS scan for this.

The workaround I ended up using

  • Wildcard Certificate

Instead of explicitly stating the full domain used in the HTTP-01 challenge
used by most automatic certificate setups,
I use the traditional DNS-01 challenge.
It is the only one that can generate wildcard certificates.

My instance uses Pangolin .
They have a guide for generating wildcard certs.
Although this method has some problems:

  1. The DNS provider is required to provide DNS API
  2. The DNS provider API key is on your proxy, so if that key leaks someone could mess with your DNS
  3. Wildcard certs are kind of hard to manage if you have more than 1 machine using the cert
  4. Leak of wildcard cert key is a big problem

But it worked for me. No more bot scanning after the change.
It also has one extra bonus: I don’t have to wait for Let’s Encrypt to generate a cert every time I add something to the domain.
Bots could still see the wildcard certificate I generated, but nothing more.

Yay ٩(◕‿◕)۶.