That's a crazy issue to have discovered. Maybe you could try a different reverse proxy like nginx to narrow down potential causes for the issue
this post was submitted on 06 Aug 2023
14 points (88.9% liked)
Selfhosted
40330 readers
365 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 1 year ago
MODERATORS
Yeah this smells like a bug in Caddy or something. I agree to try nginx or something else to see if it’s Caddy or if it’s something with the configuration of the host. The only thing I could think of is if caddy isn’t caching DNS responses and maybe is getting rate limited so it appears slower while it’s waiting on the DNS request but I am shooting in the dark as I haven’t spent much time with caddy.
I mentioned above- the subdomains were using HTTP/3, and the root entry does not. I don't know if it's something I have mis-configured or just HTTP/3 being new and maybe buggy. Either way, i disabled it globally and performance is the same.
I can't remember if it's enabled by default or not, but it's easy enough to enable pprof and get a helpful performance profile from /debug/pprof. See https://caddy.community/t/hangs-on-reload/12010/18 for an example.
I've found that even being unfamiliar with the codebase, it's often pretty easy to identify what part of the call stack is being slow and file a very useful performance but report in GitHub. Check out the profile and see if it leads to any obvious conclusions about why domains are so much slower. There may be some function that's trivial to cache the results of that brings things back to the expected performance.
The profiles were totally different- because the subdomain uses HTTP/3 via the quic-go library. Disabling it globally has it working at 100%
is it possible the subdomain resolves to an ip your router can't see due to nat hairpinning?
I tried to dig into that but couldn't come up with a good test. But if NAT hairpinning wasn't working right, I'd be limited to my ISPs 50Mbit, right? I could get 200+ Mbit on wifi. I also tested this from work (50 Mbit sym fiber) and subdomains always were slower. I figured out today it's HTTP/3 causing my problems. I don't know if I care to troubleshoot anymore since it's working great with http 1 & 2.
Interesting stuff, perhaps report it to the Caddy devs, h3 isn't really "stable" afaik
https://github.com/caddyserver/caddy/issues/4876
Looks like they already know, and upstream is working on it. I am curious why the sub domains get http/3 and the root domain does not.
wow, some of those comments mirror your situation exactly hah