Recent Forum Issues - August 2021

Discuss issues and requests related with the forum software
User avatar
tank
Posts: 3122
Joined: 28 Sep 2013, 22:15
Location: CarrolltonTX
Contact:

Re: Recent Forum Issues - August 2021

Post by tank » 08 Oct 2021, 10:17

right. but i cannot stress that there is noo measurable difference at the server. cpu, memory, n disk usage, all show minimal load. this is to do with performance at certificate authoritties, geography and other unknowns. if u watch the netwok traffic of the page loads. specifically the waterfal. you see that afterthe handshake the communication is very snappy.

for giggles i may just for the sake of proving this out build and migreate us to a diff datacenter with our provider. while it would make nooo sense for it to matter, its almost the onlything i havent tried.

i am morally against a paid certificate issuer, but its the last thing i can try. but there is 0 and i mean none evidence that it matters. if it did big tech companies would do it or not have seen issues. none the less we have the funds to buy one for a year and if moving data centers doesnt help its worth trying
We are troubled on every side‚ yet not distressed; we are perplexed‚
but not in despair; Persecuted‚ but not forsaken; cast down‚ but not destroyed;
Telegram is the best way to reach me
https://t.me/ttnnkkrr
If you have forum suggestions please submit a
Check Out WebWriter

Rohwedder
Posts: 7510
Joined: 04 Jun 2014, 08:33
Location: Germany

Re: Recent Forum Issues - August 2021

Post by Rohwedder » 08 Oct 2021, 10:23

Hallo,
it took me 5 minutes to access the forum. This forum does not deal in money or stocks. So what is the point of this Https? Just switch back to Http!

just me
Posts: 9406
Joined: 02 Oct 2013, 08:51
Location: Germany

Re: Recent Forum Issues - August 2021

Post by just me » 08 Oct 2021, 10:34

The forums responded as usual for me the whole German morning. Since I returned about 15:00 o'clock German time after a break response times are unacceptable. I noticed a similar behaviour yesterday.

User avatar
kczx3
Posts: 1640
Joined: 06 Oct 2015, 21:39

Re: Recent Forum Issues - August 2021

Post by kczx3 » 08 Oct 2021, 11:22

@tank
Is there a way for us to engage with these authorities? We can't just sit here and wait twiddling our thumbs.

sirksel
Posts: 222
Joined: 12 Nov 2013, 23:48

Re: Recent Forum Issues - August 2021

Post by sirksel » 08 Oct 2021, 12:15

@tank, you'd know better than anyone since you can see the traffic. For what it's worth, and in case it helps at all, your theory seems consistent with what I'm seeing on my end. Without delving into any logs or anything, every time I've I looked down after reading your post, I notice Chrome is stuck at "Establishing secure connection..." in the lower left status indicator when the timeout occurs. Maybe that's the only status message that could be displayed when secure traffic times out?... I've just been noticing it more after reading your explanation.

I also tried flushing my DNS resolver cache to see if that helped... it didn't.

I never knew how much I relied on the forums until accessing them was challenging. Thanks so much for everything you're doing to help solve this perplexing issue!

User avatar
tank
Posts: 3122
Joined: 28 Sep 2013, 22:15
Location: CarrolltonTX
Contact:

Re: Recent Forum Issues - August 2021

Post by tank » 08 Oct 2021, 14:41

ca are determined by your OS unless you use firefoc. cert issuers are trusted by CA

CA are the mechinism by which the internet verifies the identity returned in dns resolution.

after that encryption keys are exchanged so that the client n server may communicate and decrypt traffic.

as a client your identity is verified by cookies and logons.

i can only tell u all that what we are seeing here and to some extent is that CA are having some kind of issues. the cert issuer in this case 'letsencrypt' isnt involved as all they do is issue the cert key. public and private. the signatures of those certs are what the CA USE TO VALIDATE OUR IDENTITY. but the list of thos is OS specific and altho trust owners can be modified that practice is usually done in corp networks.

the decertification of old CA and phase out of TLS 1.1 likely caused unexpected overtaxing of CAs.
https://duckduckgo.com/?q=tls%201.1%20end%20of%20life&ko=-1&ia=web
most Support ended about same time.

this is a complex issue that the industry has been kicking the can down the road on for 6+ yrs that i know of.

its easy to blame site owners or letsencrypt since they publicly announced changes. they are open sourcers. but most of the others news flew under the radar as it looks bad for large corp interests.
We are troubled on every side‚ yet not distressed; we are perplexed‚
but not in despair; Persecuted‚ but not forsaken; cast down‚ but not destroyed;
Telegram is the best way to reach me
https://t.me/ttnnkkrr
If you have forum suggestions please submit a
Check Out WebWriter

swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Recent Forum Issues - August 2021

Post by swagfag » 08 Oct 2021, 15:03

still not buying this certificate mumbo jumbo and still waiting on that laymans terms explanation of whats actually wrong

u claim its some kind of a "certificate handshake" thats taking too long, thats keeping connections open for the duration, thats causing ddos-like issues.

how is that possible? im using ms-edge which shows the following certificate chain:
image.png
image.png (47.29 KiB) Viewed 29582 times
which is the same as the one user @jNiZm showed
which is the same as the one stackoverflow.com apparently uses as well:
image.png
image.png (9.62 KiB) Viewed 29582 times
why then can i browse stackoverflow unimpeded? (expected: i wouldnt have been able to, same as the ahk forums)
why can i browse the ahk forums for what seems like bursts of 10-20 secs unimpeded after getting past the timeout errors? (expected: id get past the timeout error once, then browse for as long as i wanted/user remained logged in/browser remained open/internet remained up)
why can i not afterwards(after the 20s "grace period" has elapsed) until, presumably, another handshake has been successfully negotiated?? (expected: see above)
why does it seem as though that handshake isnt being performed every time i visit any particular link(? (expected: "why dont i get long loads on every page i open?")

u claim LetsEncrypt isnt involved in the process. how is that possible? what then certifies the validity of ur ahk forum certificate if not LetsEncrypt's very own Intermediate CA R3(going by the certificate chains above)???

more whys:
why can i browse the ahk forums unimpeded in the evening? (expected: i wouldnt have been able to)
why is the tls1.1 EOL link u posted, which contains references dating back more than a year ago relevant now? (the forum issues began about a week ago iirc)
tank wrote:
08 Oct 2021, 14:41
this is a complex issue that the industry has been kicking the can down the road on for 6+ yrs that i know of.
its easy to blame site owners or letsencrypt since they publicly announced changes. they are open sourcers. but most of the others news flew under the radar as it looks bad for large corp interests.
what is one to make of this post? :roll:

theres an issue. surely there must have been someone or some thing thats caused it. so identifying what that is, would clue us in as to some potential solutions.
i dont see how throwing hands up and exclaiming "yep, shits fucked, been so for the past 6 years, cant do nuttin" is at all productive

and dont take this as an attack on u. while im enjoying the noob-post-detox on account of basically not being able to use the forum at all, id prefer to get back to it at some point sooner rather than later. i think we all do

User avatar
MrDodel
Posts: 96
Joined: 28 Apr 2021, 09:03
Location: Event Horizon

Re: Recent Forum Issues - August 2021

Post by MrDodel » 08 Oct 2021, 15:41

swagfag wrote:
08 Oct 2021, 15:03
theres an issue. surely there must have been someone or some thing thats caused it. so identifying what that is, would clue us in as to some potential solutions.
i dont see how throwing hands up and exclaiming "yep, shits fucked, been so for the past 6 years, cant do nuttin" is at all productive

and dont take this as an attack on u. while im enjoying the noob-post-detox on account of basically not being able to use the forum at all, id prefer to get back to it at some point sooner rather than later. i think we all do
Sorry, you wish a tank to explain why an entire backbone lost it's shit, including faceboot, instafail,and whatshite ?

Certificate propagation.
So much universe, and so little time. GNU Sir Terry.

User avatar
Xtra
Posts: 2744
Joined: 02 Oct 2015, 12:15

Re: Recent Forum Issues - August 2021

Post by Xtra » 08 Oct 2021, 16:04

why can i browse the ahk forums unimpeded in the evening?
I'm seeing the same thing perfect at night but during the day its terrible.

-Thanks

User avatar
flyingDman
Posts: 2776
Joined: 29 Sep 2013, 19:01

Re: Recent Forum Issues - August 2021

Post by flyingDman » 08 Oct 2021, 16:44

Does anyone have this error? And is it related to our current situation?
10-08-2021 clip.jpg
10-08-2021 clip.jpg (43.93 KiB) Viewed 29530 times
14.3 & 1.3.7

User avatar
Xtra
Posts: 2744
Joined: 02 Oct 2015, 12:15

Re: Recent Forum Issues - August 2021

Post by Xtra » 08 Oct 2021, 17:49

flyingDman wrote:
08 Oct 2021, 16:44
Does anyone have this error? And is it related to our current situation?
Its been doing that for some time (6+ months) i gave up and just download the file manually and run that. Don't think it's related to the current problems but :roll:

swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Recent Forum Issues - August 2021

Post by swagfag » 08 Oct 2021, 17:53

MrDodel wrote:
08 Oct 2021, 15:41
Sorry, you wish a tank to explain why an entire backbone lost it's shit, including faceboot, instafail,and whatshite ?
Certificate propagation.
except i can browse faceboot, instafail and whatever that third one's supposed to be just fine since the outage was resolved a couple of days back. and i was also able to browse them just fine as well during the period lasting from Sep 30(when those LetsEncrypt certificates allegedly expired) until Oct 4(when the outage occurred). besides that, facebook seems to also use different CAs:
image.png
image.png (49.53 KiB) Viewed 29504 times
sooo, what gives? again, what is the relevance regarding what had happened to facebook? it seems totally unrelated as they pin it on misconfiguration changes to their infrastructure: https://engineering.fb.com/2021/10/04/networking-traffic/outage/
does the AHK forum also suffer from "misconfiguration changes to its infrastructure"? if so, could it possibly be reconfigured(like facebook did)? so it works again(like facebook already does)?

bankofamerica was also thrown around a bunch of times, which also relies on different CAs, and which i can also click through just fine

got any other examples of sites using the same LetsEncrypt certificate chain that the AHK forums use(ISRG ROOT X1 > R3) that are also stuttery?

admittedly, my understanding of network stuff is pretty pathetic but as it stands whats being claimed so far just doesnt seem to pass the smell test. answering those questions would be a good start to help clarify the issue

is there any setting i can change on my end to make it skip the certificate check altogether?? if someone wants to mitm me, then so be it. id rather have that than waste half an hour trying to submit/edit posts

@flyingDman Installer.ahk probably gets a timeout-related HTTP status code back, but doesnt actually check it. instead it tries to pattern match on the (possibly; and currently quite very probably) nonexistent/nonsense response:

Code: Select all

VersionReceived(req) {
    local w := getWindow(), latestVersion := ""
    if req.readyState != 4
        return
    latestVersion := req.responseText
    if RegExMatch(latestVersion, "^(\d+\.){3}\d+") {
        if (latestVersion = ProductVersion)
            w.opt1.firstChild.innerText := "Reinstall (download required)"
        else
            w.opt1.firstChild.innerText := "Download v" latestVersion
        w.opt1.href := "#"
        w.opt1.onclick := Func("DownloadAHK")
    } else
        w.opt1.innerText := "An error occurred while checking for updates."
}

swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Recent Forum Issues - August 2021

Post by swagfag » 08 Oct 2021, 18:51

Xtra wrote:
08 Oct 2021, 17:49
flyingDman wrote:
08 Oct 2021, 16:44
Does anyone have this error? And is it related to our current situation?
Its been doing that for some time (6+ months) i gave up and just download the file manually and run that. Don't think it's related to the current problems but :roll:
then its probably worth checking what status code it returns and why its not 200-OK(see previous post)

swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Recent Forum Issues - August 2021

Post by swagfag » 08 Oct 2021, 19:08

for example, this site https://www.jenkins.io/doc/(a popular CI/CD tool) also uses ISRG ROOT X1 > R3 for its certificate, except i dont get any timeouts when browsing it and nothing lags.
the site is ranked 20k on alexa(autohotkey.com is 30k, for reference) so lets assume it gets more or thereabout the same amount of traffic as the forums. why no timeouts? what gives?

similar site: https://blog.dnsimple.com/2021/09/letsencrypt-intermediate-expiration/ (probably also worth reading through)
similar site: https://gitea.io/en-us/ also fast

maybe of use: https://community.letsencrypt.org/t/https-very-slow-after-certificate-renewal/110509 (can try looking at the output of https://check-your-website.server-daten.de as one guy suggest. i have no idea how to interpret those numbers)
similar thread: https://community.letsencrypt.org/t/website-incredibly-slow-following-ssl-update/98089

how to check ssl times: https://help.heroku.com/NY64S5NL/how-do-i-debug-latency-issues-using-curl

Code: Select all

> curl -w "dns_resolution: %{time_namelookup}, tcp_established: %{time_connect}, ssl_handshake_done: %{time_appconnect}, TTFB: %{time_starttransfer}\n" -o NUL -s "https://www.autohotkey.com/"

> dns_resolution: 0.015000, tcp_established: 0.156000, ssl_handshake_done: 0.468000, TTFB: 0.625000
ssl time given by ssl_handshake_done - tcp_established
ah crap. looks like the forums are snappy now once again. guess we'll have to wait until tomorrow(although i certainly wish we wouldnt have had to...)

another article: https://sking7.github.io/articles/44961356.html

User avatar
tank
Posts: 3122
Joined: 28 Sep 2013, 22:15
Location: CarrolltonTX
Contact:

Re: Recent Forum Issues - August 2021

Post by tank » 08 Oct 2021, 21:22

Nice never saw that before
swagfag wrote:
08 Oct 2021, 15:03
i dont see how throwing hands up and exclaiming "yep, shits fucked, been so for the past 6 years, cant do nuttin" is at all productive
No version of that is happening. I try everything i find even when i dont think it will matter.
image.png
image.png (151.65 KiB) Viewed 29376 times
i really hope this isnt playing a role
image.png
image.png (28.2 KiB) Viewed 29376 times
We are troubled on every side‚ yet not distressed; we are perplexed‚
but not in despair; Persecuted‚ but not forsaken; cast down‚ but not destroyed;
Telegram is the best way to reach me
https://t.me/ttnnkkrr
If you have forum suggestions please submit a
Check Out WebWriter

swagfag
Posts: 6222
Joined: 11 Jan 2017, 17:59

Re: Recent Forum Issues - August 2021

Post by swagfag » 08 Oct 2021, 22:07

tank wrote:
08 Oct 2021, 21:22
No version of that is happening. I try everything i find even when i dont think it will matter.
i know u do and i know u would. which is reassuring since ure probably one of only a handful of people with sufficient expertise and insight into the forum's infrastructure, capable of getting to the bottom of this
i cant tell from the screenshot but are those troughs that appear to be always pegged at 10% CPU occurring in the evening(EDT, for reference) when the forums are snappy? or during daytime, when the forums are slow?
similar question regarding the peaks(whose patterns more closely resemble CPU activity from before we began encountering those issues)

could there be something else thats purposefully throttling connections during that period for whatever reason? cloudflare? the hosting provider? something else?

User avatar
tank
Posts: 3122
Joined: 28 Sep 2013, 22:15
Location: CarrolltonTX
Contact:

Re: Recent Forum Issues - August 2021

Post by tank » 08 Oct 2021, 22:40

i keep looking. this wwekend im going to try to move our whole setup to a new digital ocean datacenter and generate new certs again. not having a WAF scares the shit out of me but killing cloudflare at least temporarily may be something i have to try.

in the past when we had significant issues Joe or i could always find a server metric or other trackable event. with the exception of a SAN issue several weeks ago they have all been within our control.
We are troubled on every side‚ yet not distressed; we are perplexed‚
but not in despair; Persecuted‚ but not forsaken; cast down‚ but not destroyed;
Telegram is the best way to reach me
https://t.me/ttnnkkrr
If you have forum suggestions please submit a
Check Out WebWriter

MancioDellaVega
Posts: 83
Joined: 16 May 2020, 12:27
Location: Italy

Re: Recent Forum Issues - August 2021

Post by MancioDellaVega » 09 Oct 2021, 04:21

@tank
Thanks a lot for your work, now here seems all work fine :clap: :clap: :dance: :dance:
Courses on AutoHotkey

User avatar
tank
Posts: 3122
Joined: 28 Sep 2013, 22:15
Location: CarrolltonTX
Contact:

Re: Recent Forum Issues - August 2021

Post by tank » 09 Oct 2021, 14:00

shhh i didnt touch it
We are troubled on every side‚ yet not distressed; we are perplexed‚
but not in despair; Persecuted‚ but not forsaken; cast down‚ but not destroyed;
Telegram is the best way to reach me
https://t.me/ttnnkkrr
If you have forum suggestions please submit a
Check Out WebWriter

User avatar
MrDodel
Posts: 96
Joined: 28 Apr 2021, 09:03
Location: Event Horizon

Re: Recent Forum Issues - August 2021

Post by MrDodel » 09 Oct 2021, 14:49

tank wrote:
09 Oct 2021, 14:00
shhh i didnt touch it
You turned it off and on again, didn't you ? :lol:
So much universe, and so little time. GNU Sir Terry.

Post Reply

Return to “Forum Issues”