I’ve been thinking about ways to keep the site responsive and useful on election night and I’ve pretty much convinced myself that we need to shut down everything except for a few dedicated threads. In Star Trek terms, I want to move all power to forward shields. My reasoning follows, which you are encouraged to disagree with if you think I have something wrong, because I probably do.
My rough (and likely not entirely accurate) understanding is that at any given time the majority of the active forum lives in an in-memory Redis cache. If there are lots of threads being viewed they will be taking up lots of the available cache space, and the bigger those threads are the more space they take up. One person looking at some old, inactive conversation from last year ties up a lot of resources, relatively.
So I want a lot fewer threads available next Tuesday. Like maybe 5.
What I am proposing is a new Election 2020 category with a few dedicated topics, and sometime Tuesday all other categories will be completely (but temporarily!) removed from public access. We will be a forum with 5 (or 4, or 6) gameday threads and nothing else. When traffic dies down and the load becomes more manageable we can bring other categories back online. Eventually at some point the gameday threads can be merged back into the main Politics category.
So, good idea? Dumb idea? Am I missing something?
Proposed topics, clever names TBD:
POTUS
Senate
House
State & Local
Misc. chatter
I would love to help in any way I can. My first thought is if it’s possible to have the entire forum be functional we should strive for that. But, if it isn’t possible, then that’s just a reality we have to deal with.
Can you simply increase the size of the redis cache?
In redis.conf it is possible to set maxmemory to 0 which means it will utilize resources until there aren’t any and I guess then the site just crashes, so maybe that’s not what you want. But you could attempt to overprovision, and monitor memory usage of the cluster/whatever it’s hosted on and if it’s going up past a certain threshold, flip a switch and only enable certain threads.
Also there has to be some way to disable the “live db updating feature” I. E. every few seconds when you scroll through a thread it makes a db call that marks posts as read, see if you can turn that off.
It’s set in the Discourse conf and we’re already at recommended max of 25% of available system memory (4 out of 16 GB right now).
I’m sure that’s possible somewhere, but it would take me hours to even begin to know where to look and how to do it. The codebase is very large and opaque, and I am not much of a programmer.
Not exactly. I do know that what cripples us when we hit significant slowdowns is iowait, but there are no corresponding actual disk read/write spikes when it happens.
I probably wouldn’t mess with the code or recommended settings this close to the election. More likely to make things worse/unstable, imo. (Edit: This goes for whoever is running “the simulation” - no surprises please)
If you have an easy way to scale up the underlying hardware - I’d recommend that (but I assume it isn’t easy).
Your idea seems worthwhile to me. I doubt anyone is really going to complain about the other threads…
Its not really about the feature itself its more its frequency like its within 1-3 seconds, seems like there should be a way to throttle the database hit to 10-30 seconds. But whatever it just seemed really obvious to me and if you’re not seeing it forget it.
Still think an “official” discord channel makes sense though.
We’ve already done that and are at the top end of basic droplets on DigitalOcean. The only place to go from here is a dedicated virtual machine that would triple our monthly cost and be annoying to deploy and then have to scale back a couple days later.
I agree, but I don’t think many people will use it. There have already been at least a couple attempts that have failed to attract much use.
I bet the site will be fine. They have to have much bigger sites than ours. The problem when Trump got pozzed was the whole internet went down on the west coast or something.
It should be fine - I mean we only have ~50 active users. Say it bumps up to 100 on election day, and add another 100 lurkers.
That’s still well below the traffic of a production grade website - I can’t imagine we should have issues unless the discourse software is just absolute ass and not intended for high traffic, which I know is not true.
I’ll try to do a little digging and educate myself on this topic this weekend. This stuff is basically what I do for my job.
I thought everything was going fine during the last debate until everything ground to a halt for 4 minutes and lots of people got temporarily logged out. I’ve made some additional changes since then but still haven’t been able to figure out exactly where the problem was.
What makes this so annoying to plan for is that it’s going to be a one-off event. I don’t have the opportunity to make adjustments and see what works. That’s why I’m considering limiting the entire forum to just a few threads for one evening. It’s one shot with no do-overs.