You can use a free tool like jmeter to simulate load testing. Or you can shell out about $100k for SOASTA.
But I’m seeing threads on discourse forums where people have sites with 25k+ Reg’d users and are able to manage - surely they have as many concurrent users as we do?
I’m probably not understanding the problem as zikzak stated it. Is the issue a high number of concurrent users converging on a single topic, a high number of posts in a single topic, or both?
Basically what I’m asking is if the load on the server resources are worse when activity is taking place on a single topic vs when the same activity is spread across multiple topics of similar size.
If it is the same, then there’s no way we’re the first to encounter this issue.
The log out stuff smells like Redis issue as that’s probably where your sessions are stored.
People with much smaller numbers of users have encountered similar problems under similar situations. It’s something about gameday type threads, but I’ve never been able to conclusively identify the problem. Here’s the graphs that show whatever it was that happened. I was monitoring the server during this and the bottleneck was 100% I/O Wait.
Also, to keep things in perspective, neither option is end of the world. We had significant slowdowns during the last debate for less than 5 minutes. And if we go with a super limited number of available threads, I only anticipate that lasting for one evening.
Another part of my thinking is that Jeff Atwood (Stack Exchange guy, Discourse founder) has repeatedly stressed how much of a performance hit there can be with lots of large, open, active threads.
eta: The more recent load spikes were backups and forum rebuilds. They’re normal.
What about the reddit approach of just breaking up into multiple threads?
Could have a permanent EDAY MISC thread and otherwise EDAY POTUS and EDAY CONGRESS threads that get archived to read-only every X posts.
So yeah first option is that someone who works on this sort of stuff (eg JMakin who has already volunteered to take a look) finds a reliable technical change that can be made that keeps things as is. But as others have mentioned keeping it to low risk changes for sure given the timing.
Failing an easy technical solution, I would say perhaps make an entirely new Category called “Election Game Day” or something with the X threads for that day/evening, and close off access to all other categories.
Make one of those threads a “Help/Site Tech Stuff” thread so people can pop in if there are issues or there is some reason it is needed to open back up another area.
It also might be helpful to write a bot that posts “HOLD!!!” every 5 minutes.
2 Likes
Speaking of - did you ever get your site scalable?
Yea but I’m terrible and a moron so don’t take my word for it. I have no idea how this stuff works, but I do have a little experience with managing and provisioning servers at scale and deploying/managing/monitoring applications on those servers. One benefit to my job, which we dont have here, is that if I am faced with a resource problem (not enough cpu/memory/disk) then I can usually just throw more resources at it until it works.
Zikzak is far more knowledgeable than I could be on this topic though. I want to brush up this weekend.
My vote, is for doing the safe measure, and trying to look at ways we can manage it in the future. Because if we grow we’re gonna just keep running into this.
1 Like
This shit is way above my knowledge level so I fully support whatever you guys think makes the most sense.
Ugh no I’ve been lazy AF and pandemic traffic is down to about 250 players during peak hours but um maybe post election when I have more “free time”
1 Like
I’ve noticed that when I post a hosted image, the forum will eventually create a local copy. Is that something that could create slowdowns during high loads?
Not really. And I think the software considers that a low priority action that it only does when resources are available.
Would it help to shut off the twitter bots on election night?
No, they don’t even run on the server.
I was thinking about having 4 fewer users accessing threads and posting.
99% sure they don’t “read” threads and just do a post (POST) and tiny cpu usage.
2 Likes
If you do this, can we get a gambling thread? Need the last shot at getting dumb TrumpBux when he is crushing states that are overcounting in-person votes.
1 Like
I am currently asking the Discourse folks for their input. The hosting provider swears up and down that everything is fine with their hardware and there has been no unusual activity on shared resources. I’m not entirely sure I believe them.
looks like you are saving images on your server and not on s3…