6 Likes
Someone get spider another Skyline gift card.
3 Likes
Are we punishing him for fixing the server?
5 Likes
That was the longest 24 hours of my life!!
2 Likes
Everything is a tiny bit different my life is ruined
Thank God we don’t have to visit that OTHER SITE anymore huh??!?!?
3 Likes
Thanks for all the work that went into gettung us back up.
7 Likes
Work for DOGE
1 Like
Thanks for getting it back online. You may want to find a way to monitor disk usage or upgrade the droplet to prevent a sudden “emergency” as I know how stressful those can feel. Unfortunately the way discourse works, disk space will always increase. Keep backups handy.
Some discourse deployments I’ve done utilize aws S3 for storage instead, you dont get the same issues.
I said I’d leave things well enough alone, but I didn’t. Based on some googling around, I ran the /launcher cleanup command, which seems like a recommended thing to do after upgrading, and that recovered about 4.7GB. But then I got this message:
“Old PostgreSQL backup data cluster detected taking up 24G. Would you like to remove it?”
Crossed my fingers and said yes. So now we appear to be working still, and we’re at 68% disk usage. So barring any further complications, I’m definitely not worrying about anything except being more proactive on running minor upgrades once or twice a month.
3 Likes
Nice work spider
I see we got large old people text with the Discourse upgrade like SPE did.
The launcher is one of the best written scripts Ive ever seen, you need to do really stupid things for it to let you eff up too bad.
I think we need a backup/co-server admin both to assist with technical problems and if you are unavailable or otherwise incapacitated. We should also have a disaster recovery plan/forum, like a subreddit, where a response can be organized in the event Unstuck becomes inoperative, including a system where multiple people have relevant passwords, etc. In short, I think we should set up a disaster recovery committee. I guess SPE was used in this situation, which is probably sufficient, though a subreddit could be another backup, or 2p2. Main thing is it would be good to have redundancies and a few people with the keys. I’m busy too for the next week, but someone should drive this project.
3 Likes
I have not been able to do this, and I’m not sure why. I was able to successfully backup the forum today, but only by excluding uploads. Apparently it’s too big to backup including uploads. That’s not the worst problem in the world I guess, except that in the event of failure, we’d lose all the sweet, sweet memes that have been posted here.
Any solution other than increased disk space?
He gets one Skyline gift card every day the server is down, as a punishment.
5 Likes
Is it just a matter of paying for more disk space, or are we at some kind of partition limit?
Its been a while since ive been in DO but taking a snapshot/backup of the droplet itself every once in a while (not sure of cost) keeps everything you had on disk. it also should keep the machine in a similar state as whenever the snapshot was taken.
The Skyline gift cards will stop once morale improves.
9 Likes
Ok, I definitely need to stop thinking about this for a bit, but one issue is that I don’t have a good understanding of how we’re set up. To illustrate:
- When I ssh into the server, I see total available space of about 160GB.
- When I look at the monthly invoice I’m paying, I see that we’re paying $48 for the droplet, which matches what I see on the Digital Ocean account site:
But the invoice also includes “Spaces” and “Volumes” that imply additional storage. But I don’t know what that additional storage refers to or how it’s used:
Since things seem to be working fine for the moment, I’m truly going to stop thinking about this for a bit and focus on my work. But maybe the above information will prompt someone to figure out a better way to structure all this.
1 Like
I have a vague recollection that we added the 100GB volume the last time we ran out of disk space. I think there may have been some issue getting the site to use it or something but I can’t remember for sure. I know the next droplet size up is double the cost for basically double all the resources and that doesn’t make sense for the kind of usage we get.