⚠️ ADMIN POST ⚠️

apologies for the downtime! seems I'd missed the domain expiration emails 🤦‍♂️ I've now renewed the domain for a few years, should be working again. lmk if there are any lingering issues!

i've identified the issue as being caused by an overwhelmed DB and have upsized it slightly, should hold us off for a while until i get better monitoring in place. apologies for the downtime!

Show thread

⚠️ ADMIN POST ⚠️

am aware there's a performance issue currently, am looking into it. thank u for ur patience

If you are new to Mastodon and coming from the Twitter tide, here’s a few amazing Mastodon features that are NOT ON TWITTER. Enjoy! 🐘​

1. Content Warnings CW ⚠️​

It allows you to display a warning but hides the content of your post. Here’s a short guide I wrote about it: infosec.exchange/@Em0nM4stodon

2. Slow Mode 😌

In Preferences > Appearance, you can select the option “Slow mode” to hide automatic feed updates. You will then have to click at the top of the timelines to see new posts (ex: “10 new items”).

3. Custom Emojis :awesome:

When you write a post (called a toot here), you can select emojis that are custom to your server (called an instance here). In the toot field, click on the 😂 button in the upper right corner to see what’s available there. Some are really neat! :blobcatrainbow:

4. Mastodon is different :ablobsmile:

There is no suggestion algorithms here, no promotion, no ads. This is a community. You can be more of your true complex and unique self. Engage with others genuinely. Follow others you like no matter their followers count. Describe yourself and what interests you in your bio, you are not as limited by characters count here.

Be yourself. Leave the toxicity out, it is not valued here. Respect others. Be a kind human. Share what you love. Make friends :heart_cyber:

I'm still thinking about longer term sustainability, and how that might mean that to justify the time allocation might make sense to make this a paid instance (nothing crazy, with tiers ofc) but I recognize it's pretty hard to convince people to pay for a previously free service

any thoughts?

Show thread

anyway so I've fixed all these and have upgraded to latest masto (v3.5.3), which took about as much effort as I expected (had to upgrade one step at a time to account for migrations)

the server is still very much a pet, in the sense that it's manually groomed and cared for, nothing is version controlled nor managed thru IaC. this pains me greatly

hopefully this time around I'll make it more of a priority to manage it better

Show thread

- the server was self-DDoSing bc the streaming API component was bootlooping as the node binary somehow became owned by root and hence inaccessible to the service user, so requests would fallback to the much less efficient REST api
- the version of masto i was on has a known issue where it is currently impossible to install with ANY version of node, bc some very specific dep with a binary module depends on a specific version of something which is no longer available online

Show thread

if anyone is curious as to what was wrong, i've found and fixed the following issues:
- main nginx route had misconfigured TLS to use the wrong cert path, so even tho certbot was renewing, the new cert wasnt getting used
- media nginx route's TLS cert was configured properly, but was expired bc the cronjob broke

Show thread

!! ADMIN POST !!

apologies for the long maintenance hiatus

I have resolved the main issues and have upgraded to latest Mastodon version

should be good for a bit; please reach out with any concerns

ADMIN POST:

apologies for the outage, we should be back now! these should be minimized soon as I'm working on setting up monitoring/alerting

@anonymoose thanks for letting me know, its been a recurring issue; should be fixed now!

@WomanCorn oh no! ty for the notice, I'll take a look in the AM

!! ADMIN POST !!

seems like we haven't had images load for a bit over a week; apologies for taking so long to address this. the cause was that the SSL cert for the object storage domain (files schelling.pt) had expired, as I had forgotten to renew it at the same time as the primary cert for the root domain. I've resolved the issue & will try to find time to automate the cert renewal so that we don't face this issue again

major thanks to @WomanCorn and @TetraspaceGrouping for pointing this out!

@WomanCorn oh no!! thanks for brining it to my attention, I'll take a look ASAP

@TetraspaceGrouping @protoneutype ah missed this one sorry! yeah I've thought about this, not sure what the answer is tbh. don't think limiting would be a good idea as it's unfeasible to entirely switch over. it would be nice to be able to use this as the primary interface, but it doesn't really work that well in practice, as you can't really "see" Twitter from here

@TetraspaceGrouping hey thanks for this, it helped me figure out what was wrong; i knew the primary certificate wasn't expired, but the `files.` subdomain was using an older, incorrect one, and needed to be renewed independently. should be working now!

!! ADMIN POST !!

once again, apologies for the downtime! I am investigating the root cause, and setting up monitoring/alerting will be my highest priority this weekend, in order to minimize the risk of this recurring. thank you for your patience and for bearing with our growing pains!

!! ADMIN POST !!

apologies for the recent downtime! I'm still looking into what happened and will report back with findings. for the time being, we are back up.

Show older
Mastodon

a Schelling point for those who seek one