apologies for our brief outage! it was caused by an attempted DNS updated which resulted in a misconfiguration; I'll be reviewing our practices to ensure this doesn't happen again.

we're mostly recovered, still a few small issues here and there but we can expect normal operation from here on out. still working on it, lmk if you run into any issues.

Show thread

@WomanCorn @jai sorry for the trouble, this is part of the ongoing incident; I've identified the cause and am still in the remediation process. this is specifically caused by missing "preview cards", which need to be migrated

schelling.pt/@locus/1095586640

update: seems that masto is unhappy with migrating storage providers w/o copying over all files, so, doing so now. expect this to take hours, but in the meantime, images are gradually beginning to load again. apologies for the inconvenience, please bear with these growing pains as I mature our operations.

Show thread

quick update, still investigating, there's something weird going on with the nginx media proxy. switched from a Wasabi backend to S3 to simplify the setup and eliminate some variables. will keep y'all posted on developments.

Show thread

⚠️ ADMIN POST ⚠️

we're back up, but experiencing an issue loading media; as a result, no avatars or images will show for now. I am actively investigating!

Show thread

⚠️ ADMIN POST ⚠️

schelling.pt will be going down for maintenance in 15 mins at 12:45 PM EST for at most 15minutes, until no later than 1:00 PM EST

during this time, I will be upgrading to a larger server, doubling RAM from 4gb to 8gb, as we've been cutting it close for a while.

apologies for the inconvenience, working to make the best experience for all y'all

feel free to contanct @pee_zombie on Twitter during the downtime if you need anything

⚠️ ADMIN POST ⚠️

⚠️ ADMIN POST ⚠️

you may have noticed we are having short intermittent outages; we're experiencing minor scaling issues. if these keep happening, I will scale the server up ASAP to remediate. please LMK if you experience any more!

welcome new schelling point convergers! glad to see so many new "faces", hope you enjoy your stay.

i'm your local admin, when speaking in "admin voice" I use @locus, but feel free to direct any questions you have at either handle.

i'll be putting up some official rules and more guidelines soon, please feel free to provide input or feedback towards this end. i want this instance to be a collaborative project!

more coming soon, stay tuned! and make sure you're following @locus

@dagon @pee_zombie hm nothing I'm aware of! don't see any unusual errors or metric activity. you don't see it on any client?

⚠️ ADMIN POST ⚠️

apologies for the downtime! seems I'd missed the domain expiration emails 🤦‍♂️ I've now renewed the domain for a few years, should be working again. lmk if there are any lingering issues!

i've identified the issue as being caused by an overwhelmed DB and have upsized it slightly, should hold us off for a while until i get better monitoring in place. apologies for the downtime!

Show thread

⚠️ ADMIN POST ⚠️

am aware there's a performance issue currently, am looking into it. thank u for ur patience

If you are new to Mastodon coming from the Twitter, here are a few amazing Mastodon features that are NOT ON TWITTER. Enjoy! 🐘​

1. Content Warnings CW ⚠️​

It allows you to display a warning but hides the content of your post. Here’s a short guide I wrote about it: infosec.exchange/@Em0nM4stodon

2. Slow Mode 😌

In Preferences > Appearance, you can select the option “Slow mode” to hide automatic feed updates. You will then have to click at the top of the timelines to see new posts (ex: “10 new items”).

3. Custom Emojis :awesome:

When you write a post (called a toot here), you can select emojis that are custom to your server (called an instance here). In the toot field, click on the 😃 button labelled "Insert emoji" at the bottom (this will vary on mobile apps) to see what’s available there. Some are really neat! :blobcatrainbow:

4. Mastodon is different :ablobsmile:

There is no suggestion algorithms here, no promotion, no ads. This is a community. You can be more of your true complex and unique self. Engage with others genuinely. Follow others you like no matter their followers count. Describe yourself and what interests you in your bio, you are not as limited by characters count here.

Be yourself. Leave the toxicity out, it is not valued here. Respect others. Be a kind human. Share what you love. Make friends :heart_cyber:

#TinyMastodonTip #Mastodon

I'm still thinking about longer term sustainability, and how that might mean that to justify the time allocation might make sense to make this a paid instance (nothing crazy, with tiers ofc) but I recognize it's pretty hard to convince people to pay for a previously free service

any thoughts?

Show thread

anyway so I've fixed all these and have upgraded to latest masto (v3.5.3), which took about as much effort as I expected (had to upgrade one step at a time to account for migrations)

the server is still very much a pet, in the sense that it's manually groomed and cared for, nothing is version controlled nor managed thru IaC. this pains me greatly

hopefully this time around I'll make it more of a priority to manage it better

Show thread

- the server was self-DDoSing bc the streaming API component was bootlooping as the node binary somehow became owned by root and hence inaccessible to the service user, so requests would fallback to the much less efficient REST api
- the version of masto i was on has a known issue where it is currently impossible to install with ANY version of node, bc some very specific dep with a binary module depends on a specific version of something which is no longer available online

Show thread

if anyone is curious as to what was wrong, i've found and fixed the following issues:
- main nginx route had misconfigured TLS to use the wrong cert path, so even tho certbot was renewing, the new cert wasnt getting used
- media nginx route's TLS cert was configured properly, but was expired bc the cronjob broke

Show thread
Show older
Mastodon

a Schelling point for those who seek one