might be slow or down at the moment - our cache ssd died and we have to move the database off to another server

we are back online, matrix and mastodon may still need a while to synchronize

Zeige Konversation

...we did get a mail about this two weeks ago. we forgot it was today. sorry about there being no prior notice.

Zeige Konversation

and everything is down again. long story short: server room caught fire a month ago (, apparently there's still toxic dust everywhere, they hired a company to clean the server racks one by one, and it's our turn today. we've been told it might take until tomorrow, we'll wait and see.

@clerie we have a big fat 3-node k8s cluster in the pipeline that should allow for one box to die and everything should still work, we even have a floating-ip ... I hope we can migrate everything in the next months and things should run more stable.

sorry, again unexpected downtime - thanks to grub, zfs and somehow we managed to kill our serial console so we had to wait until today to reset the box because it's in the university server room - things should work again for now.

looks like we served expired certificates during the last hours - sorry - we'll look into it - looks like a stupid wildcard cert mistake - tl;dr don't put * in filenames in linux or rather don't assume it'll work if does it. is not available for short time, upgrading to 1.39rc2 triggered an expensive database migration, might take while.

we updating our server - everything might be down a few hours is still rather slow at the moment - we are still purging rooms without any local users and do some cleanup and remove some clutter - unfortunatly the synapse delete room api is slooooow - all this stuff should be finished by tomorrow.

matrix is down for a moment - we update to postgresql 13 - copying and re-indexing might take a while.. eta: 1 or 2 hours still...

everything is up again - misconfigured quota

we are down at the moment - i/o is hanging - we try to fix it asap! is up again, there will be some shorter downtimes during the day, as we update more stuff but we fixed the boot issues for now. sorry! is down, once again zfs module initrd fuckup - eta: sometime tomorrow, need to boot external iso and regen initramfs - sorry

looks like it was only a tempory routing glitch - everything looks okay.

Zeige Konversation
Ältere anzeigen

Mastodon ist ein soziales Netzwerk. Es basiert auf offenen Web-Protokollen und freier, quelloffener Software. Es ist dezentral (so wie E-Mail!).