we are back online, matrix and mastodon may still need a while to synchronize

Zeige Konversation

...we did get a mail about this two weeks ago. we forgot it was today. sorry about there being no prior notice.

Zeige Konversation

and everything is down again. long story short: server room caught fire a month ago (tlz.de/regionen/weimar/brand-i), apparently there's still toxic dust everywhere, they hired a company to clean the server racks one by one, and it's our turn today. we've been told it might take until tomorrow, we'll wait and see.

sorry, again unexpected downtime - thanks to grub, zfs and somehow we managed to kill our serial console so we had to wait until today to reset the box because it's in the university server room - things should work again for now.

looks like we served expired certificates during the last hours - sorry - we'll look into it - looks like a stupid acme.sh wildcard cert mistake - tl;dr don't put * in filenames in linux or rather don't assume it'll work if acme.sh does it.

matrix.bau-ha.us is not available for short time, upgrading to 1.39rc2 triggered an expensive database migration, might take while.

we updating our server - everything might be down a few hours

matrix.bau-ha.us is still rather slow at the moment - we are still purging rooms without any local users and do some cleanup and remove some clutter - unfortunatly the synapse delete room api is slooooow - all this stuff should be finished by tomorrow.

matrix is down for a moment - we update to postgresql 13 - copying and re-indexing might take a while.. eta: 1 or 2 hours still...

everything is up again - misconfigured quota

we are down at the moment - i/o is hanging - we try to fix it asap!

matrix.bau-ha.us is up again, there will be some shorter downtimes during the day, as we update more stuff but we fixed the boot issues for now. sorry!

matrix.bau-ha.us is down, once again zfs module initrd fuckup - eta: sometime tomorrow, need to boot external iso and regen initramfs - sorry

looks like it was only a tempory routing glitch - everything looks okay.

Zeige Konversation

update: contacts told us there were problems with the cooling over the last few days and non-essential machines are probably shut down - looks like we are deemed non-essential (to be honest, overall they might have a point, but... we think we are essential :)

I have to call them tomorrow, we are likely down until tomorrow, maybe longer if I can't convince them that we are essential...

😭

Zeige Konversation

bau-ha.us is down - we don't know the cause yet, we can't reach both machines and also not the serial-console interfaces - we suspect there is some hiccup at the university datacentre - let's hope for the best - we keep you updated!

well, we are down a few days and lost the password and the recovery-mail address was also down - that didn't work so well - but now we are prepared :)

Ältere anzeigen
MastoKIF

Mastodon ist ein soziales Netzwerk. Es basiert auf offenen Web-Protokollen und freier, quelloffener Software. Es ist dezentral (so wie E-Mail!).