Was playing it a bit in the morning while it was slow at work, seems fantastic so far!

I hate how installing or removing (or even updating) a flatpak causes the whole software center to completely refresh, and it doesn’t keep its state so if you were in the middle of a search or scrolled down through a category… say goodbye to it.

Well, that took a lot more blood, sweat, and tears than I thought it would.

Usually when performing an update, I do the following:

• Take a snapshot of the VM
• Change the version number in Lemmy docker-compose.yml file for both the lemmy and lemmy-ui containers
• Re-create the containers, and start following the logs
• If the database migration (if any) appears to be taking longer than expected, I temporarily disable the reverse-proxy so that Lemmy isn’t getting slammed while trying to perform database migrations (and then re-enable it once complete)
• Upon any issues, examine where things might’ve gone wrong, adjust if needed, and worse-case scenario rollback to the snapshot created at the start

Everything was going to plan until step 4, the database migrations. After about 30 minutes of database migrations running, I shut off external access to the instance. Once we got to the hour and a half mark, I went ahead and stopped the VM and began rolling back to the snapshot… Except normally a snapshot restore doesn’t take all that long (maybe an hour at most), so when I stepped back 3 hours later and saw that it had performed about 20% of the restore that’s where things started going wrong. It seemed like the whole hypervisor was practically buckling while attempting to perform the restore. So I thought, okay I’ll move it back to hypervisor “A” (“Zeus”)… except then I forgot why I initially migrated it to hypervisor “B” (“Atlas”) which was that Zeus was running critically low on storage, and could no longer host the VM for the instance. I thought “Okay, sure we’ll continue running it on Atlas then, let me go re-enable the reverse-proxy (which is what allows external traffic into Lemmy, since the containers/VM is on an internal network)”… which then lead me to find out that the reverse-proxy VM was… dead. It was running Nginx, nothing seemed to show any errors, but I figured “Let’s try out Caddy” (which I’ve started using on our new systems) - that didn’t work either. It was at that point that I realized I couldn’t even ping that VM from its public IP address - even after dropping the firewall. Outbound traffic worked fine, none of the configs had changed, no other firewalls in place… just nothing. Except I could get 2 replies to a continuous ping in between the time the VM was initializing and finished starting up, after that it was once again silent.

So, I went ahead and made some more storage available on Zeus by deleting some VMs (including my personal Mastodon instance, which thankfully I had already migrated my account over to our new Mastodon instance a week before) and attempted to restore Lemmy onto Zeus. Still, I noticed that the same behavior of a slow restore was happening even on this hypervisor, and everything on the hypervisor was coming to a crawl while it was on-going.

This time I just let the restore go on, which took numerous hours. Finally it completed, and I shut down just about every other VM and container on the hypervisor, once again followed my normal upgrade paths, and crossed my fingers. It still took about 30 minutes for database migrations to complete, but it did end up completing. Enabled the reverse-proxy config, and updated the DNS record for the domain to point back to Zeus, and within 30 seconds I could see federation traffic coming in once again.

What an adventure, to say the least. I still haven’t been able to determine why both hypervisors come to a crawl with very little running on them. I suspect one or more drives are failing, but its odd for to occur on both hypervisors at around the same time, and SMART data for none of the drives show any indications of failure (or even precursors to failure) so I honestly do not know. It does however tell me that its pretty much time to sunset these systems sooner rather than later since the combination of the systems and the range of IP addresses that I have for them comes out to about ~$130 a month. While I could probably request most of the hardware to be swapped out and completely rebuild them from scratch, it’s just not worth the hassle considering that my friend and I have picked up a much newer system (the one mentioned in my previous announcement post and with us splitting the cost it comes out to about the same price.

Given this, the plan at this point is to renew these two systems for one more month when the 5th comes around, meaning that they will both be decommissioned on the 5th of February. This is to give everyone a chance to migrate their profile settings from The Outpost over to The BitForged Space as both instances are now running Lemmy 0.19.0 (to compare, the instance over at BitForged took not even five minutes to complete its database migrations - I spent more time verifying everything was alright) and to also give myself a bit more time to ensure I can get all of my other personal services migrated over, along with any important data.

I’ve had these systems for about three years now, and they’ve served me quite well! However, its very clear that the combination of the dated specs, and lack of setting things up in a more coherent way (I was quite new to server administration at the time) is showing that its time to mark this chapter, and turn the page.

Oh, and to top off the whole situation, my status page completely died during the process too - the container was running (as I was still receiving numerous notifications as various services went up and down), however inbound access was also not working either… So I couldn’t even provide an update on what was going on. I am sorry to have inconvenienced everyone with how long the update process took, and it wasn’t my intention to make it seem as if The Outpost completely vanished off the planet. However I figured it was worth it to spend my time focusing on bringing the instance back online instead of side-tracking to investigate what happened to the status page.

Anyways, with all that being said, we’re back for now! But it is time for everyone to finish their last drink while we wrap things up.

At some point, yes - while I don’t have a concrete date of when The Outpost will be officially decommissioned (as the server it’s running on still has plenty of things that I can’t move over just yet), you might’ve noticed that the performance of the site is pretty shaky at times.

Sadly, that’s pretty much just due to the older hardware in the server, I’ve tried for the last four months to work around it by trying to configure various tweaks for Lemmy and postgres (the database software - which is where the heart of the issues come from), but it hasn’t had much of an effect. I’m pretty much out of options of what I can try at this point, since it not only affects Lemmy but all of the other stuff that I run on the server for myself (hence why I’ve decided to invest in a better system).

So you don’t have to move over right this second, but I would recommend it sometime in the future. The plan is to at the very least wait till Lemmy 0.19 comes out since it should let you migrate your subscribed communities (and blocked ones, if any) as far as I’m aware - but it won’t transfer over posts and comments sadly. They’re still working out some roadblocks for 0.19, so I suspect it won’t be out this month (they don’t have an estimate just yet of a release date).

Generally it’s just through my distro, it’s always occurred since I’ve used KDE unfortunately (since that was one of my first thoughts). This has been across Fedora (and derivatives), Nix, Arch, and Kubuntu.

Traversing a motherboard sounds like it would be interesting!

Only Human - Memphis May Fire (feat. AJ Channer)

https://piped.video/watch?v=lQenGGMa8fc

(Note, the linked music video has a seizure warning at the start of it)

As far as I know, if you don’t have it on Steam then yes.

The Steam build still gets all of the updates to the game… for now, so if you grabbed it on Steam before it was delisted you can continue to play through that.

I used to justify it with “I’ve had a shit day, I deserve to be able to have something for the convenience” - not to mention, I don’t have a car so realistically it was “Do I want fast food or not”.

Then I started to realize that every day tends to be a bad day for me, due to a multitude of reasons. I live paycheck to paycheck (which is why I don’t have a car in the first place) and the amount I was spending on takeout was way too high.

Now the only time I do so is on Fridays because my workplace lets us spend $25 on their tab just for joining the weekly staff meeting. Aside from that, I might order a takeout once, maybe even twice, during a pay period as a “congrats for making it through last month” but I’d like to even stop doing that ideally.

This doesn’t read as a global Blocklist for all Android phones in the world. It reads more as a local database/API for blocked numbers on your phone.

So blocked numbers would theoretically be applied to your messages apps and other “telephony” based apps that use phone numbers such as WhatsApp (should said apps implement the API).

Google already seems to have a spammer database for numbers, though I’m not sure if that applies to just Fi users, Pixel users, or anyone who uses the Google Phone app. If I have call screen disabled, I’ll see numbers on an incoming call have a red background with a “likely spam” description.

But based on the comments on this post, I feel as if I’ve overlooked something in the article here (I’ve just woken up so it wouldn’t surprise me) - is there a mention of it being a worldwide list?