crashdoom

joined 2 years ago
MODERATOR OF
[–] crashdoom@pawb.social 2 points 8 months ago

Media uploads should now work again!

[–] crashdoom@pawb.social 5 points 8 months ago

We're investigating! Appreciate the report!

[–] crashdoom@pawb.social 2 points 8 months ago

Taking a peek now, appreciate the heads up!

[–] crashdoom@pawb.social 6 points 9 months ago (4 children)

The server hosting the database crashed for a still unknown reason. Sometime around 5 am MT, the database server had a kernel panic and locked up, it’s been like that since then until we woke up to the alert this morning.

We’ve got a watchdog on the server and an auto-restart in case it hangs, and we’re working to migrate it away to a known stable server while we troubleshoot.

[–] crashdoom@pawb.social 7 points 10 months ago (1 children)

Honestly, like you said, I genuinely have no idea what the US will do; Currently things are being so shaken up that anything is plausible right now.

For outside the US, I don't really know what reach other countries have. The UK's Online Safety Act is the current one I'm watching for since that would require "highly effective age assurance" which... if the folks writing the law had been online like, ever, they'll have realized is an utterly stupid statement unless they plan to enforce integrating ID Verification via a Credit Agency (Experian, Equifax) which does personal question-based verification (which imo is a huge breach of user privacy since while WE might not know who you are still, the credit agency does and where you signed up).

[–] crashdoom@pawb.social 7 points 10 months ago (3 children)

Yes, several US states along with other nations have been considering porn ban laws, along with the rather blatant anti-LGBTQ stance of the currently elected US party.

[–] crashdoom@pawb.social 6 points 10 months ago (1 children)

In the works! We're planning to have some info very soon, though we're still building out an improvement for the NAS to allow us to expand the storage capacity for long term, high size file storage specifically for Pixelfed!

[–] crashdoom@pawb.social 2 points 10 months ago

Time (if it's been persistent for a while), what you were trying to do, what content you were interacting with, etc.

If you've got network logs from it, that'd be a cherry on top! Though for the most part, it'll be our local logs I'll need to correlate your issues with in order to figure out what happened.

[–] crashdoom@pawb.social 4 points 10 months ago

Previously, we were just operating under myself as a sole proprietor. I hadn't formalized things as I hadn't expected the big growth, but we're big enough to justify it to ensure the long-term security of all of our instances, as well as my own personal protection against the changing political climate.

[–] crashdoom@pawb.social 4 points 10 months ago

Yeah, it's been on my plate for a while, and honestly I really should have

[–] crashdoom@pawb.social 15 points 10 months ago (1 children)

Yeah, I have to admit, I spent a while trying to workshop how to make it sound less ominous and I'm really sorry I failed in that; Definitely something for me to work on in the future, and I apologize for it coming across rather legal and scary.

[–] crashdoom@pawb.social 1 points 10 months ago (2 children)

Hi! Apologies for the delay on seeing this; Yes, if this continues to happen, please also @-mention myself or @natebluehooves@pawb.social so we get alerted to it and can investigate.

I'm hoping performance has improved as we haven't been doing any major work on the infrastructure in quite a while so I'd be concerned if we're having slow downs again, though that may have been due to the beta version we were previously running.

 

Currently, we’re running the Ubiquiti Dream Machine directly as the modem via PPPoE, but there appears to be an intermittent issue with the software implementation that results in periodic downtimes of a few minutes while it reconnects.

We’re looking at switching this back out with the ISP provided router in pass through mode to negate the PPPoE connectivity drop.

We don’t expect this to take longer than 1 hour to switch over and test for reliability before bringing the services back up.

We’ll be performing this maintenance around 11 AM US Mountain Time, and will provide updates via the Telegram channel at https://t.me/pawbsocial.

1
liberdon.com (pawb.social)
submitted 2 years ago* (last edited 2 years ago) by crashdoom@pawb.social to c/fediblock@lemmy.dbzer0.com
 

One of the data storage systems (CEPH) encountered a critical failure when Proxmox lost connection to several of its nodes, ultimately resulting in the CEPH configuration being cleared by the Proxmox cluster consensus mechanism. No data, except ElasticSearch, was stored on CEPH.

When the connection was lost to the other nodes, a split-brain occurred (when nodes disagree on which changes are authoritative and which should be dropped). As we tried to recluster all of the nodes, a resolution occured that resulted in the ceph.conf file being wiped and the data on CEPH being unrecoverable.

Thankfully, we’ve suffered no significant data loss, with the exception of having to rebuild the Mastodon ElasticSearch indexes from 6 AM this morning to present.

I’d like to profusely apologize for the inconvenience, but we felt it necessary at the time to offline all services as part of our disaster recovery plan to ensure no damage occurred to the containers while we investigated.

1
submitted 2 years ago* (last edited 2 years ago) by crashdoom@pawb.social to c/adminlog@pawb.social
 

We've migrated all of the Pawb.Social (Lemmy) media from within the container to the new local NAS.

We'll be performing the same updates for pawb.fun and furry.engineer over the course of the next week, minimizing downtime where possible.

We will need to take the services offline once the initial transfer is completed to perform a final check for any new data, and transition to the local NAS, but we'll post an announcement prior to the final maintenance for each instance.

These changes are to allow us longer and cheaper retention of remote and local media across all of our services, and to allow us to begin backing up media in case of a catastrophic failure.

1
test (infosec.pub)
 

test

1
submitted 2 years ago* (last edited 2 years ago) by crashdoom@pawb.social to c/adminlog@pawb.social
 
  • Instance: asimon.org
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Racism, homophobia, transphobia, trolling, abusive admin
  • Fediseer Action: Censured

Evidencehttps://asimon.org/@coin/posts/Aa8tebnT4kYwTVwrwW https://asimon.org/@coin/posts/AZxmCGq8YxrIx5B6UC https://asimon.org/objects/eda7faae-454b-46f7-a309-94066c261ce6 https://asimon.org/@coin/posts/AZKA8C9YIF3zLb2WLA

 

With the upgrade to Mastodon 4.2.0, the way the search system works is changing to allow more broad searches across posts made across the fediverse.

Cutting to the chase: Full-text search across opted-in posts will be possible in the next few days.


In more depth, the system will allow you to search across your own posts and posts you've interacted with from others irrespective of their opt-in status for full-text search, but will additionally allow you to search ANY posts made by users that opt-in to allow their posts to be indexed.

To that effect, a new setting was added to the Public Profile > Privacy and Reach page called "Include public posts in search results" which is disabled by default. Turning this on will allow any user across the Fediverse to search and find your posts using full-text search.

Shortcuts:

To allow our users to decide if this is a feature they want to use on their posts going forward, we won't be enabling ElasticSearch until Friday Sept. 22.


Additionally, new keywords will be added to the search system:

  • from:me
  • before:2022-11-01, after:2022-11-01, during:2022-11-01
  • language:fr
  • has:poll
  • in:library for searching only in posts you have written or interacted with

These will be enabled on Friday along with the rest of the changes to the search system.


If you've got any questions, please let us know!

1
submitted 2 years ago* (last edited 2 years ago) by crashdoom@pawb.social to c/adminlog@pawb.social
 
  • Instance: eveningzoo.club
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Trolling, racism, harassment, no moderation policy

Evidence

1
submitted 2 years ago* (last edited 2 years ago) by crashdoom@pawb.social to c/adminlog@pawb.social
 
  • Instance: shitpost.cloud
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Trolling, racism, harassment, no moderation policy

Evidence

 
  • Instance: hexbear.net
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: Trolling, disruptive behavior, tankie instance, harassment and abuse towards instances defederating them.

screenshots

 
  • Instance: justbros.xyz
  • Type: Defederation
  • Affects: Pawb.Social, furry.engineer, pawb.fun
  • Reason: unmoderated, troll, epileptic imagery
view more: ‹ prev next ›