Selfhosted

48471 readers

346 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz

Have I been DoS'd? (lemmy.world)

submitted 2 years ago by Solvena@lemmy.world to c/selfhosted@lemmy.world

18 comments fedilink hide all child comments

Hi,

I know this is quite impossible to diagnose from afar, but I came across the posting from lemmy.world admins talking about the attacks they are facing where the database will get overwhelmed and the server doesn't respond anymore. And something similar seemed to have happened to my own servers.

Now, I'm running my own self-hosted Lemmy and Mastodon instances (on 2 seperate VPS) and had them become completely unresponsive yesterday. Mastodon and Lemmy both showed the "there is an internal/database error" message and my other services (Nextcloud and Synapse) didn't load or respond.

Login into my VPS console showed me that both servers ran at 100% CPU load since a couple of hours. I can't currently SSH into these servers, as I'm away for a couple of days and forgot to bring my private SSH key on my Laptop. So, for now I just switched the servers off.

Anyway, the main question is: what should I look at in troubleshooting when I'm back home? I'm a beginner in selfhosting and I run these instances just for myself and don't mind if I'd have to roll them back a couple days (I have backups). But I would like to learn from this and get better at running my own services.

For reference: I run everything in docker containers behind Nginx Proxy Manager as my reverse proxy. I have only ports 80, 443 and 22 open to the outside. I have fail2ban set up. The Mastodon and Lemmy instances are not open for registration and just have 2 users each (admin + my account).

you are viewing a single comment's thread
view the rest of the comments

[–] tmjaea@lemmy.world 4 points 2 years ago (2 children)

I can't help much regarding the service denial issue.

However Port 22 should never be open to the outside world. Limiting to key authentication is a really good first step.

To avoid automated scans you should also change the port to a higher number, maybe something above 10,000.

This both saves traffic and CPU. And if a security bug in sshd exists this helps, too.

[–] mea_rah@lemmy.world 5 points 2 years ago

Moving off from port 22 is effectively just security by obscurity. It will save you some logs but the bandwidth and CPU time saving is negligible - especially with fail2ban.

[–] dan@upvote.au 2 points 2 years ago (1 children)

To avoid automated scans you should also change the port to a higher number, maybe something above 10,000.

This doesn't really work any more. Port scanning is trivial with IPv4, and tools like masscan can scan the entire IPv4 internet (all IPv4 addresses) in less than 15 minutes.

[–] tmjaea@lemmy.world 1 points 2 years ago (1 children)

Very interesting, thanks for sharing!

I know it's just anecdotal evidence, however fail2ban in my one machine which does need ssh on port 22 to the open internet bans a lot of IPs every hour. All other ones with ssh on a higher port do not. Also their auth log does not show any failed attempts.

[–] dan@upvote.au 1 points 2 years ago

Yeah it's probably script kiddies that are only trying port 22 because it's easier than port scanning.

Make sure you disable PasswordAuthentication in the OpenSSH configuration (/etc/ssh/sshd_config on Debian at least). People won't be able to try to brute force their way in if the only way in is without a password (ie using an SSH key) :)