Managing docker-compose files over a few servers.
Notepad++ is fantastic, I can have multiple tabs with docker-compose files from different servers (via a ssh plugin), in a single window.
I was advised to move to VSCode or VSCodium. Neither can handle a single window with files from different servers, I need to switch workspace every time. Back to Notepad++.
You absolutely can mount different files from different servers simultaneously in vscode. I do it all the time with the SSHFS plugin. You just set the server, login, and path, then you can the remote folder to any workspace, and you can mount multiple different folders simultaneously. Right now I have a workspace which has mounted folders from 3 different servers for my Grafana/Prometheus/Influx DB setup
You absolutely can mount different files from different servers simultaneously in vscode
It's not in vscode then, it is on the filesystem if I understand correctly. Then even notepad can handle that.
sshfs causes weird permission issues for every server I use and cannot save files, so that's not an option.
I actually was a Notepad++ fanatic for years and still use it lightly but I have made the switch to VSCode about 6 months ago and I'll never look back. As the other commenter said you absolutely can mount files from multiple workspaces as well. Hell you can even copy files between file systems.
I'm sorry what? You can edit text files on a different machine from notepad++??????? Omg this changes... everything. Wtf I did not know this. How could not have known this!!!
You can have a central repo for your files. I have a different folder for each server. VSCode has access to these folders to edit any data needed.
Most of the compose files are managed in Portainer, but some of the containers need to exist outside of Portainer.
Forgetting how x service works when it is dependent on y service which is dependent on the NAS and the NAS broke and you wonder why everything else is broken too.
Being kind of dumb.
Thank you for the laugh :)
In my experience, it's the noise and cable management.
The rack we chose was a cheapo open-frame off Amazon so all the cables are kind of just hanging off the side while only being partially tied up and run through loops made of zip ties. Proper loops, wire arms, drawers, etc, are expensive to obtain in this area, and we've moved to the rack several times over the past several years. We've also lost a few bolts because they literally snap off while being removed.
Yes, having worked in server rooms before setting up home equipment, I chose to use consumer grade equipment. Less noise, less heat, potentially less power. My first "rack" was a wooden storage rack to hold the then 3 desktop towers. I did get the fun of building each system from scratch to get the specific bang for the buck.
Dropping a machine on my foot
Having a friend drop an crazy heavy enterprise class server on your leg...
The space, I've had full racks in the past, but now I'm down to just a SonicWall and a single high power desktop pc as a host. It works out well though for what I need. 24 threads and 128gb can run quite a bit.
Buying good equipment
Utilizing it properly
Editing other peoples' docker compose or dockerfiles to suit my needs.
Will battery back up hold... for shut of stuff
I learned last week that the battery backups draw a lot more amps while charging after a power outage. Blew the breaker trying to have all 3 UPS's come online. Once online the whole rack pulls just over 2A.
Better to learn it now then later
Finding obscure "outdated " parts to upgrade a system that's "outdated".
Truth.
I kept an old (Only 8th gen!) CPU around because I thought it'd be easier than a full build for one of my systems.
Took far too damn long to find a good Mini ITX motherboard for it, let alone one in stock, and in the end the one I did get didn't even come with a CMOS battery.
This isn't entirely related to homelab, but I had this exact problem dealing with my mATX gaming PC a few years back. Spilled a beverage on it and somehow the mobo was the only casualty. But it wasn't that easy to find a six-gen mATX mobo for under $100. If I actually had the time for games, I'd probably use it as an excuse to upgrade everything, but I only keep it around for the occasional indie or Fallout 4 binge or Fortnite session with my son.
designing, spending money on a setup, only to find out I need to scale more 6 months later and realizing that overhauling the current setup is not gonna work leading to basically building a new homelab all over again.
I am separating home lab itself and servers running some useful staff at home, like homeassistant, which more like "home production services". Pure home should be powered off most of time, if you just doing some on-prem experiments there.
All the small little quirks that aren’t well documented or documented at all. Then I’ve got to go back through my notes and lessons learned, but invariably I’ve forgotten to write something down and have to resolve a problem I know I’ve solved but don’t remember.
Troubleshooting in general.
Welp, everyone beat me to the dad jokes.
I agree OP, trying to find the time to learn docker outside of my full time job and juggling family stuff has been the hardest.
Breaking your internet connection and not being able to google how to fix it
Paying the invoice … and cable management
Networking,
I’m a statistician by degree not a networking guy.
I don’t know anything other than what an IP address is. Figuring out subnetting, DNS, reverse proxy, VPN stuff.
I don’t know how secure my current setup is but it works so for now I’m not touching it.
networking is also my weakest point. literally just last week as i was reading through cisco docs that L2 networking is mac based and thats why switches need ARP tables. i felt so dumb for not realizing this like 10 years ago lol
Convincing my wife that we need X
Justifying hardware purchases to my wife.
My biggest fear is thinking my network and server is secure. But missing something. If anyone has any tips. I’m all ears.
Just make sure general network (i.e home public* wifi/wired) are properly spararated from lab net, also make sure to have different mgmt net, have a different wifi/wired net just for you, monitor & firewall those correctly (including outbound connections), keep software up2date, isolate servicies, rootless & ditroless & read-only containers, and read common daily secnews (bleeping computer, hackernews, seclists & fulldisclosure, ...) you should be good.
*public in this context doesn't mean passwordless, but rather being used by others than you (wife, kids, friends)
Most painful? Guess that depends on your insistence on making it good.
Most painful for me was definitely running shielded cat 6a through my whole house and my ceilings don't have strapping so getting my ceiling mounted ap's installed and my AI pickup mics was a nightmare. I ended up cutting a 12 hole in one wall and having to use a 50" drill extension to drill into a floor joist cavity and then carefully and painfully fushing my endoscope through wiring holes for lights etc pulling it around with giant magnets. Took me 3 days to get the one in the kitchen...
But I mapped every pipe, and every wire so thats good.
I ran 6 runs to my office alone because I wanted 6 sets of poe++ lines in my office so I could play with poe circuits on my electronics bench.
Also for the servers I had to install 2 new breakers and run two sets of romex for two 25 anp circuits. And I wall mounted that rack, so that sucked too.
However while my servers are in the garage, my router, fiber modem ,switches etc are under the living room stairs. I wasnt about to run 25+ ethernet lines out to the garage so the garage only has 4.
I rain All my upstairs ethernet out of the attic into the same cavity straight down the side wall of the stairs directly behind our entertainment center and into the under stairs soace behind it. Then into the crawl space for the downstairs stuff and garage.
Wiring stuff, definitely the pain point.
I went with ubiquiti dream machine se, unify, ubiquiti aps, ubiquiti bullet cams, and poe for stuff.
Even my living room lights are poe.
Im a junior developer, so with my starting salary so far its cost of hardware. Electronics be expensive. Currently saving up for a synology rs1221 and 3 samsung 8tb SSDs.
I'll probably beat everyone...living in Australia!..
Super super high electricity bills and finding a small case to build a NAS here... Everything here is practically Apple or off the shelf...people look at you strange for even having a NAS, let alone a DIY NAS.
The only case that just became available here is a Silverstone cs351 unless you're going to spend thousands.
I thought, this is exactly what I wanted... After unpacking it, what do you think I've been doing?
Practically rebuilding the case because it has almost zero air flow...no front fan, and only exhaust fans...and because I'm in Australia, it's almost summer where we get easily over 110f or 40c...
I won't go into the price of things in Australia or having already ordered 2 different cases from different shops for this build and being told it could be a 6 month wait....after I paid for it...
documenting everything I do. It's important to keep detailed notes on setups, config locations, pitfalls, etc.
Probably keeping stuff up2date. Yes, "99%" can be automated, but there are still annoying things like switches & APs fw, UEFI, HV OS updates which can't be sanely automated unless you are in for bricking the crap.
Bonus: proper monitoring and logging systems are just plain annoyance in general, because good ol' there are always some edge cases.
Finding the right house.
Every house I look at, the first thing I check is the breaker box, how many amps are trucked in from the street, can I get more, where is the nearest space to put the servers vis-a-vis suitable outlets (or if I can run electricity easily from the breaker area to the place where I want my servers).
I pretty much spend time looking at how I use my space. Then I start looking at proximity to outside walls, if I can close off the space and pipe in cold air from a split unit air-con, etc etc.
Then I check what internet I can get, if the address is eligible for business broadband with at least tolerable connection speeds.
Between that and looking for a good school district, a decent sized kitchen with shelving I don't have to redo entirely again, proper lighting in the rest of the house, it's kinda hard to find everything I want in one place.
Recognize the security concept was not good enough. Like finding Maleware on proxmox, or some strange outbound traffic to the darknet. 😂
Thinking about how little the wife will sell the lab for if I die. Too many "50$" servers 😪
Learning that you don’t have enough upload to do everything you want….
Setting up a MikroTik Router with zero knowledge and experience on RouterOS. Yup, that's what I'll be doing very soon.
Troubleshooting 10Gb/sec connectivity problems. Is it the router, is it the switch, is it the SFP+ ethernet enclosure, is it the cable, is it the network card on the client.. etc etc
A lot of the tech is quite cutting edge and the cheaper stuff doesn't always work very well together. I don't have a ton of money so I have to troubleshoot instead of upgrade.
Cable management