Aside from the commonly stated - experience:
The key point to keep in mind is that, at the end of the day, your building an application to satisfy a customer. Or to be exact, a list of requirements that may or may not be constantly changing. In this case from what i'm reading, I am assuming its your hobby project, so the 'customer' is you.
In this case, over-engineering is when you add more functionalities/services than what you needed to hit the bare minimum requirement. Ideally you want to hit basic requirements first, then start designing/engineering on top of what you have when your customer wants more features, etc.
Your design providing more features than the bare minimum should be an 'accident' more than intentional ideally, unless you think the extra feature takes 0% effort to implement. (Though TBH safe rule of thumb is, never design for additional unrequested features)
With the above context in mind, you should be asking the questions:
- How much downtime is acceptable? (Usually measured in per year)
- How much $ cost in SaaS/API services is acceptable? (per month, or year, etc...)
- How much time/money spent in maintenance is acceptable? (Helps in determining the API/Service you are going to use)
- Other questions related to acceptable risk/costs, etc...
Yea, welcome to client engineering. Usually its handled by senior developers or project managers, unless your in startup.
TLDR; It's not about finding a design that is perfect, its about finding a design that is acceptable by the involved. What is acceptable? that is for the involved parties to decide.
I personally use Debian. For your case, you can install lightweight desktop environments such as XFCE.
Honestly from my point of view after reading your post, you don't have a terminal or operating system issue, it feels like you are new to self-hosting and don't know how to start configuring from scratch.
Ideally you want to look for documentations or keep asking for online help. For example, with installing docker, you would want to refer to this: https://docs.docker.com/engine/install/. Welcome to system admin life, where you spend more time reading/understanding than configuring.
Personally, you can even use AI Chatbot to help you with stuff, just be specific on the system you are on, the goal you are trying to achieve and the problem you are tring to solve.
Which brings me to answer your next point about CasaOS: It exists so that you can skip most of the 'system admin life' step. It skips almost all the setup you would have needed to do on a fresh machine, and just leaves configurations. The downside is usually it eats up more resources than a self-configured install since it comes with redundant features you are unlikely to use.
TLDR; Pre-configured OS such as CasaOS is a solid choice if you just want to set it up and be done with it. If you are here to really learn about system admin stuff, then pick any of the Linux Operating system (Debian-derivatives recommended) with a lightweight DE.
Happy self hosting :v