PoisonedPrisonPanda

high risk/(hopefully) reward situation yes. But also probably because I am lazy - or lets say, I dont want to change my private life center because of the job.

It always boils down to many factors and my gut feeling tells me thats the best compromise. Or its the anxiety to push against change. nobody knows.

But thanks. may your life be prosperous. Especially if mine wont. ;)

um goutteswühllen.

Ich lach ja selbst über steirisch.

used. Are you in a startup? And you are learning the ropes on the business side?

yes. and yes. basically everything. I assume summarizing my tasks/work would require 3 job positions filled.

is there a comparison between blockchain energy use and AI related energy use?

Well, it's Groundhog Day... again... and that must mean we're up here at Gobbler's Knob waiting for the forecast from the world's most-famous groundhog weatherman, Punxsutawney Phil, who's just about to tell us how much more ~~winter~~ shit we can expect.

The thing is, I am already in a niche - since I am coming from the engineering / non-IT side and sliding into development. Probably the start-up character of my company doesnt help here expecting high income for now.

But I am going with either earning, learning or leaving. And since I learn so fucking much I cannot leave.

Since all trains commute via Salzburg - this is probably the most classic city to visit.

Lots of culture. Probably lots of tourists too. It is a beautiful town - with many options to enjoy a beautiful day. probably enough places to enjoy also during bad weather.

If you like it quiet Linz is also an option. but to be honest Linz is probably worth a day trip without the need to stay overnight - except you want to visit nightlife.

Nevertheless given your preferences to not divert from the main train line - Salzburg is the way to go.

I am getting paid very well

You guys are getting paid well? what the heck am I doing wrong.

broadly you want to locate individual records as quickly as possible by using the most selective criteria

What can be more selective than "if ID = "XXX"? Yet the whole table still has to be reviewed until XXX is found?

... and to familiarize yourself with normalization.

based on a quick review of normalization, I doubt that this helps me - as we are not experiencing such links in the data. For us we "simply" have many products with certain parameters (title, description, etc.) and based on those we process the product and store the product with additional output in a table. However to not process products which were already processed, we want to dismiss any product which is in the processing pipeline which is already stored in the "final" table.

It isn’t just a big bucket to throw data into to retrieve later.

thats probably the biggest enlightment I have got since we started working with a database.

Anyway I appreciate your input. so thank you for this.

Ms sql is trash Running anything on a Hdd is a joke

thank you, I convinced management to migrate to a modern hardware, and we switch to PostgreSQL together with refactoring our design and approach.

You read write and compare continuously? Did you try to split it into smaller chunks?

Locks are handled by SQL. but yes, multiple tables are read, written and the final table compared with multiple requests/transactions (connections?) simultaneously. Split into smaller chunks would nonetheless mean that the query would loop through the whole table - in chunks? how would this help with simultaneous transactions?

A hot DB should not run on HDDs. Slap some nvme storage into that server if you can. If you can’t, consider getting a new server and migrating to it.

Did this because of the convincing replies in this thread. Migrating to modern hardware and switch SQL server with PostgreSQL (because its used by the other system we work with already, and there is know-how available in this domain).

You should avoid scanning an entire table with a huge number of rows when possible, at least during requests.

But how can we then ensure that I am not adding/processing products which are already in the "final" table, when I have no knowledge about ALL the products which are in this final table?

Create an index and a table constraint on the relevant columns. ... just so that the DB can do the work for you. The DB is better at enforcing constraints than you are (when it can do so).

This is helpful and also what I experienced. In the peak of the period where the server was overloaded the CPU load was pretty much zero - all processing happened related to disk read/write. Which was because we implemented poor query design/architecture.

For read-heavy workflows, consider whether caches or read replicas will benefit you.

May you elaborate what you mean with read replicas? Storage in memory?

And finally back to my first point: read. Learn. There are no shortcuts. You cannot get better at something if you don’t take the time to educate yourself on it.

Yes, I will swallow the pill. but thanks to the replies here I have many starting points on where to start.

RTFM is nice - but starting with page 0 is overwhelming.

With MSSQL, the first thing you should check is your indexes. You should have indexes on commonly queried fields and any foreign keys. It’s the best place to start because indexing alone can often make or break database performance.

Indexing is the most answered step. But for foreign key, to my understanding, I apologize is this is maybe wrong, would lead to split the data into separate tables all related by this key right? What would be the difference in splitting the columns of a table into multiple tables - all related by an mutual column, lets say "id"?