this post was submitted on 11 Mar 2026
129 points (97.1% liked)

Programming

26022 readers
280 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev



founded 2 years ago
MODERATORS
 

So my manager today asked me if I could stay later when there's broken things in prod, and then today his star dream employee yolo'ed a full stack change into prod without review. It's fucking massive and implements new API endpoints, touches >20 files. Many of the diffs are too large to render in the browser.

It's almost comical, but something immediately broke.

Most of my day, I'm digging through code to identify bugs created from this shit, just to get a stealth merge midday.

I kind of don't know what to do.

all 37 comments
sorted by: hot top controversial new old
[–] VibeSurgeon@piefed.social 12 points 15 hours ago

Don't provide bailouts for neither your manager nor your colleague. Highlight as far up the chain as you can the damage their behavior is having on the business.

[–] thedeadwalking4242@lemmy.world 35 points 1 day ago (2 children)

Do NOT under any circumstances fix it. The other guy needs to and you need to be clear he broke it and how

[–] Tharkys@lemmy.wtf 14 points 23 hours ago

I came here to say exactly this. The offending dev and your manager aren't going to understand the severity of issue until he spends days trying to find the bugs. Not your circus, not your monkeys.

[–] Corbin@programming.dev 45 points 1 day ago

You need SRE concepts. First, if you break it then you fix it; in a system where anybody can make a change, it's the changer's responsibility to meet service objectives. Second, if your boss doesn't find that acceptable then they need to appoint a service owner and ensure that only the owner can make changes; if the owner breaks it then the owner fixes it. Third, no more than half of your time should ever be spent fixing things; if something is constantly broken then call a Code Yellow or Code Red, tell your service users that you cannot meet your service levels, and stop working on new features until the service is stable again.

Under no circumstances, ever, should anybody stay late. There should only be normal business hours, which are best-effort, and an on-call rotation which is planned two months in advance. Also, everybody on call should be paid hourly minimum wage on top of salary for their time.

[–] slazer2au@lemmy.world 115 points 1 day ago (2 children)

I would say no to staying back. The employee that broke it needs to fix his code, at the very least roll the change back.

At the next meeting or post mortem bring up why code can be pushed to prod without review. Your git forge should be able to block merging to prod by a single person.

[–] reabsorbthelight@lemmy.world 38 points 1 day ago

Yeah. I log off at EOD and don't check messages. It's not my families fault he lets his employees vibecode shit into prod. The Vibelord is the main problem, but others do similarly stupid things.

[–] Auster@thebrainbin.org 19 points 1 day ago (1 children)

Adding to that, about the "stealth merge", couldn't it be just the rollback? Don't change what works, as the saying goes.

And expanding on the meeting or post mortem idea, take this time not to fix the code, but to make an essay showing what broke, how it broke, and how it could be avoided in the future. Also preferably not letting your tone getting acid or those hearing from the company could instinctively refuse to listen to you even if your points are solid.

[–] Reginald_T_Biter@lemmy.world 2 points 15 hours ago

If only there was some way of reverting the merge commit and redeploying.

[–] Feyd@programming.dev 26 points 1 day ago

Roll it back and go home.

[–] Jesus_666@lemmy.world 34 points 1 day ago

Pushing to prod without review and breaking the running application is a resume-generating event in many companies. In many others it's not even possible because of programmatically enforced policies.

If your company's response is not to prevent or dissuade it but to have other people work overtime to fix the mess then that's a major management fail.

Try to educate your boss about best practices. This incident should give your arguments some more weight.

Deployment to prod should not be something a developer can do by themselves; a proper CI/CD system can be configured so that prod can only be deployed to by people with an appropriate role (product owners or lead devs if your company doesn't have POs).

If you don't have such a system, make it an explicit policy: Only Steve the lead dev (or someone specifically appointed by him while he's absent) can push to prod; if anyone else does it they get invited to an uncomfortable meeting with Steve. If they do it again the meeting will be with HR.

But seriously, you should lobby for a proper CI/CD system (if none is present) and for the system to be configured so that a) you can't merge to the main branch without a code review and b) deploying to prod only works from main and with explicit approval by a PO/lead dev. That should stop most of the shenanigans.

[–] traches@sh.itjust.works 44 points 1 day ago

Roll it back and go home

[–] owenfromcanada@lemmy.ca 29 points 1 day ago

If your manager won't hear anything negative about Vibelord, polish up your resume. There's nothing you can do to change the culture of your team if the boss isn't on board with it.

Not sure what kind of person your boss is, but if you want to try to win him over, try to figure out what might sway him. If he's technically illiterate, talk to him about best practices (like, I dunno, code reviews) and cite reliable sources and data if you think that would help. Avoid calling out Lord Vibington, the main thing is to put a picture into your boss's head of what this could/should look like. He might not realize that these issues are preventable.

The goal here is to get your boss to take more ownership of the team's culture, and start insisting on preventative measures. Mr Viberator will either have to conform, or there will be increasing friction between him and the boss.

[–] custard_swollower@lemmy.world 21 points 1 day ago* (last edited 1 day ago)

Explain to your manager that if an employee can merge code without reviews, he should expect breakages on regular basis.

Propose that your team can meet to talk about how to make the process better, so no breakages occur.

Mention personal responsibility and knowledge - it’s a good practice that a person who wrote the latest code changes either helps either is responsible for making the service working again, if there are issues - specifically because they might have some idea what’s broken.

If your manager will ignore it, I’d start looking.

[–] not_woody_shaw@lemmy.world 13 points 1 day ago* (last edited 1 day ago)

Changes that big need to be broken down into smaller PR-sized steps. To do that you need an actual plan of how to get from here to there. Bro is swaggering like a senior then coding like a junior.

What to do? Well now you have evidence that AI tools are not yet good enough to write code that doesn't need human review before deploying to prod. Chat with whoever is in charge of Process, and get some team guidelines in place to prevent unreviewed code going to prod, and unreviewable PR's being submitted for review.

[–] MonkderVierte@lemmy.zip 7 points 1 day ago* (last edited 1 day ago) (1 children)

You render diffs in the browser?

And no, let the yolo fix the things he broke, or he will never learn to be a responsible coder.

[–] jtrek@startrek.website 11 points 1 day ago

GitHub and similar tools show PR diffs in the browser.

[–] 87Six@lemmy.zip 9 points 1 day ago

I'd just talk to the guy, clarify HE needs to stay late not me, then log off. If he's nice about it I'd consider ataying but from your description, he's anything but.

What you gotta do in situations when someone is beingan asshole is spark a meeting with the higher ups so they can see the BS you deal with and how it's none of your business that prod was messed up.

[–] ExLisper@lemmy.curiana.net 6 points 1 day ago

This is not how you do development. Like, at all. If changing jobs is an option for you do it. If not, well, not much you can do. You're not going to fix the entire company if management and other devs don't know how it should work and don't want to change.

[–] kivarada@programming.dev 8 points 1 day ago* (last edited 1 day ago)

Pray that the whole system breaks sooner than later...

[–] Kissaki@programming.dev 5 points 1 day ago* (last edited 1 day ago) (1 children)

if I could stay later when there’s broken things in prod

In general, or on this instance?


Do you have team retrospectives? That's where I would bring it up in my team. Raise my concerns, explore and understand what team consensus is around this topic, around risks, quality, etc.

If the team consensus and/or management consensus is YOLO - then I try to protect myself from personal investment and going beyond contractual obligations. Because I already know what will come and how it will negatively affect me personally.

It's possible a honest discussion with management about goals and risks could lead to clarified guidelines, requirements, and goals. If it doesn't, I'd probably be looking for a better job/environment. Because I'll be miserable if colleagues YOLO, no matter how careful I am personally.

[–] reabsorbthelight@lemmy.world 6 points 1 day ago (1 children)

I told my manager that we need to focus on quality. He partially agrees, but part of the problem is that he didn't even know how to get to our company git repo till I showed him. All he knows is the front end.

Unfortunately the Vibelord is his favorite employee

[–] Skullgrid@lemmy.world 6 points 1 day ago

Unfortunately the Vibelord is his favorite employee

escalation time

[–] mesamunefire@piefed.social 4 points 1 day ago (1 children)

Make ci/cd part of the process. It stops a lot of people who push huge unmaintanable changes.

If it doesent pass the tests and cant build then it stops them from merging.

[–] reabsorbthelight@lemmy.world 1 points 1 day ago (1 children)

Yeah this is my general tool for dealing with shit code. I'm currently working on getting that into the CI

[–] mesamunefire@piefed.social 2 points 1 day ago* (last edited 1 day ago)

It helps! Ive seen it with two jobs completely change the culture around code review.

  • Instead of you/team lead being the bad guy, its now the code/process.
  • Code becomes more stable and releases actually become more frequent.
  • If something goes wrong, your VM/docker/box/etc.... can just be re-spun up by the same process.
  • Easier to onboard since the same build process is in the CI. And is constantly being used rather than relying on the README (that may not have been updated in a while).

Mind you its not perfect of course. You can still "Vibe" test and/or remove tests that dont work. And make the project more brittle. Or go overboard with lint rules (I actually had to break up a fight with that one). But its much better than the old process of merge and pray.

[–] cecilkorik@piefed.ca 4 points 1 day ago (1 children)

When Rome is burning, do as the Romans like Nero do. Fiddle a nice tune, tell Claude to fix it for you, and go home. If they're not willing to give any fucks, why should you?

[–] 13igTyme@piefed.social 3 points 1 day ago* (last edited 1 day ago)

Fix AI coding with AI coding. Genius. Also update your resume and start looking elsewhere.