For me, this is where utilitarianism falls apart. It makes the observer the person who gets to decide what "the best lives possible" means.
Every ethical framework requires making some affirmative presumption to begin with. The is-ought gap cannot be closed. Many ethical frameworks begin with less tangible things, like a belief in a deity, which can also lead to either heinous or benevolent outcomes.
How can the outside observer have the authority to make this decision?
When talking about humans, consider when people defend colonialism by saying they brought "civilization" and modern medicine, comforts, etc. to people who did not live the way the colonizers did. I'm not saying that non-colonized people live in some utopia, but the people who thought they were doing good didn't give a single fuck about what the colonized people wanted, disregarded all their knowledge and experience and forced their ways on them. Even if we take lessons learned from that and try and be more open minded about listening to people before making decisions about them (my skin is crawling as I type this omg) we don't know what we don't know and it makes no sense to apply this framework to decision making impacting others.
I don't disagree with any of this. And this is why I also strive to do whatever I can to accomplish the goal that I care about -- everyone having the best lives possible -- to do whatever I think results in other having the greatest degree of autonomy. It's because I believe that no one knows what would result in a better life for themselves than themselves. I will always defer toward what empowers them to have as much autonomy as possible, provided they aren't harming others (like carnism, colonialism, capitalism, ethno-supremacy, etc. do).
Now consider non-human animals and how we are even less effective at communicating with them...
Yeah. I think there's an interesting conversation to be had about how one can cause the least harm and be most helpful to someone that we can't effectively communicate with. I don't have a good answer for this, so I just want to make sure their basic needs are met (or in the case of non-human animals, not actively sabotaging them) so that they can try to do whatever is best for them.
I don't think it's inherently anti-justice. But maybe we can pivot to something more constructive than just disagreeing about it: if you have an alternative to utilitarian-adjacent ethics, I'd love to hear you out. I might still disagree, but I want to know if there's a better way to form my ethical positions.
At the end of the day, we're only human, and we should be doing the best that we can to be ethical, and if there's something better than what I'm currently doing, I want to know about it.