The problem with buttons is that they're expensive (relatively speaking) and ruin the advantages of having everything in a single place like the HA app.
I am not giving access to "everything" to everyone in the house: I already managed to accidentally turn off the 3D printer mid print instead of a light, shouldn't expect others to be better than me.
Right, I see your point and it makes sense. But at the same time consider this: would you still like all the smart stuff if it was completely automated and you had no access to it? It just works, and you can't make it work a little differently if needed? And I offer an example of something that I can't imagine how it could be automated... I have a smart fan, power and orientation is dependent on where I am and how hot I feel. Easy enough to change that with the app as needed. Remember this is something I can control but would not want to give control to other people in the house.
On top of that what about sensors? Without the app you can't see them so for example no quick glance to check the temperature in multiple rooms and outside.
Also while the app is a bit clunky, it sure beats using that one rather than all of the Philips Hue, LIFX, Yeelight, Xiaomi, Tapo, Kasa, Daikin...