I think if there were more cyclists that bike rage could be a thing too. You touched on it in your reply. It's the endangerment. The threat to self and property.
Definitely with cars, the property and safety issues are magnified, so I think that all things being equal, if we magically had bikes as primary transport you'd still see road rage, but less than with cars.
I too have a case of the Sunday scaries.