If you took Element/Matrix and removed everything from it that is required for federation, then it would perform like Signal.
This is only true for the server side. Federation doesn't affect the client much at all (for example, rocket.chat is now a matrix client, even though it wasn't one for the majority of its life).
Matrix clients are mildly harder to build because of, as you mentioned, end to end encryption, but also because of the many ways you can make mistakes in the code that synchronises clients to the server and cause performance issues or bugs. Even dispute this, though, there are a massive variety of matrix clients: https://matrix.org/ecosystem/clients/
Email is federated. You can set up your own server using something like Stalwart in a few hours. I deliver thousands of emails a month on my personal server. The problem is deliver ability to Gmail and outlook, but if you don't care about that then you're golden.