Only the buyer pays the buyer fee. The buyer fee goes directly to Envato and we're not taxed on it. We are taxed on the author fee though, which goes directly to Envato. Why we're taxed on the author fee, I'm not really sure.
As for Envato doing nothing, it may seem that way, but they spend millions every year to maintain and promote the site. I'd happily bypass Envato if I was able to generate 200 sales a month from my own website, but that's not going to happen. As a result, I'm happy for Envato take a cut of my money. After all, 50% of something is better than 100% of nothing!
Not sure where you're getting the 15% from. As an exclusive author, the minimum you'll get is 50%. Even if you factor in the taxes, you're from a non-treaty country, and all your sales are to the US... the minimum you'll get is 26%. Sure, that's not much, but it's a lot more than 15%.
As an author from India, you should be paying 10% withholding, which would result in you getting 42% of the price rather than 26%. And as US sales probably make up around a half, then you're really getting around 46%... and you should be able to offset that 4% against taxes in India, then you're not really losing anything in the long run.