IMO, it's because of the heavy animalic notes (civet, castoreum, cumin adds to it as well). Some have compared EdH to a tiger who was sprayed with Eau d'Orange Verte, it's probably not far off, and I mean that in a good way.
Unfortunately, Eau d'Hermes is no longer that powerful a fragrance. The new 2009 reformulation by Ellena has gutted the animalic to a whisper and increased the citrus to a point where the whole experience lasts only 60 minutes. It's all topnote, some medium note, and no more base note. What was once groundbreaking and polarizing is now pleasantly quaint. In this case the vintage versions would be worth exploring.