The other day, while wearing a Panama hat, I was in line at a store. They guy in front of me asked me was I Mafia. He said "that hat" made me look like one of those Mafia men in the movies. I told him no, but I worked for a power company.
Still can't think of why a Panama hat would have such a connotation. I have a Fedora that makes me look like a hood, but never associated this with a Panama hat.
