I've got both types, and wear what I want, when and where I want.How about a fedora, or a trilby which is similar but with a narrower brim. For fedora think 1940s Indiana Jones, for trilby think 1960s Sean Connery as James Bond. I would go trilby.
View attachment 1265632
My only advice is this: if you feel self-conscious, then you need to not buy that hat. Go to a place to try on different styles, like someone else already said.
Right off the top of my head I'd guess you'd FEEL more comfortable in a Trilby, since you want to be able to fit through the door.
I've got Tilley's coming out of my ears, but I rarely wear any except my wool winter one that has the fold out forehead and ear warmers. A finer looking/feeling hat is not made by ANYONE else.