Lawrence Tierney...a real Hollywood tough guy.....Jack Palance....a boxer.....Victor McLagen.....6' 3"...boxer....boxer, British Provost Marshall (Military Police) of Bagdhad after WW 1.
I'm afraid I never really thought of John Wayne as a tough guy. Grumpy, yes, but not the kind of man I'd find intimidating. Maybe it's because his movies were made in a more restrained era. (I would consider most/all of his movies family-friendly.)
I voted Lee Marvin. If I met him in a dark alley (in his prime.) I'd be concerned.
Of course outside of movies, Bruce Lee would be the one with actual fight skills to worry about.