The reasoning I use is that it's not one hair. It's a different one every time. If you cut every one that sticks out, pretty soon you won't have any left.
Seriously, though, leave 'em be. They don't hurt anything. =D
After using and rinsing, drop it into a toilet paper tube and leave it there until it's completely dry. Then lift the tube up and off.
It "molds" those wild hairs up against the main loft.
One of my brushes had a single hair that was darker and stiffer than all the rest. It may not of mattered, but every time I used it, I thought I felt it. I trimmed down that single hair as low as I could easily reach and I'm no longer bothered by it.
I certainly would not try to trim down very many of them though.