I agree that this is very much a YMMV thing dependent upon the coarseness of your beard and the sensitivity of your skin.
For those with a light beard and less sensitive skin, grain direction might not matter much.
I agree this is ymmv, and I suspect your beard hair has quite an influence. I have no problem with ATG on the third pass. I can do it on the second pass, but it is not particularly pleasant. On the first pass it feels a bit like torture.
But like most things, a little experimenting should show what works for you.