Then that's probably it, since the irritated spots are where the beard grows in those swirly patterns you mentioned I guess I have to accept the little irritation I get with the alum bar or settle for a more comfortable but not as close shave.
Those swirly spots are the exact reason I switched to DE/SE/ST shaving. I was getting terrible burn in those areas. It took awhile, but I got to a point where I can get a very close shave in those spots (I wouldn't say BBS, but definitely DFS) without issue now.
Sounds like you're well on your way to getting there, too!! Congrats.
Not sure what blades you're using, but you'll likely find that certain blades will do better in those areas.