Don't know if it has been mentioned yet, but simply take out a "No Trespassing" complaint with your local law enforcement.
I am a pretty hot headed person about such things and would not have been so nice as you. I would probably be sending this rep from prison if he got in my face like that...just genetics, I suppose.
Its simple to make a "No Trespassing" complaint with your local law enforcement..at least it was for my crazy azz neighbor...(long story about him letting his pit bulls run freely in my yard...) the wuss wouldn't even step out in the street where we could, shall we say get better acquainted....
Good luck
My grandkids usually play in the front yard only when I am there. I have a rule about pit bulls- or any large dog for that matter. If they charge onto my lawn and the grandchildren are there, animal control will be removing the carcass.