Same and I am becoming more comfortable with that instead of assuming I am just doing it wrong. I am very confident in my bevel sets if I am going back that far, I don't want to do that again to keep spine wear down. After remembering to start inking my edges, I realised I simply wasn't spending enough time in the middle. I kept thinking I had to move on or I would be over doing it.Ah, good to hear an expert say that. Here I've assumed I am among people who get it perfect on the bevel setter, and everything else is a couple of superficial passes.
That's not how it works for me. I can get it really wonderful on the bevel-setter, shave-worthy (yes, I've tried that shave), but still find that I need to do significant work as I go up the grits. I like to think of it as like magnification (or higher energies in particle colliders, if that resonates). The more fine-grained your measuring, the more subtly you can examine things.
The one thing that that brings up though is whether having 2 or 3 different mid range stones isn't more beneficial that collecting dozens of finishers? Would it make sense to put the razor on my 5k and decide it is working too slow for that razor and switch to a 3k instead of assuming I need to go back to the bevel setter? I am still fighting spending $100 on an 8k that is only going to see less that 20 laps whereas, mid-range stones are more affordable.