Lots of sanding very lightly with a high grit (1000 or 1500) should do it. You're most likely going to change the draw a bit as well, moreso if you use lower grit sandpaper.
I did sand for a good bit with 800, didn't seem to help. A tester razor (completely trashed edge, scratches all over the place, no real pattern), showed signs of paste scratch after 20 passes and picked up a bold and evident paste scratch pattern after ~50 passes. I'll try it a bit more, but failing that I've heard that Goop hand cleaner may work.
Ok, I dropped down coarser and completely resurfaced it. Seems to have done the trick. Thanks.
Now if I could just get the last bits out of the linen side. A run through the wash didn't do it. Probably just going to have to repeat several times until it's gone.