Most natural type places like Whole Foods will have the Kiss My Face version of this. Shouldn't be too expensive.
Check TJ Maxx and Marshalls. They usually have imported high quality soaps, including pure olive oil based, found at department stores for cheap.
Any Castile soap is at least 30% olive oil.