Since decimate comes from the Latin decimatio, and decim is Latin for ten, the Romans likely understood it in the sense of military unit forced to kill one in ten of their own.
Yes, "decimatio" is the removal of one tenth, and the root of "decimate" and "decimation." I need to be a bit picky, though. "Decem" is Latin for ten (think "December"), and "decimus/a/um" is Latin for "tenth" (think "decimal" or even "decimate"). I imagine it all worked out in some way beyond what I recall from those three years of Latin I completed thirty years ago!