"Talk short. Drop grammar. Save token." This phrase is widely promoted as a way to reduce inference costs, but whether it saves anything depends on whether you compress the prompt or the response. We present CAVEWOMAN, a two-channel evaluation protocol that scores every question–answer pair on task accuracy, realized per-item cost, and whether the response still says what the model would have written without compression. It is tested on nine models, five short-answer benchmarks, and five grammatical reduction levels, with a ten-language replication and a two-dataset summarization study. Constraining the response reduces realized API cost in dollars by 1.7× on average and up to 3× without losing accuracy. Compressing the prompt raises net cost on three of the four API models, by up to 96%, and lowers accuracy at every level. Under output compression, over half of responses on the six non-reasoning models are correct but no longer entail the response the model wrote without the constraint. How much accuracy a model keeps under compression varies across models and languages. Task accuracy alone does not capture the effects of compression, so compression should be evaluated on accuracy, agreement with the unconstrained response, and cost, each reported separately.

Code is available at https://github.com/danielle34/cavewoman.