Slicebench

What this is

Slicebench is a measurement bench for language model outputs. You give it a prompt, pick one model or several, and tell it how many times to run the same request. It stores every response verbatim — including the exact model string the provider returned, not just the one you asked for — and computes statistics over the set with an explicit confidence tier on each number.

Four gauges read each run. Zipf rank–frequency slope fits a line to the log-log plot of how often each word appeared. Vocabulary variety measures how much of the wording is fresh rather than repeated, in fixed 50-word windows so a long answer is not punished for its length. Lexical density is the share of words carrying content rather than grammar glue. Most-used words gives you two boards — raw counts, and counts with grammar words removed — always shown side by side, because filtering is a judgment and should never run unlabelled.

Every result carries a confidence tier (solid / tentative / insufficient / not-applicable) and a one-line plain-language explanation of what the number means and when to ignore it.

What the gauges can distinguish

• Whether a model's word-frequency distribution is stable across regenerations of the same prompt.
• Large shifts in lexical diversity or vocabulary range between two sets of responses.
• Whether a model converges on particular words. Per-model boards pool that model's regenerations and report how many of them used each word, so "34 uses across 8 of 8 generations" is legibly different from one long-winded outlier owning the board.
• Whether the provider silently swapped the model you requested (the returned model string is logged and flagged).

What a run gives you back

• What it cost. Tokens actually used, priced server-side from reported usage at OpenRouter's published rates, beside the pre-run estimate. This is not your bill — catalog pricing is not billing truth, so check your OpenRouter dashboard for that. A partial total is shown with its coverage stated and its confidence dropped, never quietly presented as a final number.
• Named failures. A failed generation says who failed and what to do about it, not a bare status code.
• Export. Download a run as JSON or CSV, with or without the generated text.

What it cannot distinguish (yet)

• Short responses. Below roughly 60 words there is no readable frequency curve at all, and the gauge reads "insufficient" rather than inventing a number. A Zipf slope only reads "solid" at about 400 words and up. Do not read significance into small-N, short-text results.
• Meaning, quality, or correctness. Slicebench measures text; it does not grade answers.
• Code, and text that isn't English prose. The gauges here assume English prose, and they do not all fail the same way, so do not read a run of source code as if it were an essay. Output that is mostly symbols or numbers reads "not-applicable" on the Zipf, vocabulary-variety and lexical-density gauges. But code that is mostly words — TypeScript, say — clears that check: onlylexical density reliably stands down on it, because its word list is English-only. Vocabulary variety and Zipf will still report a number at reduced confidence, and the most-used-word boards have no prose check at all. Treat any gauge reading over non-prose as unsupported, whatever tier it claims.
• Provider-level differences. You can compare up to 50 models in one run, but every request is routed through OpenRouter — so this measures models as OpenRouter serves them, not direct provider integrations. Models are also never pooled into one combined average, because an aggregate mixing different models is a meaningless number.
• Hedging, register, and style — still ahead. The gauges here are word-frequency gauges.