Same LLM, Different API Responses: Did the Model Change?

Separate a changed answer from changed behavior

Two endpoints with the same model name can return different answers to the same prompt without either serving a replacement model. Sampling allows multiple completions. A single pair of answers cannot show whether the broader pattern has shifted or whether you simply drew different samples.

Model Equality Testing, by Irena Gao, Percy Liang and Carlos Guestrin, asks a more precise question: do responses from the target API follow the same distribution as a reference, for the prompt distribution you care about? The method compares two collections of prompt–completion pairs rather than asking the model to identify itself.

A reference can be an implementation using published weights or a saved snapshot of responses from an earlier API version. The first comparison concerns fidelity to that reference implementation. The second concerns stability over time. Neither comparison identifies an undisclosed replacement model by name.

Quantization, fine-tuning, watermarking, decoding settings and inference implementation can affect the responses. A detected difference gives a reason to investigate those possibilities; it does not select one cause automatically. For a migration decision, the next question is whether the changed behavior affects successful work.

The paper found two very different ways to fail a coding task

One result explains why distribution testing cannot replace task evaluation. In the paper’s HumanEval experiments, some pairs had a large Hamming MMD distance but little difference in accuracy. Several such pairs had accuracy below 10% on both sides.

The researchers describe a comparison involving the then-current Perplexity endpoint for Llama-3.1 8B and an nf4-precision version. The endpoint produced incorrect code snippets, while the nf4 version produced nonsensical repeated text. These outputs were quite different, yet both often failed to solve the coding problems. This is a historical example from the study, not a finding about either service today. See the paper’s task-accuracy analysis.

HumanEval evaluates functional correctness by testing generated solutions against the task requirements. Its repository is now a historical, unmaintained reference. The useful principle for a current application is to apply the same requirements and tests to both candidates, rather than equating a different-looking response with worse code.

What a response collection needs to contain

The researchers’ Python package accepts collected samples. It does not turn two endpoint URLs into a complete audit by itself. Each response must remain associated with the prompt that produced it.

In the README’s example, prompt IDs are integers and can repeat. A first sample has prompt IDs [0, 1, 0]; a second has [0, 0, 1]. The corresponding completion array follows each collection’s own order. The following coding-task illustration uses that shape; it is not measured API output or a prompt set from the paper.

The first and second rows within the collections are not automatically matched pairs. Prompt IDs preserve which task each completion answers. The m=2 parameter in CompletionSample means the prompt distribution supports two prompt identities, not that there are two responses. Three completions illustrate the data structure; they are not a recommended audit sample size.

Use matching prompt distributions and generation settings when collecting real data. If one endpoint receives more sorting prompts and the other more deduplication prompts, a difference in the resulting text can reflect the task mix. Different system instructions, conversation histories or retrieved documents also change what is being compared.

The README converts strings into Unicode code points and pads both collections to a shared maximum length. A common character representation avoids comparing tokenizer-specific IDs as if they meant the same thing. The preprocessing policy must also match: truncating one side differently can introduce a difference, while truncating both too early can remove the part you wanted to inspect.

Read the p-value as a test result

The method summarizes differences through an MMD statistic. The Hamming kernel uses comparisons at corresponding token or character positions. It is not a correctness scorer, so output format and wording can influence it even when the task result is unchanged.

run_two_sample_test returns a p-value and the statistic. The README uses a permutation procedure: combine the samples, repeatedly shuffle their reference/candidate labels and recompute the statistic. This estimates how large a statistic could arise under the same-distribution assumption. The p-value uses the frequency of permuted statistics at least as large as the observed one. Appendix A.2 provides the procedure.

As a hypothetical interpretation, suppose the significance level was set to 0.05 and the test returns 0.02. That test rejects the same-distribution hypothesis. It does not say there is a 2% probability that the models are identical, and it certainly does not say one model has lost 98% of its quality.

A p-value above the chosen threshold means the test did not detect a difference in these samples. A small collection may miss infrequent behavior changes. Separately, Hamming MMD is a pseudometric, so its features cannot distinguish every pair of different distributions, even with more samples. These limitations are why a migration evaluation also needs task results.

Ten samples per prompt is not ten calls in total

The ICLR 2025 presentation reports median power of 77.4% against a range of distortions, including quantization and watermarking, with an average of ten samples per prompt for a language-modeling task defined over 25 prompts.

For that setting in Section 4.1, N is 250 samples from each distribution. Both the reference and target collections are needed. If no reference responses have been saved, collecting them is part of the work too. “Average per prompt” also differs from a rule that every prompt must be called exactly ten times.

Power describes how often a test detects a real difference under the evaluated alternative. It is not model-identification accuracy and is not a promise for a new summarization, support or coding workload. The paper estimates it through repeated simulations across model and prompt conditions.

The experiments also show why the conditions matter. Detecting fp16 versus fp32 was harder than detecting several other distortions. Moving from token space to character space reduced power for quantization and watermarking comparisons. A practical evaluation therefore needs a prompt set representing the application, a recorded sampling budget and task-level inspection when important failures remain despite no detected distribution shift.

What the 11-of-31 API result actually tested

The work was first submitted in October 2024 and revised in April 2025. Its commercial audit sampled 31 endpoints across nine providers serving four Llama models in Summer 2024. Eleven endpoints were flagged as differing from the reference distributions.

The reference was more nuanced than “one fp32 implementation is correct.” The audit allowed an endpoint to match either an fp32 or fp16 reference. It conducted both comparisons and required rejection against both to reject the composite same-distribution hypothesis. Matching an accepted reference was therefore treated differently from deviating from both.

The audit used Wikipedia continuation, plus HumanEval and UltraChat for the smaller models. It included repeated tests and a multiple-comparison correction when combining tasks. A single p-value from the README’s small example does not reproduce that entire protocol.

Eleven counts endpoints, not eleven deceptive companies. The result does not identify every implementation change behind each difference, and it is not a current provider ranking. What carries into a new evaluation is the explicit reference, task distribution and comparison procedure.

Turn the evidence into a migration decision

Consider an application that asks for a function returning integers in ascending order without changing the input list. The following is an application example, not a measured result from the study. Fix the requirements and test cases before comparing the API responses.

Collect repeated responses from both APIs under the same request and settings, then apply the same tests to each response. Different variable names and explanations can still produce correct functions. Similar-looking code can fail the no-mutation requirement. These are application outcomes that text-distribution distance alone cannot settle.

Keep the distribution-test sample size and p-value alongside successful completions, total completions and actual failure cases for each task. Do not compare one provider’s best response with all responses from the other. Execute generated code in an existing evaluation environment separated from production data, and retain execution errors as well as functional failures.

For larger collections, Google’s LLM Comparator helps analyze side-by-side evaluation results by subsets and individual outputs. That can reveal whether failures cluster around empty inputs or duplicate values even when overall averages look similar. The tool helps inspect comparison results; it does not replace the distribution test or execute the correctness tests on your behalf.

If a distribution shift accompanies more requirement failures, those failures provide concrete reasons to delay migration. If the distribution changes but the same task requirements remain satisfied, the shift alone does not establish a quality regression. And if no shift is detected while an important test fails, that failure still needs resolution before the API can be trusted for that task.