CharacterBench: An Independent Reproduction Report

1 The original benchmark

1.1 Sources

The paper states that the data and models are released for research only and that the morality data contain harmful content. The platform downloads the test set at the pinned commit at run time, does not copy it into its own repository, and does not display test samples publicly: the public list of test samples shows only their dimension, sample id and character name.

1.2 Dimensions

Eleven dimensions are grouped into six aspects; attribute and behavior consistency are each tested with bot-side and human-side queries, giving 13 result columns. Sparse dimensions use target-oriented queries; dense ones (morality, believability) use target-free queries. Each test file has 250 samples.

1.3 Evaluation procedure

1. Generation. The paper’s role-play prompt (its Table 11) packs the character profile and the dialogue context into one message and asks for a reply within 15 characters (Chinese) or 15 words (English). The released process.py uses a different prompt (Section 3.1). Each sample is generated in Chinese and in its English translation.
2. Judge input (construct_prompts/process_wo_context_{zh,en}_all.py). For each test file, a dedicated prompt combines profile, context, user turn, reply and, for sparse dimensions, the target (e.g. the correct fact or the target emotion).
3. Judging. CharacterJudge outputs an integer score; the paper decodes with self-consistency over 10 outputs [4].
4. Aggregation (all_eval.py). A column is the mean score divided by the column’s maximum and multiplied by 5; AVG is the plain mean of the 13 columns.

2 Published results and their provenance

The leaderboard lists the 18 models of the paper’s Table 4 and the two models of its appendix table (GPT-4o and Claude-3.5-Sonnet; their rows are commented out in the source of Table 4 but published in the appendix). Values were extracted from the LaTeX source by a script (tools/experiments/characterbench_paper.py) that records the source’s SHA-256 and writes the same data module for the server and the worker; no value was typed by hand. CharacterYuyan has no English result.

Aspect scores on the leaderboard are means of the printed columns. The paper’s Table 5 reports aspect scores computed from unrounded values, so the two can differ by 0.01. Because the paper reports Chinese and English separately and defines no bilingual score, the leaderboard ranks each language separately and offers no combined view.

The published results contain no per-sample outputs, no confidence intervals, no generation settings and no evaluation dates; model versions are only as precise as the names in the paper (e.g. GPT-4-1106).

3 Sources of the protocol

3.1 Every setting and its source

Table 4 lists every setting that can affect scores, what the paper and the released code say about it, what the platform does, and why. The sources are the paper’s LaTeX source (arXiv e-print v1; aaai25-main.tex and appendix.tex, whose SHA-256 the check script pins) and the repository at commit 66c113c. The check script (Section 7) reads each source value from these files and compares it with the platform’s.

3.2 Reproduction policy

Each setting is taken from the first of these that defines it: (1) the paper’s text, and where its text and its reported numbers disagree, the reading that reproduces the numbers on the released data; (2) the released code at the pinned commit; (3) the released artifacts (test set, judge checkpoint and its generation_config); (4) a platform choice, stated as such. The judge input is the one exception to (1) over (2) in kind rather than in conflict: the paper gives its contents, not its text, and the released checkpoint only reads the format it was trained on, which the code defines. No setting was chosen or adjusted after seeing a model’s results.

The platform departs from the released code where the paper says otherwise (the role-play prompt and, following from it, the reply the judge reads) and in four places where the code cannot be run as released:

1. The model under evaluation is called. gpt_call.py sends model="gpt-4o" whatever model name it is given. The platform calls the model being evaluated, with the generation parameters of its endpoint, which every run records.
2. Ten judge samples, averaged. The judge script draws one sample; the paper reports self-consistency over 10 outputs. The paper’s text calls the combination a majority vote, but on the 3,250 annotated test responses only the mean reproduces the correlations of its Table 3, and every self-consistency prediction released with the test set is a multiple of 0.1 (Table 9). The platform draws 10 samples at the checkpoint’s generation_config and averages them.
3. One judge for both languages. The paper describes separate Chinese and English judges; one checkpoint is released.
4. An unreadable first line. When a judge output has a line break and its first line is not a number, all_eval.py stops with an exception. The platform reads such an output as 3.0, the value the script gives an unreadable output without a line break, and counts it. None occurred in the validation (Section 5).

Two points are platform choices because neither source covers them: surrounding white space is removed from a reply, and an empty reply is requested again (the released judge script leaves empty responses out); and at least 1,024 output tokens are allowed, far above a 15-word reply, because some APIs require a limit and reasoning tokens count toward it. A third detail is the judge endpoint’s: it returns each output with surrounding white space removed, which changes the reading of an output only where all_eval.py would stop (an output beginning with a line break); none occurred (Section 6).

3.3 Where the released code departs from the paper

Protocol 1.2 ran the released code here (Section 9.2). Its differences from the paper are recorded because they explain why results under 1.2 and 2.0 differ, and because they are what a reader running the released code would get.

The length instruction. The paper’s prompt asks for a reply within 15 words in English and 15 characters in Chinese; the released prompt asks for “15 characters” in English and 30 characters in Chinese, as JSON. Models differ widely in how literally they take “15 characters” (Table 5): two of four answered almost every English sample within 15 characters, and one answered 87% of English samples in Chinese.

Three models answered the same 52 English samples (4 per column) twice: with the released prompt, and with only “15 characters” replaced by “15 words”. The change raised one model’s score by a third of a point and left the other two unchanged within noise (Table 6), so the released prompt does not affect all models alike and can change their order.

What the judge reads. With the released code the judge reads Python’s str() of the parsed JSON object, and Chinese replies with their full-width ，。 replaced by , . (the clean-up applied before parsing). Judging the 312 replies of Table 6 both ways with the same seeds changed the mean sample score by +0.02 (95% interval −0.004 to +0.05). Under 2.0 the reply is plain text and the judge reads it as written.

Out-of-range judge scores remain part of the protocol, since only the code defines how a judge output is read: all_eval.py keeps a score outside the column’s range. In the validation, the judge answered 5 on the 1–4 scale of English attribute consistency (bot query) in 339 of 2,500 sampled outputs.

4 The protocol on this platform

Protocol version characterbench-2.0.1 · 2026-10-27. Every run stores its version, the prompt templates and the judge settings in its snapshot, so a run can be audited against the protocol it used. Runs made under 1.x are not on the leaderboard. Runs made under 2.0 made exactly the calls 2.0.1 makes; they are on the leaderboard once their stored judge outputs have been re-scored under 2.0.1, which the run records together with a log entry (Section 9.2).

4.1 Test set and prompts

The platform downloads 37 files from the repository at commit 66c113c: the 13 test files and the judge prompt modules for both languages. It records the SHA-256 of each file; the manifest of all 37 (one line “path hash” per file, sorted, each ending with a line break) has SHA-256 ae33d898…4c1276c5. Judge templates are read from the Python sources as string constants; no code from the repository is executed. The role-play prompt is the paper’s Table 11, held in the platform’s code; the check script extracts the table from the paper’s LaTeX source and requires the two to be identical, character for character (lines the table wraps for width are joined; the table’s own spacing, such as “[Character Profile-End ]”, is kept).

The prompt is filled as process.py fills its placeholders, including Python’s formatting of non-string values: in each language 109 character profiles are dictionaries and appear as Python’s str() of the dictionary. The dialogue context is written as process.py writes it, one “role: content” line per message (roles “user” and “assistant”); a dialogue that would start with the character starts with an empty user turn instead: 3,199 samples per language, 3,156 of them because the sample has a greeting and 43 because its dialogue opens with the character. The judge’s dialogue history numbers turns by position, user first, and omits the greeting, as the judge script does.

process.py normalizes the Chinese and English versions of a sample together and drops both when either fails. Three samples fail, all in the English translation, and are dropped in both languages: empathetic responsiveness #22 (a speaker name that matches neither participant) and morality robustness #48 and #84 (the dialogue does not end with the user). The platform keeps 3,247 samples per language.

Equivalence check. A script fills the paper’s Table 11 (as extracted from the LaTeX source) with the authors’ own functions (norm_session_zh/en from process.py; process from process_wo_context_{zh,en}_all.py for the judge input) on the downloaded files and hashes every prompt; a second script hashes the prompts the platform builds. For all 6,494 samples, both the role-play prompt and the judge prompt are identical, and the same three samples are dropped (Section 7). The audit of Section 6 repeats this on the samples production actually holds and on every reply the platform’s runs actually judged.

4.2 Generation

The model under test receives the role-play prompt as the only message (no system message) and is called with the generation parameters of its endpoint, which the run records, with at least 1,024 output tokens. The reply is the model’s output with surrounding white space removed. As in process.py, a sample’s Chinese and English replies are requested together; if either is empty, both are requested again, up to five attempts in all, after which the sample is excluded from all scores in both languages, shown in the run as excluded, and counted. As in process.py, a sample whose replies are read only on the fifth attempt is excluded too (2.0.1; no sample of a 2.0 run was in that case). The model’s raw output is stored, and the cost of all attempts is kept in the record.

The output limit is the one place where generation differs from the released code by construction, since the code sends none. A reply that reaches the limit is kept as cut and judged. In the 8 protocol-2.0 runs audited in Section 6, 595 of 51,842 replies (1.1%) reached it, almost all from one run whose replies ran to thousands of characters against a 15-word instruction; the released code would have let them run longer. Each stored reply keeps its finish reason.

4.3 Judging

CharacterJudge is served by the platform operator with vLLM behind an OpenAI-compatible endpoint, stated to be the checkpoint at commit f2272fa. The endpoint itself reports no revision or weight hash. The check that would settle it, characterjudge_server_check.py (in the audit bundle), reads the model directory from the serving process’s command line and hashes every weight and tokenizer file against the SHA-256 Hugging Face publishes for f2272fa; it has to be run by the operator on the server and has not been run yet, so Table 10 lists the weights as unverifiable. Its address and key are deployment settings stored on the server, not part of the protocol, and its calls are not billed to the run. Each reply is judged once, on its sample’s column, with the system message “You are a helpful assistant.” and the judge prompt as the user message; the reply appears in the prompt after the character’s name and a full-width colon, as the released code builds it.

One request returns 10 samples (n = 10) at the checkpoint’s generation_config (temperature 0.7, top-p 0.8, top-k 20, repetition penalty 1.05; at most 512 new tokens). The request carries a seed fixed by the sample (the first four bytes of the SHA-1 of “judge:” followed by the sample’s id, as an unsigned integer masked to 31 bits), so that repeating a run on the same deployment returns nearly the same samples. When all 3,250 + 3,250 validation items (Section 5) were judged again a day later with the same seeds, 94.2% (Chinese) and 93.2% (English) returned the same ten outputs, and a sample’s score moved by 0.010 and 0.013 points on average. The server batches concurrent requests, which changes floating-point results, so exact repetition is not guaranteed; Section 6 measures what it does to scores. Each of the 10 outputs is read as all_eval.py reads it (the first line as a number if the output has a line break, else the whole output; 3.0 when it is not a number; a number outside the column’s range is kept), and the sample’s score is their mean. Out-of-range and unreadable outputs are counted in the judgement record.

4.4 Scores

Per language, a column is the mean sample score divided by the column’s maximum and multiplied by 5, and the overall score is the mean of the 13 columns, as in all_eval.py (the divisor there is the largest human score in the file, which equals the scale maximum for every file). Sample scores are kept unrounded; only displayed values are rounded to two decimals. Protocol 2.0 rounded each sample’s column score to 0.01 before averaging, which raised four-point columns by about 0.0025 and changed 15 of 224 displayed values by 0.01 in the audited runs, without changing any ranking; 2.0.1 corrected it (Section 6). Aspect scores are the means of their columns. Confidence intervals, which the paper does not report, are a percentile bootstrap over samples (1,000 resamples). A run enters the leaderboard only with the current protocol, the full test set in both languages, every sample generated and judged (excluded samples count as complete), and a single host for the model under test.

5 Judge validation

Each test sample ships with the response the annotators scored, its English translation, the human score, and the authors’ own CharacterJudge predictions (single-sample and self-consistency). This lets the judge be validated as deployed, before any model is evaluated: the judge input is built with the authors’ function for each annotated response, judged as in Section 4.3 and, for comparison, greedily, and compared with the human scores by Pearson correlation per column, the metric of the paper’s Table 3. The paper’s correlations were measured on these test responses with separate Chinese and English judges, so differences can come from the single released checkpoint as well as from decoding.

Findings. Averaged over the 13 columns, the deployed judge with self-consistency correlates with the human scores at 0.675 (Chinese) and 0.631 (English), against 0.68 and 0.64 in the paper; greedy decoding gives 0.633 and 0.594, against 0.64 and 0.60 for the paper’s “w/o SC”, so self-consistency adds about four points, as the paper reports. Per column, all but three Chinese and English correlations are within 0.05 of the paper. The exceptions are English morality stability (0.56 against 0.64) and morality robustness (0.46 against 0.55), and English engagement (0.59 against 0.53). The morality columns are near-binary (1–2) with about 10% unsafe responses, so a few disagreements move their correlation; the authors’ released predictions correlate higher there (0.64 and 0.56 in English). On the 5-point scale used for results, the mean over the 13 columns for the annotated responses is 3.60 with our judge and 3.62 with the released predictions in Chinese, 3.66 and 3.65 in English; the largest column difference is 0.23 (English boundary consistency).

In English attribute consistency (bot query), whose range is 1–4, the judge answers “5” in 339 of 2,500 sampled outputs and 10 of 250 greedy ones. The released script keeps these as 5, and so does the platform (Section 3.3); averaging only in-range outputs, as the platform did before protocol 1.2, would give this column a correlation of 0.765 instead of 0.772 and change no other column. No output was unreadable. Scores in this section were re-read from the same judge outputs when protocol 1.2 was adopted; no judge request was repeated.

Combining the ten samples. The paper says the self-consistency outcomes are combined by majority vote. The same judge outputs were combined both ways (Table 9). A vote, with any rule for ties, correlates with the human scores at the level of the paper’s own “w/o SC” row, so it cannot have produced the paper’s self-consistency figures; the mean reproduces them, and the authors’ released self-consistency predictions are means. The platform therefore takes the mean.

Judge outputs outside the column’s range: 0 of 32,500 sampled outputs in Chinese and 339 of 32,500 in English, and 10 greedy outputs in all; unreadable outputs: 0. Our greedy score equals the authors’ released single-sample prediction for 85% of Chinese and 80% of English responses; the mean absolute difference between our self-consistency score and theirs is 0.21 (Chinese) and 0.27 (English) points.

6 Reproduction verification

This section states what has been shown to be reproduced, by what test and over what range, and what cannot be shown. It was produced by running the authors’ own functions (read from the repository at 66c113c and executed unchanged) and the platform’s functions on the same inputs: the files of the repository, the samples and templates production actually holds (exported from its database), and every reply and judge output stored by the platform’s 8 completed protocol-2.0 runs. Platform runs enter only as test material, in totals; the report says nothing about any model. Three kinds of agreement are kept apart throughout: with the paper’s text, with the released code as it executes, and with the authors’ reported results.

6.1 What the audit found and changed

1. Score precision (corrected in 2.0.1). Protocol 2.0 rounded each sample’s column score to 0.01 before averaging; all_eval.py averages unrounded scores. On four-point columns half of all sample scores end in 5 at the third decimal, so the column was raised by about 0.0025. Emulated on the stored outputs of the 8 audited runs, 2.0’s aggregation changes 15 of 224 displayed values by 0.01 (on the runs audited before the correction, exactly the values found to differ from the authors’ program); no ranking changed. Version 2.0.1 keeps sample scores unrounded. The completed 2.0 runs were re-scored from their stored judge outputs (no model or judge was called again), each with a log entry and a record in its configuration, and the audit was then repeated on the re-scored runs: 0 differences.
2. The last attempt (corrected in 2.0.1). process.py drops a sample whose replies are read only on the fifth attempt; 2.0 kept it. No sample of a 2.0 run was in that case, so no result changed.
3. Output limit (disclosed). The limit the platform sets cut 1.1% of the audited replies; Section 4.2.
4. Report statements corrected. Section 4.4 claimed equality with all_eval.py while 2.0 rounded; Section 4.1 attributed all 3,199 leading empty user turns to greetings (43 are not); Section 4.3 gave a 300-item repeatability check, now replaced by the full-scale one; Section 7 listed expected outcomes instead of recorded ones.
5. Acceptance criteria made explicit. An earlier version of Table 10 marked judge sampling noise as passing although single columns varied by more than its stated 0.01, and named criteria such as “close agreement” without a threshold. Every row now states a measurable criterion, and its status is computed from it.
6. Recorded, not changed. The samples production holds carry a user-profile field left from the 1.x prompt, which no 2.x prompt reads (6,494 samples); every field the pipeline reads is identical to the authors’ construction.

6.2 Conclusions

1. Is the released test set and its preprocessing reproduced? Yes, completely: the same 3,247 samples per language are kept and the same three dropped, and every field the pipeline uses is identical for all 6,494 samples production holds.
2. Is prompt construction reproduced? The role-play prompt is the paper’s Table 11, identical for every sample; it is not the released code’s prompt, by design (Section 3). The judge input is identical to the released code’s for every sample and for every one of 51,842 real replies.
3. Are the judge input and the scoring algorithm reproduced? Yes under 2.0.1: every judge output is read as the authors’ code reads it, and columns, AVG and rankings equal the authors’ program up to floating-point error. Under 2.0 they did not, by up to 0.01 in displayed values.
4. Does CharacterJudge behave as in the paper? In Chinese, within 0.05 of the paper’s correlation on every column; in English, slightly below (AVG 63.1 vs 64, three columns more than 0.05 below), and with systematic column differences from the authors’ own released predictions. The paper used a separate English judge that is not released; whether the served weights are the released checkpoint cannot be verified.
5. Is the platform’s protocol equivalent to the chosen reference? To the reference stated in Section 3 (paper first, released code where the paper is silent): yes, stage by stage as in Table 10, with three stated departures from the released code (the paper’s prompt, ten averaged judge samples, the output limit) and one from the paper’s wording (mean, not vote, which its numbers require).
6. How far are the paper’s model scores reproduced? Not within the measured noise. Only one paper model can still be called in a plausibly matching version; its AVG differs from the paper by +0.04 in Chinese and +0.12 in English, while two runs agree within 0.01, and columns differ by up to 0.55. The differences follow the released judge’s own systematic differences only in part, and the model snapshot, its 2024 behaviour and the paper’s English judge cannot be checked (Section 9.1). The other published rows cannot be re-run: their versions are retired or not named precisely.
7. What cannot be shown to be 1:1? The judge weights served; the paper’s per-language judges; the paper’s generation parameters (not reported); the paper’s exact model versions; and sampled judge outputs across deployments (repeatable only to about 93–94% on one deployment).

7 Reproducing this report

Every number in Sections 3 to 6 can be recomputed. The audit is published as a bundle, /audit/characterbench/2026-10-27/, with the SHA-256 of every file: the scripts and the platform source they execute, each step’s output and its log (command, time, exit status, full output), and the frozen inputs of the scoring — every judge output of every audited run with the platform’s reading of it, every sampled validation output, and each run’s protocol snapshot and result. It contains no test-set text, dialogue or model reply. With only the bundle and the authors’ repository, a third party can recompute every audited run’s 13 columns and AVG with the authors’ own reading and aggregation (characterbench_recompute.py; recorded: all values equal at two decimals) and the judge statistics of Section 5 and Table 10. The results below are the ones recorded on 2026-10-27 for protocol characterbench-2.0.1 · 2026-10-27, not expectations.

1. Every setting against its source. Download the paper’s e-print (arXiv 2412.11912v1) and clone the authors’ repository at 66c113c; run tools/experiments/characterbench_spec_check.py eprint repo [val_zh.jsonl val_en.jsonl]. It checks the SHA-256 of the paper’s source files, extracts Table 11 and compares it with the platform’s prompt character for character, and checks the length instruction, the placeholders, the score scales (against the paper’s text and the test files), the number of judge samples, the normalization, the AVG, the judge request, the judge’s sampling settings (against the checkpoint’s generation_config), the number of attempts and the last-attempt rule, unrounded averaging, the size of the test set and the modified profiles; with the validation outputs of item 3 it also recomputes Table 9. Recorded: 20 of 20 checks passed.
2. Prompts. Run characterbench_spec_check.py table11 eprint to write the paper’s prompt, then tools/experiments/characterbench_prompt_check.py (the authors’ functions) and characterbench_prompt_hashes.ts (the platform), and compare the two hash files with --compare. Recorded: 6,494 samples, 0 mismatches.
3. Judge. Serve CharacterJudge f2272fa with an OpenAI-compatible server that supports n, seed, top_k and repetition_penalty; set CHARACTERJUDGE_BASE_URL and CHARACTERJUDGE_API_KEY; run tools/experiments/characterbench_judge_validation.py run for each language (with CB_SC_ONLY=1 and CB_SEED_OFFSET=k for the repeat and seed runs), then characterbench_judge_audit.py on the outputs. Sampled scores repeat almost exactly only on the same deployment; elsewhere the correlations should agree within the seed variation of Table 10.
4. The pipeline on real runs (Section 6). Export a run’s samples, prompt snapshot, replies and judge outputs from the database (read-only queries), then run characterbench_audit_platform.ts (the platform’s functions on the exported material) and characterbench_audit.py (the authors’ functions on the same material; compares data, prompts, judge input and scoring); characterbench_audit_module.py turns the outputs into the data of Section 6. Recorded: the figures of Table 10.
5. Paper anchor (Section 9.1). characterbench_anchor.py compares runs of a paper model with its row, read from the LaTeX source.
6. The released prompt (Section 3.3). tools/experiments/characterbench_length_ablation.ts and its summary script produced Tables 5 and 6 under protocol 1.2; they run against the released prompt and are kept as the record of that experiment.
7. A model. On the platform, open “运行评测”, choose the CharacterBench tab, and run the full test set. Every raw output, every judge output and the run’s protocol snapshot are kept and can be downloaded.

8 Third-party results

Two later papers report CharacterBench scores [5] [6]. They are not listed on the leaderboard: most models are research checkpoints trained by those authors and not released, one paper does not state the language evaluated, and the other contains apparent transcription errors (its table is captioned as another benchmark, one baseline row repeats another exactly, and its CharacterGLM row equals the English column of CharacterGLM-6B in the original paper).

9 Platform results and paper anchor

A platform run enters the leaderboard when it uses the current protocol, covers the full test set in both languages, every sample is generated and judged (excluded samples count as complete) and the model under test was served by a single host. Platform results and the published ones are ranked together on each language’s board, and every row shows its origin; equal scores at two decimals share a rank. The published results come from 2024 versions of the models, judged by the paper’s per-language judges, so a published row and a platform row are not a controlled comparison of the same model.

9.1 Paper anchor

A check of the whole pipeline against the paper’s own results needs a model the paper evaluated, in the same version. Of the paper’s 20 rows, only one can still be called in a version that plausibly matches: GPT-4o in the appendix table. The paper does not give its snapshot; gpt-4o-2024-05-13 is the only GPT-4o snapshot that existed before the paper’s submission and was what the name “gpt-4o” (which the released gpt_call.py calls) pointed to until October 2024, so it is the most likely one, not a certain one. GPT-4-1106, Claude-3-opus, GPT-3.5-turbo (as dated in 2024) and the open-source models of Table 4 are no longer served by the platform’s provider; the role-play products are not available in their 2024 versions. The anchor was run twice under protocol 2.0 through one host (Azure, via OpenRouter) with the provider’s default sampling, as experiment runs that do not enter the leaderboard, and re-scored under 2.0.1 from the stored judge outputs. Azure’s content filter refused the prompts of some morality samples in every attempt; as for any provider moderation, those samples were excluded in both languages after five attempts.

Result. The two runs agree with each other to within 0.01 on AVG in both languages, so generation and judging noise is small against the differences from the paper. In Chinese the AVG is +0.04 from the paper and 6 of 13 columns are within 0.10; in English the AVG is +0.12 and 2 columns are within 0.10, with the largest differences in behavior consistency. Across the 13 columns the differences follow the deployed judge’s own systematic differences from the authors’ released predictions (Section 5) with a correlation of 0.61 (Chinese) and 0.57 (English): part of the gap comes from the judge, consistent with the English judge of the paper not being the released checkpoint, and part remains unexplained.

What this can and cannot show. Agreement within the noise of Table 10 would show that nothing in the pipeline moves scores grossly; a difference cannot be attributed to the pipeline alone, because the paper’s judges were per language and are not released, its sampling settings and the model’s exact snapshot are not stated, and a 2024 model served today may not behave as it did then. The anchor therefore bounds the reproduction; it does not prove it.

9.2 Protocol history

1. 1.1 (2026-10-26) departed from the released code in three places found in an audit after the first runs: each language was generated on its own, the judge read the reply text, and out-of-range judge outputs were left out of a sample’s mean.
2. 1.2 (2026-10-27) ran the released code as released, including its role-play prompt, and departed from it only where it cannot be run (Section 3.2, items 1–4).
3. 2.0 (2026-10-27) follows the paper first (Section 3.2). Asked why a current model scored below the paper’s 2024 models in English, we found that the caption of the paper’s Table 11 names the prompt with which the general-purpose models of its Table 4 were evaluated, and that this prompt differs from the released one in the length limit (15 words, not “15 characters”, in English; 15, not 30, characters in Chinese), the output format (plain text, not JSON) and the user profile (absent). Under 1.2, published rows produced with one prompt were therefore ranked against platform rows produced with another. Every run made under 1.x was withdrawn from the leaderboard and is run again under 2.0. The release of 2.0 also added the check script of Section 7 and Table 9.
4. 2.0.1 (2026-10-27) corrects the two departures from the released code found by the audit of Section 6: sample scores are averaged unrounded, and a sample read only on its fifth attempt is dropped. Generation and judging calls are unchanged, so runs made under 2.0 were re-scored from their stored judge outputs instead of being run again; each such run records the re-scoring (versions, time, number of sessions) in its configuration and log.

10 Limitations

1. The leaderboard reproduces the authors’ numbers; their correctness cannot be checked without per-sample outputs.
2. The published evaluations date from 2024; current versions of the same commercial models may score differently.
3. The released judge is a single checkpoint for both languages, whereas the paper describes separate judges.
4. The paper describes the self-consistency combination as a majority vote; the platform takes the mean, which reproduces the paper’s reported correlations (Table 9). The sampling settings of the authors’ self-consistency runs are not stated; the platform uses the checkpoint’s generation_config.
5. The paper gives the role-play prompt but not how the dialogue context is written into it; the platform writes it as the released code does. Table 11 is stated for general-purpose models; how the paper prompted its four role-play-specific models is not described, and the platform gives every model Table 11.
6. Generation parameters of the models in the paper are not reported; platform runs use each endpoint’s recorded parameters.
7. Seeded sampling makes judge scores nearly repeatable on one deployment (about 93–94% of samples identical a day apart), not across servers or hardware.
8. The served judge’s weights cannot be verified against the released checkpoint from outside the serving endpoint; the evidence is statistical (Section 5 and Table 10).
9. The platform’s output limit cuts replies that run far beyond the 15-word instruction; the released code sets no limit (Section 4.2).
10. Only one of the paper’s models can be re-run in a probably matching version, so the paper’s model scores are reproduced only as far as Section 9.1 shows.

References

1. J. Zhou, Y. Huang, B. Wen, et al. CharacterBench: Benchmarking Character Customization of Large Language Models. AAAI-25, pp. 26101–26110. arXiv:2412.11912 (v1, 16 December 2024). https://arxiv.org/abs/2412.11912
2. thu-coai. CharacterBench: code and test set. GitHub, commit 66c113c (1 August 2025). https://github.com/thu-coai/CharacterBench
3. thu-coai. CharacterJudge (judge model checkpoint). Hugging Face, commit f2272fa (5 April 2025). https://huggingface.co/thu-coai/CharacterJudge
4. X. Wang, J. Wei, D. Schuurmans, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arXiv:2203.11171. https://arxiv.org/abs/2203.11171
5. Z. Xu, D. Chen, S. Wang, et al. AdaMARP: An Adaptive Multi-Agent Interaction Framework for General Immersive Role-Playing. arXiv:2601.11007, 2026. https://arxiv.org/abs/2601.11007
6. Y. Tang, K. Chen, M. Yang, et al. Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning. arXiv:2506.01748, 2025. https://arxiv.org/abs/2506.01748