SociaBench · Independent reproduction report
CharacterEval: An Independent Reproduction Report
Scope. This platform is not affiliated with the authors of CharacterEval. Its CharacterEval leaderboard lists the results the authors published, read unchanged from the paper's LaTeX source; no model has yet been evaluated on CharacterEval here, because the benchmark's reward model is not served by any inference provider (Section 3).
Abstract
CharacterEval [1] evaluates Chinese role-playing conversational agents on 77 characters from Chinese novels and scripts. A model continues a multi-turn dialogue as the character; the authors' reward model CharacterRM [3] scores twelve metrics on a 1–5 scale, and a thirteenth, personality back-testing, measures how well the model's answers to an MBTI questionnaire match the character's known type. The authors published results for 15 models, which this platform lists unchanged. The paper defines no overall score; the leaderboard ranks by character consistency, which the authors call the most crucial aspect, and states this on the board. The printed dimension averages agree with their metrics within rounding.
1 The original benchmark
1.1 Sources and data
| Source | Version | Licence |
|---|---|---|
| Paper [1] | arXiv:2401.01275 v2 (9 Jan 2024); ACL 2024 | arXiv licence |
| Code, test set [2] | commit c3d44a6 (28 May 2025) | MIT |
| CharacterRM [3] | commit e953da5 (10 Jan 2024); Baichuan2-13B-Base backbone with a scoring head; custom code | None stated |
The data comprise 1,785 multi-turn dialogues (9.28 turns on average) with 77 characters; profiles come from Baidu Baike. Examples (character, context, response) are split into 6,811 for training the reward model and 4,564 for testing. Each test example is scored only on the subset of metrics annotators judged relevant to it (sparse evaluation).
1.2 Metrics
| Metric | Definition | Dimension |
|---|---|---|
| Flu. | Fluency | Conversational ability |
| Coh. | Coherency | Conversational ability |
| Cons. | Consistency | Conversational ability |
| KE | Knowledge exposure | Character consistency |
| KA | Knowledge accuracy | Character consistency |
| KH | Knowledge hallucination (higher = fewer) | Character consistency |
| PB | Persona behavior | Character consistency |
| PU | Persona utterance | Character consistency |
| HL | Human-likeness | Role-playing attractiveness |
| CS | Communication skills | Role-playing attractiveness |
| ED | Expression diversity | Role-playing attractiveness |
| Emp. | Empathy | Role-playing attractiveness |
| MBTI | Personality back-testing accuracy (0–1) | Personality back-testing |
1.3 Scoring
CharacterRM reads the character profile, the context, the response and the metric name, and outputs a value that the released code maps to 1–5 (output × 4 + 1); the score is deterministic. A metric's score is the mean over the examples assigned to it; the paper reports a mean per dimension. The reward model was trained on scores from 12 annotators; its overall Pearson correlation with human scores is 0.631, against at most 0.385 for GPT-4 with one to three demonstrations. Personality back-testing uses 54 characters whose MBTI types were collected from a public database; the paper reports accuracy without specifying whether it is computed per type or per letter.
2 Published results and their provenance
The leaderboard lists the 15 models of the paper's Table 4, extracted by a script (tools/experiments/charactereval_paper.py) from the LaTeX source, which records its SHA-256 and writes the same data module for the server and the worker. It is the same table as the leaderboard image in the authors' repository. CharacterGLM has no personality back-testing result. Each printed dimension average agrees with the mean of its printed metrics within rounding (0.001 on three-decimal values). The table was not compared number by number with the ACL camera-ready version.
Because the paper defines no overall score, the leaderboard ranks by the character-consistency average and labels the column accordingly; every other dimension and metric can be used for sorting. Personality back-testing is an accuracy and is shown as printed, outside the score charts. The published results carry no confidence intervals, generation settings or evaluation dates; GPT-4 and GPT-3.5 are gpt-4 and gpt-3.5-turbo-1106.
3 Reproduction plan
Running CharacterEval here requires serving BaichuanCharRM, a 13B model with custom modelling code, at full precision; no inference provider offers it. Generation is a single reply per example and needs no new infrastructure. The paper states that the same prompt was used for every model with minor adjustments for closed-source APIs, but the prompt for API models is not published; the released generation script targets ChatGLM3. Before platform results are listed they will be checked on models the paper reports and that are still available, and ranked together with the published results, each row marked with its origin.
4 Limitations
- The leaderboard reproduces the authors' numbers; without per-example outputs they cannot be checked independently.
- The evaluations were run before January 2024; current versions of the same products may score differently.
- Ranking by character consistency is this platform's choice, grounded in the authors' text; the paper itself ranks no models overall.
References
- Q. Tu, S. Fan, Z. Tian, R. Yan. CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation. ACL 2024. arXiv:2401.01275 (v2, 9 January 2024). https://arxiv.org/abs/2401.01275
- morecry. CharacterEval: code, test set and reward-model training data. GitHub, commit c3d44a6 (28 May 2025). MIT licence. https://github.com/morecry/CharacterEval
- morecry. BaichuanCharRM (CharacterRM checkpoint). Hugging Face, commit e953da5 (10 January 2024). https://huggingface.co/morecry/BaichuanCharRM