SociaBench · Independent reproduction report
CoSER: An Independent Reproduction Report
Scope. This platform is not affiliated with the authors of CoSER. Its CoSER leaderboard lists the results the authors published, read unchanged from the paper's LaTeX source; no model has yet been evaluated on CoSER here (Section 3).
Abstract
CoSER [1] evaluates role-playing of established characters through given-circumstance acting (GCA): a model plays every character of a scene taken from a renowned book in a multi-agent simulation, and GPT-4o identifies flaws against the original conversation in four dimensions. The authors published results for 21 models, averaged over three runs, with an overall score defined as the mean of the four dimensions. This platform lists them unchanged, ranked by that average. The table is identical in all three arXiv versions and each printed average agrees with its dimensions within rounding. Two features of the protocol are relevant when reading the ranking: the critic, GPT-4o, is also among the evaluated models, and the next-speaker predictor is the authors' CoSER-70B, which is evaluated as well.
1 The original benchmark
1.1 Sources and data
| Source | Version | Licence |
|---|---|---|
| Paper [1] | arXiv:2502.09082 v1–v3 (results table identical); ICML 2025 | arXiv licence |
| Code and test data [2] | commit 624efc5 (2 Apr 2026) | None stated in the repository |
The dataset covers 17,966 characters from 771 books. CoSER Test holds 200 conversations from the final 10% of books: 100 from books used to train the CoSER models and 100 from books not used. Test conversations are in English.
1.2 Given-circumstance acting and scoring
- Simulation. The evaluated model plays every character, with the scenario, character profiles and motivations. CoSER-70B predicts the next speaker, GPT-4o acts as the environment, and the simulation stops at an end signal or after 20 turns. The main results start from scratch (k = 0) and judge without inner thoughts.
- Judging. GPT-4o, given the original conversation and the plot summary, lists flaws per dimension in separate requests; each flaw has a severity from 1 to 5. A dimension scores 100 − 5 × the sum of severities, plus 1.5 × the simulation length as a length correction [3].
- Dimensions. Storyline consistency (against the original), anthropomorphism, character fidelity and storyline quality. The overall score is their mean; BLEU and ROUGE-L against the original dialogue are reported for analysis only.
The paper reports 68.6% agreement between the GPT-4o critic and human judges.
2 Published results and their provenance
The leaderboard lists the 21 models of the paper's main results table, extracted by a script (tools/experiments/coser_paper.py) that records the source's SHA-256 and writes the same data module for the server and the worker. The table is identical in arXiv v1, v2 and v3. Each cell is printed as a mean with a ± value; the paper states that results are averaged over three runs but does not define the ± further, so the leaderboard shows the means only. Each printed average agrees with the mean of its four dimensions within rounding.
Model versions are as given in the paper (e.g. GPT-4o 2024-08-06, Claude-3.5-Sonnet 2024-06-20, Doubao-pro 241215). CoSER-8B and CoSER-70B are the authors' models trained on this dataset and are marked on the board. The paper's appendix re-scores the models with other judges and reports judge-specific preferences.
3 Reproduction plan
Unlike CharacterBench and CharacterEval, CoSER's critic is a commercial model available through APIs; running it here requires implementing the multi-agent simulation and serving CoSER-70B (or substituting another next-speaker predictor, which would be a registered deviation). Platform results will be checked on models the paper reports before they are listed, and ranked together with the published results, each row marked with its origin.
4 Limitations
- The leaderboard reproduces the authors' numbers; without the simulations they cannot be checked independently.
- The critic is also an evaluated model, and the next-speaker predictor is one of the authors' evaluated models.
- GPT-4o snapshots change over time; current results with the same model name may differ.
References
- X. Wang, H. Wang, Y. Zhang, et al. CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation. ICML 2025. arXiv:2502.09082 (v1 13 Feb 2025; v3 4 Jan 2026). https://arxiv.org/abs/2502.09082
- Neph0s. CoSER: code, test data and GCA evaluation. GitHub, commit 624efc5 (2 April 2026). https://github.com/Neph0s/CoSER
- Y. Dubois, B. Galambosi, P. Liang, T. B. Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv:2404.04475, 2024. https://arxiv.org/abs/2404.04475