🧪 Annotator tests (LMArena 2024 subset)

• The 101 general LMArena (2024) comparisons of the Feedback Forensics paper's human study (Appendix G.2), with ten personality traits labelled by two human labellers, and the same traits annotated by AI annotators. Human annotators: both labellers individually and their soft unanimous aggregate (a trait is relevant if at least one labeller selected a response and the labellers did not select opposite responses). AI annotators from the paper (annotated 2025-09-24): GPT-4o-mini, GPT-4.1-mini, GPT-5-mini and Gemini-2.5-Flash with 3-vote unanimous voting, and Gemini-2.5-Flash with single votes (gemini-2.5-flash-single), three runs (seeds) each. AI annotators from a later annotator comparison (annotated 2026-11-09): claude-haiku-5.5, deepseek-v4.1-flash, gemini-2.5-flash, gemini-3.5-flash-lite, gemini-3.8-flash, gpt-6-luna, each with single votes in both response orders and position-debiased votes (kept only where both orders agree). Each AI annotator voted on 40 traits per comparison; only the ten human-labelled traits are included. Includes prompts, responses and LMArena metadata. Annotator run details are in metadata.custom_metadata. The human and paper AI annotations are also available in human_personality_annotations.json in rdnfn/ff-annotations.
• Source: https://huggingface.co/datasets/lmarena-ai/arena-human-preference-100k

🏟️ LMArena (Llama4 special)

• Llama-4-Maverick-03-26-Experimental arena results, combined with public weights version of Llama-4-Maverick. Personality traits annotated with Gemini-2.5-Flash.
• Source: https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03-26-Experimental_battles/tree/main/data

🏟️ LMArena (2024)

• 10k subsample of LMArena text dataset (100k) released alongside Arena Explorer work, crowdsourced human annotations from between June and August 2024 in English, including topic labels automatically generated by Arena Explorer pipeline. LMArena is also known as Chatbot Arena. Personality traits annotated with Gemini-2.5-Flash.
• Source: https://huggingface.co/datasets/lmarena-ai/arena-human-preference-100k

🦙 AlpacaEval

• 648 cross-annotated human preference pairs used to validate AlpacaEval annotators. (Note: legacy annotations - used older annotation pipeline version)
• Source: https://huggingface.co/datasets/tatsu-lab/alpaca_eval/

🚑 Anthropic helpful

• 5k subsample of human preference pairs favouring helpful responses from RLHF dataset by Anthropic. (Note: legacy annotations - used older annotation pipeline version)
• Source: https://github.com/anthropics/hh-rlhf

🕊️ Anthropic harmless

• 5k subsample of human preference pairs favouring harmless responses from RLHF dataset by Anthropic. (Note: legacy annotations - used older annotation pipeline version)
• Source: https://github.com/anthropics/hh-rlhf

💎 PRISM

• ~8k human preference pairs from PRISM dataset, focused on controversial topics with extensive annotator information. Originally four-way annotations, subsampled using 1-of-3 rejected responses to get pairwise preferences. Personality traits annotated with Gemini-2.5-Flash.
• Source: https://huggingface.co/datasets/HannahRoseKirk/prism-alignment

🏋️ OLMo-2 0325 pref-mix

• 10k preference pairs subsampled randomly from original 378k pairs used for fine-tuning OLMo 2 model by Ai2. Synthetically generated via multiple different pipelines. (Note: legacy annotations - used older annotation pipeline version)
• Source: https://huggingface.co/datasets/allenai/olmo-2-0325-32b-preference-mix

🔄 MultiPref

• 10k preference pairs, each annotated by 4 human annotators as well as GPT-4-based AI annotators. Whilst each pair is annotated by 4 human annotators, these annotators are not identical across all pairs (i.e. more than four annotators overall worked on the dataset). Personality traits annotated with Gemini-2.5-Flash.
• Source: https://huggingface.co/datasets/allenai/multipref

🎭 Model Personality Comparison

• Model Personality Comparison dataset between openrouter/openai/gpt-4o-2024-11-20, openrouter/openai/gpt-4.1-mini, openrouter/x-ai/grok-4, openrouter/google/gemini-2.5-pro, openrouter/moonshotai/kimi-k2, openrouter/meta-llama/llama-4-maverick, openrouter/mistralai/magistral-medium-2506, openrouter/anthropic/claude-sonnet-4, openrouter/openai/gpt-oss-20b, openrouter/openai/gpt-5, openrouter/mistralai/mistral-medium-3.1, openrouter/anthropic/claude-sonnet-4.5, openrouter/z-ai/glm-4.6, openrouter/anthropic/claude-haiku-4.5, openrouter/google/gemini-3-pro-preview, openrouter/openai/gpt-5.1, openrouter/openai/gpt-5.1-chat, openrouter/allenai/olmo-3-32b-think, openrouter/allenai/olmo-3-7b-instruct, openrouter/allenai/olmo-3-7b-think, openrouter/mistralai/mistral-large-2512, openrouter/google/gemini-3.1-pro-preview, openrouter/mistralai/mistral-medium-3-5, openrouter/anthropic/claude-opus-4.7, openrouter/openai/gpt-5.3-chat, openrouter/anthropic/claude-3.7-sonnet, openrouter/openai/gpt-5-chat, openrouter/openai/gpt-3.5-turbo. Using openrouter/openai/gpt-4o-2024-11-20 as reference model(s). Created and annotated using Feedback Forensics, see https://huggingface.co/datasets/rdnfn/ff-model-personality for more details.
• Source: Unknown source