The LMSYS platform for serving and evaluating chat models, including the MT-Bench multi-turn judge harness and the infrastructure used for crowd-sourced model comparisons.
Source: Official GitHub repository. Last checked 2026-09-18. Spot an error? Tell us.
Nearby and similar
A library for building LLM applications that also auto-tunes them, optimizing prompt text and few-shot demonstrations against a training set. It aims to make prompt tuning as routine as model training.
A prompt tuning framework built around intent-based prompt calibration. It generates challenging edge-case examples, collects judgements on them and rewrites the prompt to close the gap.
An open framework and hosted service for building conversational AI interfaces with built-in observability, feedback collection and evaluation of the conversations produced.
A Python library for reproducible synthetic data and prompting workflows, with caching, resumable sessions and export of the full prompt history for a run.
A Python framework that replaces hand-written prompt strings with declarative modules and signatures, then compiles them against your data. Optimizers search for the wording and few-shot examples that score best on a metric you define.
An optimizer that improves prompts and other text artifacts through reflective evolution, reading execution traces and natural-language feedback to propose edits. It integrates with DSPy and other pipelines.