LLM Evaluation Harness

Overview

A comprehensive framework for evaluating the capabilities of Large Language Models. Supports popular benchmarks like MMLU, GSM8K, and HumanEval.

Features

• Automated Benchmarking: Schedule runs across multiple models.
• Visual Dashboard: Compare performance side-by-side.

graph TD
  A[Input Prompt] --> B[Model Inference]
  B --> C[Score Evaluation]
  C --> D[Results Dashboard]

Metrics

We use perplexity $PPL(X)$ as one of our core baseline metrics:

$$PPL(X) = \exp \left( -\frac{1}{t} \sum_{i=1}^t \log p_\theta(x_i | x_{<i}) \right)$$

Example Code

from eval_harness import evaluate
results = evaluate(model="gpt-4", benchmark="mmlu")
print(results.accuracy)