← All selected work

Mizan: Arabic-English Policy Benchmark

Tests whether a policy clause keeps its meaning across English, Arabic and mixed text

Starter benchmark · 2026 Source
github.comOpen full size ↗Mizan overview showing a live benchmark run with per-language accuracy for the local model and GPT, agreement across languages and median response time

The problem

Policy work in the Gulf moves between English and Arabic. A model that labels a clause as an obligation in English should give the same label for the Arabic and mixed-language versions. This benchmark checks whether that holds, and what a small local model gives up against a hosted one.

My contribution

Personal project: the clause dataset and protocol, a local scoring bridge for Qwen3.5-4B, the GPT baseline with structured outputs, budget enforcement, and the results dashboard.

  • FastAPI
  • SQLite
  • Qwen3.5-4B (MLX)
  • GPT-5.2
  • React + TypeScript

What I built

The approach

A benchmark that classifies policy clauses as obligations, prohibitions, permissions, definitions or procedures, in English, Modern Standard Arabic and mixed-language versions of the same clause. It compares a local 4B model scored by direct option logits with a hosted GPT baseline.

Full live run (6 Oct 2026)

60 / 60correct for both models on the starter set
0mslocal median latency (GPT: 1,207ms)
$0.00estimated GPT cost for all 60 clauses

One run on 60 synthetic clauses (20 English, Arabic and mixed triplets) that have not yet had bilingual expert review. Both models scoring 100% means this set is too easy to separate them. Latency compares a warm local Mac with a hosted API, not equal hardware.

Follow the workflow

Each policy clause exists in English, Modern Standard Arabic and a mixed version with the same meaning.

Engineering decisions

01

Test meaning, not translation

Each clause comes as an English, Arabic and mixed triplet, so the benchmark can check that all three receive the same label.

02

Score the local model without generation

The local model picks a label from option logits with zero generated tokens, so there is no free text to parse.

03

Cap spend before a run starts

A conservative cost estimate is checked against a per-run ceiling before any hosted call is made.

Limitations & next steps

Have a bilingual domain expert review the clauses, add a harder held-out set with negation and exceptions, and repeat runs across prompts before comparing models.