Graph Attention Engine

RGI for LLM Evals

LLM-as-Judge scores. GAE learns from them. Same eval rubric — only one discovers which criteria actually predict quality.

pip install graph-attention-engine # ~844 params, Apache 2.0

Side by Side

Score & Archive vs. Score & Learn

LangSmith runs your LLM-as-judge rubric and logs the scores. GAE takes the same scores, learns which criteria matter, and rewrites the rubric from verified outcomes.

LangSmith · Score & Archive

# LangSmith — LLM-as-judge pipeline from langsmith import Client from openevals.llm import create_llm_as_judge client = Client() evals = [ create_llm_as_judge( prompt=CORRECTNESS_PROMPT, model="openai:o3-mini" ), create_llm_as_judge( prompt=CONCISENESS_PROMPT, model="openai:o3-mini" ), ] results = client.evaluate( my_agent, data="summary_eval_dataset", evaluators=evals, experiment_prefix="summary-q3" ) # Scores logged. Dashboard updated. # Which criteria predict satisfaction? # No feedback loop. No way to know.

GAE · Score & Learn

# GAE — ~844 inspectable parameters from gae import ( ProfileScorer, run_ablation, run_evaluation, check_conservation, compute_theta_min ) scorer = ProfileScorer(mu=mu, actions=ACTIONS) # Learn from verified outcomes for task, scores, verified in eval_data: result = scorer.score(scores, cat_idx[task]) scorer.update( f=scores, category_index=cat_idx[task], action_index=result.action_index, correct=(result.action_name == verified), gt_action_index=act_idx[verified], ) # c += η·(outcome − c) # Which criteria predict quality? abl = run_ablation(scorer, scenarios, FACTORS) # → #1 correctness drop=+33% # → #5 conciseness drop=+8%

What Score-and-Archive Can Never Do

NO TRAJECTORY 4.2/5 for six months, complaints rising. No curve — rubric can't see drift.
NO VERIFICATION Two agents, same score. One improving, one gaming the rubric. No way to tell.
NO TRANSFER Year tuning summarization evals. Now need RAG retrieval. Start from zero.
NO DISCOVERY Five criteria. Which predict satisfaction? No outcome feedback — you'll never know.
NO DEFENSE "Conciseness" gets 4.8. Users prefer the longer version. Rubric can't self-correct.

Running Results

The Evidence

Same rubric. Five criteria: correctness, grounding, conciseness, fluency, relevance. 12 evals for training, 8 held-out for testing. All code below runs with pip install graph-attention-engine.

Learning Curve — 5 Rounds of Training

After each round, accuracy measured on 8 held-out evals the scorer has never trained on.

Ablation — Which Criteria Predict Quality?

Remove each criterion. Measure accuracy drop. Bigger drop = actually matters.

correctness

+33.3%

grounding

+33.3%

relevance

+25.0%

fluency

+16.7%

conciseness

+8.3%

Your rubric weights all five equally. GAE shows correctness matters 4x more than conciseness. The rubric rewrites itself from outcomes.

Transfer — Learned Geometry vs. Blank Slate

New task: RAG retrieval. Transfer scorer seeds from trained summarization geometry. Cold scorer starts flat. Both train on 2 RAG evals, test on 4 held-out.

3x accuracy from 2 examples. Structure learned from summarization transfers to RAG retrieval. Static rubrics start cold every time.

Gaming Detection

6 evals injected: agent scores high on correctness, verified outcomes all "poor." A gaming attack.

Predictions Wrong

6 / 6

All gaming evals caught

Centroid Drift

0.056

Trajectory increasing — attack

Accuracy Impact

58%→39%

Override rate: 100%

Conservation gate: AMBER. A static rubric sees 6 more data points. GAE sees an attack — centroid drift increasing, the gate holds. Static rubrics have no defense against this.

Conservation Gate — α·q·V ≥ θ_min

Coverage × accuracy × volume must clear a floor before any weight changes. Thin feedback can't corrupt a production rubric.

GREEN

V=120 evals · signal=8.16 · θ=2.45

Safe to update

RED

V=3 evals · signal=0.20 · θ=98.0

Weights frozen

At Scale — 2,000 Decisions

Judgment Memory (centroid learning) vs. k-NN retrieval (the mechanism behind agentic memory). 4 categories, 6 factors, 4 actions.

Crossover

N=639

JM overtakes k-NN

Final Gap

+6.2 pp

95% CI: [+3.7, +8.4]

Ablation

+9.2 pp

CI excludes zero at N=500

Test Suite

5,702

pytest · passing · 34.5s

Working Code

The Full Script

Everything above runs from one file. pip install graph-attention-engine, copy, run.

"""GAE for LLM Evals — running example (v6).""" import numpy as np from gae import ( ProfileScorer, EvaluationScenario, run_evaluation, run_ablation, compute_judgment, check_conservation, compute_theta_min, ) # Domain: LLM eval rubric CATEGORIES = ["summarization", "qa", "code_review", "rag_retrieval"] ACTIONS = ["poor", "fair", "good", "excellent"] FACTORS = ["correctness", "grounding", "conciseness", "fluency", "relevance"] # Training data — 12 evals across 3 task types CORE_EVALS = [ # Summarization: users prefer grounded, even verbose ("summarization", [0.8,0.9,0.3,0.8,0.9], "excellent"), ("summarization", [0.7,0.2,0.9,0.9,0.6], "fair"), ("summarization", [0.9,0.8,0.4,0.7,0.8], "good"), ("summarization", [0.3,0.1,0.8,0.6,0.3], "poor"), # QA: correctness + relevance dominate ("qa", [0.9,0.7,0.5,0.4,0.9], "excellent"), ("qa", [0.3,0.5,0.8,0.9,0.3], "poor"), ("qa", [0.7,0.6,0.6,0.7,0.8], "good"), ("qa", [0.5,0.3,0.7,0.8,0.4], "fair"), # Code review: correctness is everything ("code_review", [0.9,0.6,0.8,0.5,0.7], "excellent"), ("code_review", [0.2,0.3,0.9,0.9,0.4], "poor"), ("code_review", [0.7,0.5,0.7,0.6,0.6], "good"), ("code_review", [0.4,0.2,0.5,0.7,0.3], "fair"), ]

# PART 1: Multi-round learning scorer = build_scorer() for round_num, batch in enumerate(ROUND_BATCHES, 1): train_batch(scorer, batch) report = run_evaluation(scorer, test_scenarios) # → accuracy climbs as centroids refine # Round 1: 25% → Round 5: 38% # PART 2: Ablation — which criteria matter? ablation = run_ablation(scorer, all_scenarios, FACTORS) for name, drop in ablation.items(): print(f"{name}: drop={drop:+.1%}") # → #1 correctness drop=+33.3% # → #2 grounding drop=+33.3% # → #3 relevance drop=+25.0% # → #4 fluency drop=+16.7% # → #5 conciseness drop=+8.3% # PART 7: Calibration report report = run_evaluation(scorer, final_scenarios) # → Accuracy: 58% (7/12) # → ECE: 0.1358 # → summarization: 75% qa: 50% code_review: 50%

# PART 3: Conservation gate theta = compute_theta_min(alpha=0.08, V=120) check = check_conservation( alpha=0.08, q=0.85, V=120, theta_min=theta ) # V=120 → GREEN (safe to update) # V=3 → RED (weights frozen) # PART 4: Transfer — learned geometry vs blank slate transfer_mu = scorer.centroids.copy() transfer_mu[cat_idx["rag_retrieval"]] = \ transfer_mu[cat_idx["summarization"]] transfer_scorer = build_scorer(mu_override=transfer_mu) cold_mu = build_scorer().centroids.copy() cold_mu[cat_idx["rag_retrieval"]] = 0.5 cold_scorer = build_scorer(mu_override=cold_mu) # Both train on 2 RAG evals, test on 4 held-out train_batch(cold_scorer, RAG_TRAIN) train_batch(transfer_scorer, RAG_TRAIN) # → Cold: 25% (1/4) Transfer: 75% (3/4) # PART 6: Score a new eval with judgment judgment = compute_judgment( scoring_result=result, f=new_scores, mu=scorer.centroids, category_index=cat_idx["summarization"], factor_names=FACTORS, actions=ACTIONS, ) # → good (60% confidence) # → key factors: fluency, relevance, grounding # → Not a label: a distribution over all verdicts

# PART 5: Gaming detection GAMING_EVALS = [ # High correctness scores, verified outcomes all "poor" ("summarization", [0.9,0.8,0.7,0.8,0.9], "poor"), ("qa", [0.85,0.7,0.6,0.75,0.8], "poor"), # ... 4 more gaming evals ] pre_gaming = scorer.centroids.copy() for task, scores, verified in GAMING_EVALS: result = scorer.score(scores, cat_idx[task]) scorer.update(...) drift = np.linalg.norm(scorer.centroids - pre_gaming) # → drift=0.0563, override_rate=100% # → accuracy: 58% → 39% # → conservation: AMBER # PART 8: Scale — 2,000 decisions # JM (centroid learning) vs k-NN (agentic memory) # 4 categories × 6 factors × 4 actions # # Crossover at N=639 # Final gap: +6.2 pp [95% CI: +3.7, +8.4] # JM accuracy: 72.5% vs k-NN: 66.3% # # Ablation (drop 2 of 6 factors): # Effect: +9.2 pp [+7.2, +11.3] # CI excludes zero at N=500 # CI width tracks 1/√N reference # # Production: 5,702 pytest cases · all passing · 34.5s

Market Context

The Category Is Forming

Three independent results bracket the same mechanism. Different teams, different domains.

External convergence (not our numbers)

jev (Hugging Face) — Typed machine decisions: action + probability + confidence. RGI adds a provenance tier.

rrsi (Google Research) — Governed recursive self-improvement. Noise-adjusted floor blocking gains inside measurement variance.

GEPA (arXiv, ICLR 2026 oral) — Reflective prompt evolution outperforms RL at ~90x lower cost.

RGI for LLM Evals

Score & Archive vs. Score & Learn

What Score-and-Archive Can Never Do

The Evidence

Learning Curve — 5 Rounds of Training

Ablation — Which Criteria Predict Quality?

Transfer — Learned Geometry vs. Blank Slate

Gaming Detection

Conservation Gate — α·q·V ≥ θmin

At Scale — 2,000 Decisions

The Full Script

The Category Is Forming

External convergence (not our numbers)

Conservation Gate — α·q·V ≥ θ_min