russell@unturf.com
84d5b5cd76
Lands aborist/qa/model_profiles.py with two profiles:
- adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic
tight caps for broad intensities (ALL=8, COMPREHENSIVE=5,
OPEN_REQUEST=5) reflecting the 2026-05-02 runaway case.
- default
large-reasoner-class fallback (ALL=12, COMPREHENSIVE=15,
OPEN_REQUEST=12). Used when model_profile_id has no entry.
cap_for_intensity() resolves caps via three-source chain:
1. policy_overrides (per-call dict, highest priority)
2. per-model profile from PROFILES
3. "default" profile fallback
EXPLICIT_COUNT sentinel handles SMALL_NUM_EXPLICIT and
COMPARATIVE_BOUND — cap is the question's explicit count, not a
profile-set value. Defensive fallback to MANY cap if classifier
fired the rung without extracting a count.
Four new policy fields, all folded into governance_policy_hash via
_VERIFIER_POLICY_FIELDS:
- quantifier_guard_enabled master kill (default True)
- quantifier_guard_apply_caps dry-run gate (default False per
§10.11.3 — cap LOOKED UP and reported
on result, but NOT applied to the
verifier until operator flips True)
- quantifier_caps_by_intensity per-call override dict
- quantifier_guard_modes per-mode opt-in list (default
["claim_lattice_pointer",
"claim_lattice"]; quote opts out)
Six-level disable hierarchy (§10.11.2) implemented:
- Per-test: policy={"quantifier_guard_enabled": False}
- Per-call: --no-quantifier-guard (Phase 4)
- Per-phase: each policy switch is independent
- Per-mode: quantifier_guard_modes filter
- Per-model: model_profiles.py lookup
- Master: governance_policy_hash invalidation on flip
Wired through both query() and runner.ask() — both compute
effective_max_claims from the (classifier_intensity, model_profile,
policy_overrides) triple and pass it as max_claims_per_answer to
the verifier. Dry-run mode keeps effective_max_claims at the policy
default (12) until apply_caps flips True.
Result dict surfaces claim_cap_applied (the LOOKED-UP cap, even in
dry-run) plus all Phase-1 quantifier fields on miss-path AND
cache-hit path so bench rows stay column-aligned.
19 new tests pin: per-model selection, EXPLICIT_COUNT sentinel,
override precedence, governance-hash invalidation on every cap
field, profile shape (all ten rungs covered), default profile
presence.
132 lines
5.1 KiB
Python
132 lines
5.1 KiB
Python
"""Per-model quantifier-cap profiles — Ticket #000008 Phase 2.
|
|
|
|
PROMETHEUS-Σ owns the mapping ``model_profile_id → quantifier_intensity
|
|
→ claim_cap``. Each profile is a static dict; profile selection is by
|
|
exact match against ``model_profile_id`` (the configured chat-completion
|
|
model id, e.g. ``adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic``). Falls
|
|
back to ``"default"`` profile when no entry matches.
|
|
|
|
Folds into ``governance_policy_hash`` via two policy fields surfaced in
|
|
``aborist.qa.runner.DEFAULT_POLICY`` /
|
|
``aborist.qa.query.DEFAULT_QUERY_POLICY``:
|
|
|
|
quantifier_caps_by_intensity per-call cap dict (overrides the
|
|
profile when set on the policy)
|
|
quantifier_guard_apply_caps bool — Phase 2 dry-run gate
|
|
(default False through rollout)
|
|
|
|
The profile table lives here; the *application* of the cap lives in the
|
|
runner (`aborist/qa/runner.py:ask`) and `aborist/qa/query.py:query`.
|
|
This file is pure data + a lookup helper.
|
|
|
|
Per ticket §10.11.3 dry-run discipline: Phase 2 lands the cap-table and
|
|
the lookup wiring with ``quantifier_guard_apply_caps=False``. The
|
|
``claim_cap_applied`` field on the query result reports the cap that
|
|
WOULD have been applied — letting bench measure classifier+profile
|
|
behavior across the full question set before any verdict moves.
|
|
|
|
Once dry-run telemetry is reviewed and the classifier is calibrated
|
|
(per §10.11.3 step 3), an operator flips ``quantifier_guard_apply_caps``
|
|
to True for the targeted broad-quantifier subset.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Mapping
|
|
|
|
# Symbolic count for SMALL_NUM_EXPLICIT — the cap is the explicit
|
|
# count from the question, not a profile-set value. Resolved at
|
|
# lookup time (`cap_for_intensity` reads `explicit_count` when it
|
|
# sees this sentinel).
|
|
EXPLICIT_COUNT = "explicit_count"
|
|
|
|
|
|
# Per-model intensity → cap. Keys are model ids exactly as returned
|
|
# by the configured chat client (case-sensitive). Add new entries as
|
|
# we calibrate caps for new models.
|
|
|
|
PROFILES: Mapping[str, Mapping[str, int | str]] = {
|
|
# Hermes-3 Llama-3.1-8B-FP8-Dynamic — the live production endpoint
|
|
# (hermes.ai.unturf.com). Tight caps reflect the model's broad-
|
|
# quantifier weakness surfaced 2026-05-02 by the
|
|
# "winners of all major sports?" case (51-claim runaway).
|
|
"adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic": {
|
|
"ABSENT": 1,
|
|
"SINGULAR": 1,
|
|
"PROPORTIONAL": 1,
|
|
"SMALL_NUM_EXPLICIT": EXPLICIT_COUNT,
|
|
"COMPARATIVE_BOUND": EXPLICIT_COUNT,
|
|
"FEW": 5,
|
|
"MANY": 8,
|
|
"ALL": 8,
|
|
"COMPREHENSIVE": 5,
|
|
"OPEN_REQUEST": 5,
|
|
},
|
|
# Default — large-reasoner-class models (Qwen 3 reasoner / GPT-4 /
|
|
# Claude / Gemini). Looser caps because format discipline holds
|
|
# under broad-quantifier pressure on these models. Used as the
|
|
# fallback when model_profile_id has no explicit entry.
|
|
"default": {
|
|
"ABSENT": 1,
|
|
"SINGULAR": 1,
|
|
"PROPORTIONAL": 3,
|
|
"SMALL_NUM_EXPLICIT": EXPLICIT_COUNT,
|
|
"COMPARATIVE_BOUND": EXPLICIT_COUNT,
|
|
"FEW": 5,
|
|
"MANY": 12,
|
|
"ALL": 12,
|
|
"COMPREHENSIVE": 15,
|
|
"OPEN_REQUEST": 12,
|
|
},
|
|
}
|
|
|
|
|
|
def cap_for_intensity(
|
|
*,
|
|
model_profile_id: str | None,
|
|
intensity: str,
|
|
explicit_count: int | None = None,
|
|
policy_overrides: Mapping[str, int | str] | None = None,
|
|
) -> int | None:
|
|
"""Return the claim cap for ``(model_profile_id, intensity)``.
|
|
|
|
Resolution order:
|
|
1. ``policy_overrides`` (per-call dict) wins if present and
|
|
contains the intensity. Lets the operator override the
|
|
table without touching this file.
|
|
2. Per-model profile entry from ``PROFILES``.
|
|
3. ``"default"`` profile fallback.
|
|
|
|
The ``EXPLICIT_COUNT`` sentinel resolves to ``explicit_count``
|
|
when it's an int, or the default-profile MANY cap (12) when no
|
|
explicit_count was extracted (defensive — should not happen for
|
|
SMALL_NUM_EXPLICIT / COMPARATIVE_BOUND but covers regressions).
|
|
|
|
Returns None when the intensity is not in any profile (caller
|
|
should treat None as "no cap; use the policy default").
|
|
"""
|
|
sources: list[Mapping[str, int | str]] = []
|
|
if policy_overrides:
|
|
sources.append(policy_overrides)
|
|
if model_profile_id and model_profile_id in PROFILES:
|
|
sources.append(PROFILES[model_profile_id])
|
|
sources.append(PROFILES["default"])
|
|
|
|
for src in sources:
|
|
if intensity in src:
|
|
cap = src[intensity]
|
|
if cap == EXPLICIT_COUNT:
|
|
if explicit_count is not None and explicit_count > 0:
|
|
return int(explicit_count)
|
|
# Defensive fallback — no explicit count parsed.
|
|
# Use the MANY cap from the same profile as a
|
|
# sensible upper bound.
|
|
return int(src.get("MANY", 12))
|
|
return int(cap)
|
|
return None
|
|
|
|
|
|
def profile_id_present(model_profile_id: str | None) -> bool:
|
|
"""Return True iff ``model_profile_id`` has its own entry in
|
|
PROFILES (i.e. won't fall through to the default profile)."""
|
|
return bool(model_profile_id) and model_profile_id in PROFILES
|