Exact estimation with estimand plugins

Business context: Before trusting approximate samplers on production prompts, teams need ground-truth attributions on tiny player sets. Exact enumeration is the reference path for estimator validation and compliance sign-off on toy models.

Goal: Walk through every public exact-estimation API — coalition enumeration, estimand delegation, budget guards, and plugin resolution.

Prerequisites: mllm-shap, Python 3.12.

Theory: See the exact estimation theory page.

Expected output: Eight coalitions for three players; Shapley and Banzhaf diverge on a majority game; Shapley matches coefficients on an additive game.

from itertools import product

from mllm_shap import (
    BanzhafAggregator,
    Estimand,
    ExactEstimator,
    ExplainConfig,
    ShapleyAggregator,
)
from mllm_shap.domain.coalition import CoalitionMask
from mllm_shap.domain.players import PlayerSet
from mllm_shap.plugins import (
    PluginContext,
    PluginGroup,
    PluginRegistry,
    register_builtin_plugins,
)

Coalition enumeration

Business: Exact runs must declare how many model calls will occur — exponential in player count, but fully predictable for audit planning.

Technical: ExactEstimator.iter_masks lazily yields all 2^n masks, from the empty to the grand coalition. Inside ExplainRunner the grand coalition is scored from the precomputed base generation, so it costs no extra backend call. Materialize with list(...) only for tiny player sets.

player_set = PlayerSet(player_ids=("p0", "p1", "p2"))
estimator = ExactEstimator()

first_mask = next(ExactEstimator.iter_masks(player_set))
sampled_masks = list(
    estimator.sample_masks(
        player_set,
        budget_fraction=1.0,
        include_minimal_masks=False,
        seed=0,
    )
)
print("num_coalitions:", ExactEstimator.num_coalitions(player_set.num_players))
print("sampled:", len(sampled_masks))
print("first mask (lazy):", first_mask.present)

num_coalitions: 8
sampled: 8
first mask (lazy): (False, False, False)

Majority game — estimand divergence

Business: Reporting Banzhaf values under a Shapley label invalidates fairness claims; exact enumeration makes the estimand choice explicit on identical payoffs.

Technical: estimate_attributions delegates to ShapleyAggregator or BanzhafAggregator without embedding weights in the estimator.

all_masks = [
    tuple(bits) for bits in product([False, True], repeat=player_set.num_players)
]


def majority_payoff(mask: tuple[bool, ...]) -> float:
    """One when at least two players are present."""
    return 1.0 if sum(mask) >= 2 else 0.0


payoffs = [majority_payoff(mask) for mask in all_masks]
coalition_masks = tuple(CoalitionMask.from_sequence(mask) for mask in all_masks)

shapley = estimator.estimate_attributions(
    coalition_masks,
    payoffs,
    ShapleyAggregator(),
)
banzhaf = estimator.estimate_attributions(
    coalition_masks,
    payoffs,
    BanzhafAggregator(),
)

print("Shapley:", shapley)
print("Banzhaf:", banzhaf)
print("diverge:", shapley != banzhaf)

Shapley: [0.3333333333333333, 0.3333333333333333, 0.3333333333333333]
Banzhaf: [0.5, 0.5, 0.5]
diverge: True

Additive game — Shapley ground truth

Business: Linear utility games are the sanity baseline for any new estimator — coefficients should recover exactly before scaling to LLM prompts.

Technical: The full 2^n table (including the empty and grand coalitions) makes every marginal lookup complete, so Shapley values sum to v(N) - v(∅).

coefficients = (1.0, 2.0)
two_player_masks = [
    tuple(bits) for bits in product([False, True], repeat=len(coefficients))
]
additive_payoffs = [
    sum(coef for coef, present in zip(coefficients, mask, strict=True) if present)
    for mask in two_player_masks
]
additive_masks = tuple(CoalitionMask.from_sequence(mask) for mask in two_player_masks)

additive_shapley = estimator.estimate_attributions(
    additive_masks,
    additive_payoffs,
    ShapleyAggregator(),
)
print("coefficients:", coefficients)
print("Shapley:", additive_shapley)

coefficients: (1.0, 2.0)
Shapley: [1.0, 2.0]

Budget guard and plugin resolution

Business: Partial budgets belong to approximate estimators; exact mode must fail fast rather than silently under-sample.

Technical: register_builtin_plugins plus load_entry_points(PluginGroup.ESTIMATORS) resolves the exact entry point.

try:
    estimator.sample_masks(
        PlayerSet(player_ids=("p0",)),
        budget_fraction=0.5,
        include_minimal_masks=False,
        seed=0,
    )
except ValueError as exc:
    print("budget guard:", exc)

registry = PluginRegistry()
register_builtin_plugins(registry)
registry.load_entry_points(PluginGroup.ESTIMATORS)
config = ExplainConfig.model_validate({"backend": {"kind": "mock", "model_id": "stub"}})
resolved = registry.resolve(
    PluginGroup.ESTIMATORS, "exact", PluginContext(config=config, registry=registry)
)
print("plugin name:", resolved.name)
print("Shapley estimand:", ShapleyAggregator().estimand is Estimand.SHAPLEY)

budget guard: exact estimator requires budget_fraction == 1.0
plugin name: exact
Shapley estimand: True