Exact estimation with estimand plugins

Business context: Before trusting approximate samplers on production prompts, teams need ground-truth attributions on tiny player sets. Exact enumeration is the reference path for estimator validation and compliance sign-off on toy models.

Goal: Walk through every public exact-estimation API in nlp-shap 0.1.5 — coalition enumeration, estimand delegation, budget guards, and plugin resolution.

Prerequisites: nlp-shap 0.1.5+, Python 3.12.

Theory: See the exact estimation theory page.

Expected output: Seven coalitions for three players; Shapley and Banzhaf diverge on a majority game; Shapley matches coefficients on an additive game.

from itertools import product

from nlp_shap import (
    BanzhafAggregator,
    Estimand,
    ExactEstimator,
    ShapleyAggregator,
)
from nlp_shap.domain.coalition import CoalitionMask
from nlp_shap.domain.players import PlayerSet
from nlp_shap.plugins import PluginGroup, PluginRegistry, register_builtin_plugins

Coalition enumeration

Business: Exact runs must declare how many model calls will occur — exponential in player count, but fully predictable for audit planning.

Technical: ExactEstimator.iter_masks lazily yields 2^n - 1 masks, excluding the grand coalition that backends often precompute. Materialize with list(...) only for tiny player sets.

player_set = PlayerSet(player_ids=("p0", "p1", "p2"))
estimator = ExactEstimator()

first_mask = next(ExactEstimator.iter_masks(player_set))
sampled_masks = list(
    estimator.sample_masks(
        player_set,
        budget_fraction=1.0,
        include_minimal_masks=False,
        seed=0,
    )
)
print("num_coalitions:", ExactEstimator.num_coalitions(player_set.num_players))
print("sampled:", len(sampled_masks))
print("first mask (lazy):", first_mask.present)

num_coalitions: 7
sampled: 7
first mask (lazy): (False, False, False)

Majority game — estimand divergence

Business: Reporting Banzhaf values under a Shapley label invalidates fairness claims; exact enumeration makes the estimand choice explicit on identical payoffs.

Technical: estimate_attributions delegates to ShapleyAggregator or BanzhafAggregator without embedding weights in the estimator.

all_masks = [
    tuple(bits) for bits in product([False, True], repeat=player_set.num_players)
]


def majority_payoff(mask: tuple[bool, ...]) -> float:
    return 1.0 if sum(mask) >= 2 else 0.0


payoffs = [majority_payoff(mask) for mask in all_masks]
coalition_masks = tuple(CoalitionMask.from_sequence(mask) for mask in all_masks)

shapley = estimator.estimate_attributions(
    coalition_masks,
    payoffs,
    ShapleyAggregator(),
)
banzhaf = estimator.estimate_attributions(
    coalition_masks,
    payoffs,
    BanzhafAggregator(),
)

print("Shapley:", shapley)
print("Banzhaf:", banzhaf)
print("diverge:", shapley != banzhaf)

Shapley: [0.3333333333333333, 0.3333333333333333, 0.3333333333333333]
Banzhaf: [0.5, 0.5, 0.5]
diverge: True

Additive game — Shapley ground truth

Business: Linear utility games are the sanity baseline for any new estimator — coefficients should recover exactly before scaling to LLM prompts.

Technical: Supply all 2^n coalitions (including the grand coalition) when aggregating so marginal lookups remain complete.

coefficients = (1.0, 2.0)
two_player_masks = [
    tuple(bits) for bits in product([False, True], repeat=len(coefficients))
]
additive_payoffs = [
    sum(coef for coef, present in zip(coefficients, mask, strict=True) if present)
    for mask in two_player_masks
]
additive_masks = tuple(CoalitionMask.from_sequence(mask) for mask in two_player_masks)

additive_shapley = estimator.estimate_attributions(
    additive_masks,
    additive_payoffs,
    ShapleyAggregator(),
)
print("coefficients:", coefficients)
print("Shapley:", additive_shapley)

coefficients: (1.0, 2.0)
Shapley: [1.0, 2.0]

Budget guard and plugin resolution

Business: Partial budgets belong to approximate estimators; exact mode must fail fast rather than silently under-sample.

Technical: register_builtin_plugins plus load_entry_points(PluginGroup.ESTIMATORS) resolves the exact entry point.

try:
    estimator.sample_masks(
        PlayerSet(player_ids=("p0",)),
        budget_fraction=0.5,
        include_minimal_masks=False,
        seed=0,
    )
except ValueError as exc:
    print("budget guard:", exc)

registry = PluginRegistry()
register_builtin_plugins(registry)
registry.load_entry_points(PluginGroup.ESTIMATORS)
resolved = registry.resolve(PluginGroup.ESTIMATORS, "exact")
print("plugin name:", resolved.name)
print("Shapley estimand:", ShapleyAggregator().estimand is Estimand.SHAPLEY)

budget guard: exact estimator requires budget_fraction == 1.0
plugin name: exact
Shapley estimand: True