Release notes

Changes in each published mllm-shap version. Newest first.

Unreleased

Fixed

  • Wav2Vec2Ctc no longer prints the Transformers load report for the training-only masked_spec_embed weight; any other missing weight is logged as a warning.

0.6.0 (2026-10-12)

Added

  • The loo estimator gives leave-one-out values \(v(N) - v(N \setminus \{i\})\) from \(n + 2\) generations, the baseline any attribution method should beat.

  • The reference_wer value function scores 1 - min(1, WER) against a reference such as a transcript, or against the full answer.

  • deletion_curve() removes the top-k players at once for each k and rescores, to test faithfulness with a second value function.

  • copy() returns an independent copy of a registry.

0.5.0 (2026-10-12)

Added

  • Snapshot JSON carries audio: a message may give a "content" list of text and audio parts ("path" relative to the snapshot file, or base64 "data"), so mllm-shap explain runs on recordings (Command line).

  • sgpa_segments option transcript: align_only uses the transcript only to build the players and sends every coalition’s audio alone, so the values reflect what the model hears, not what it reads.

0.4.1 (2026-10-11)

Added

  • The transformers backend hears audio: a checkpoint AutoModelForCausalLM does not know (or any backend.options.model_class) loads with its processor and takes audio parts, for example Granite Speech (Audio input). backend.options.attn_implementation is passed to from_pretrained.

  • The api backend sends audio parts as OpenAI input_audio parts (base64 WAV) instead of refusing them.

  • The lmstudio option audio: transcript sends the text parts of a message with audio; by default audio is still refused, since the LM Studio SDK takes none.

Changed

  • The transformers extra adds torchaudio (Granite Speech’s feature extractor needs it), and the backend reads the model config before loading.

0.4.0 (2026-10-11)

Added

  • The mock_audio backend (MockAudioBackend) echoes message words and answers one token per voiced run of each audio part, so audio players get attributions in tests and demos without a model. It reads 16-bit WAV without the audio extra.

0.3.1 (2026-10-11)

Fixed

  • The api backend gives an empty reply the logprobs () instead of None (LM Studio sends logprobs: null for it), so reanalyze accepts archives of runs that contain empty generations.

0.3.0 (2026-10-11)

Added

  • players: tokens takes the plugin option roles (like messages): only the words of the listed roles become players, other messages stay present.

  • plot_conversation() and render_conversation_html() draw a conversation message by message under role and speaker headers, with token or message players shaded in place and text that is no player in grey. Messages with audio players get a waveform strip shaded on the same colour scale.

  • The sgpa_segments option text_roles also makes the words of typed messages of those roles players, so typed and spoken words share one run.

Changed

  • The project is renamed to mllm-shap: install mllm-shap from PyPI, import mllm_shap, and find the repository at Pawlo77/MLLM-SHAP. Environment variables start with MLLM_SHAP_, entry-point groups with mllm_shap., and the base error is MllmShapError. The old nlp-shap package is archived.

  • plot_audio_segments() with values lightens every second segment and keeps fills translucent, so neighbouring words stay distinct (equal values no longer merge into one block) and the waveform shows through.

0.2.1 (2026-10-10)

Added

  • explanation.players: sgpa_segments (SgpaSegmentPartitioner) makes each spoken word of an audio message’s transcript a player. Its PlayerSpan carries the new samples range, and delete cuts absent audio while pad and neutral silence it. See Audio alignment.

  • mllm_shap.Message.with_audio() replaces a message’s first audio part; AudioSegment records token_index and the rough CTC edges ctc_start_time and ctc_end_time.

  • plot_audio_segments() draws aligned segments over the waveform, the shift of each boundary from its CTC edge and the RMS energy; values colours segments by attribution.

0.2.0 (2026-10-10)

Added

0.1.19 (2026-10-09)

Fixed

  • The transformers backend warns once when explanation.kv_cache runs with float16 or bfloat16 weights: cached prefixes round differently from one forward pass, so greedy outputs can differ from the cache off. The backends guide no longer promises equality; backend.options.dtype: float32 is identical.

  • The LM Studio backend accepts a model key that LM Studio lists once per downloaded quantisation; earlier versions refused it as ambiguous.

0.1.18 (2026-10-09)

Fixed

  • Without model embeddings (mock, LM Studio and API backends), embedding_mode: contextual now hashes the generated text alone, so explain and reanalyze agree and reanalyze --strict accepts the archive (embedding_source: hash). The values of such runs change.

  • The token_text and token_bar renderers no longer log matplotlib’s Failed to find font weight semibold warning.

  • Renderer labels no longer overlap: token_text wraps long text onto new lines, and token_bar puts value labels beyond the bar ends and the sign legend outside the axes.

  • Complementary and Neyman samplers now evaluate all n singleton pairs ({i}, N - {i}) after the grand and empty pair. On an additive 8-player game at budget.fraction: 0.5 the attribution sum was on average 1.7 above grand_value - base_value; it is now within 0.03.

  • Neyman phase one walks its grid size by size, round-robin over players, so a short budget leaves the same sizes empty for every player; its small-budget error now matches complementary sampling (Approximate estimation).

  • Complementary and Neyman budgets never drop below 2n + 2 masks (2 ** n when smaller), so all singleton pairs fit. With the earlier 2n floor, the game v(S) = |S| with three players gave [1, 1, 1.333] instead of [1, 1, 1].

  • Monte Carlo and other partial-sample estimates log a WARNING naming the players with no observed marginal (their value is 0), typically at small budgets where only the grand and empty coalitions were sampled.

  • The transformers backend reuses cached prompt-prefix KV state (every prompt token but the last). A miss computes the prefix once and stores it. A hit copies the longest cached prefix, crops it to the shared tokens and extends it. No prompt token is processed twice; earlier versions ran one forward pass per prefix length.

  • kv_cache.max_entries (default 8) bounds the prefix states, and model calls (tokenization included) are serialized. metrics.kv_cache_reused_tokens reports the prompt tokens served from the cache next to metrics.kv_cache_hits. Both count the current run only, also when ExplainRunner(backend=...) reuses the backend. In float32 greedy outputs are identical with the cache on and off.

  • The transformers backend runs on Apple MPS when available. The new backend.options.dtype (auto, float32, float16, bfloat16) defaults to auto: bfloat16 on CUDA and MPS when supported, float16 on other accelerators and float32 on CPU. Earlier versions loaded float16 on every accelerator, which overflows to NaN on many bfloat16-trained models.

  • backend.options.revision pins the Hub revision of the model and tokenizer.

  • The transformers backend computes static and contextual embeddings only when the value function needs them (see RequiresEmbeddings), and explain then scores the model’s embeddings. Earlier versions ran two extra forward passes per coalition and discarded the result.

  • reanalyze() still scores hash embeddings of the archived text, because archives store text only. Embedding-based values of a transformers run therefore differ between explain and reanalyze.

  • The manifest records which embeddings the run scored: embedding_source (none, hash or backend) and embedding_model. Reanalyzing a backend run with an embedding value function logs a WARNING; ExplainRunner.reanalyze(..., strict=True) (mllm-shap reanalyze --strict) raises ReanalyzeMismatchError instead.

  • The HTTP API backend retries status 408, 429, 500, 502, 503 and 504 and transport errors up to backend.max_retries times. The backoff is exponential and honours Retry-After up to backend.timeout_s. Non-JSON responses raise BackendUnavailableError, and at most 1024 responses are cached.

  • The HTTP API backend sends MLLM_SHAP_API_KEY as the bearer key to any host. It sends OPENAI_API_KEY only when that is unset and backend.api_host is api.openai.com; earlier versions sent it to every host, local servers included.

  • The LM Studio backend retries only connection failures and re-raises ValueError and TypeError at once. It matches downloaded models exactly first, then by the prefix of the last path segment, then by substring; an ambiguous id raises with the list of candidates.

  • With auto_download, the LM Studio backend refuses (BackendUnavailableError, listing the hits) when no repository hit is flagged exact or matches the name. Earlier versions downloaded the first hit.

  • explain_sync() called inside a running event loop raises a RuntimeError that points to await runner.explain(...). With an injected backend, repeated explain_sync calls run on one event loop owned by the runner, so loop-bound clients and locks keep working.

  • reanalyze() refuses with ArchiveIncompleteError the archive of a run that failed part-way, even before its first coalition. It also refuses archives without the grand or empty coalition, full-budget archives (budget.fraction: 1.0) missing rows, and archives not written by ExplainRunner.

  • Re-running explain replaces the archive of a run that did not finish without needing on_exists: overwrite.

  • load() opens the database read-only and writes nothing, so archives on read-only storage can be reanalyzed. It raises ArchiveNotFoundError (a FileNotFoundError) when manifest.json or archive.sqlite is missing, instead of creating an empty database.

  • load() raises ArchiveCorruptError when either file cannot be read or the manifest’s fields are invalid, so the command line exits with 2 on a damaged archive.

  • Monte Carlo sampling counts the grand and empty coalitions toward the budget; earlier versions could exceed it by two calls.

  • Progress callbacks report one growing plan and a monotonic finished count across the two Neyman phases.

  • Partial-sample marginal aggregation renormalizes over observed strata with the semivalue’s stratum probabilities, for any estimand.

  • explanation.neyman and explanation.embedding_mode now reach the Neyman estimator and the embedding value functions; earlier versions ignored them. neyman.initial_fraction must lie in (0, 1] and neyman.initial_num_samples be at least 1.

  • run_id hashes the model id of an injected backend, not the one in backend.model_id.

Changed

  • The api backend requires backend.api_host; the OPENAI_BASE_URL environment fallback and the http://127.0.0.1:1234/v1 default are gone, so a configuration without a host fails at load time with ConfigError instead of silently targeting a local server.

  • The coalition scheduler runs jobs in an asyncio.TaskGroup. The first failing job still cancels the others and its exception is raised unwrapped, not as an ExceptionGroup, also when it sits next to a nested group of cancellations.

  • estimator: complementary and neyman_cc with estimand: banzhaf are refused with ConfigError when the config is validated instead of failing after every coalition was generated.

  • The context_id argument of generation_record_from_text() is ignored and raises a DeprecationWarning; it will be removed in 0.2.0. Custom backends call generation_record_from_text(text).

  • Plugin factories receive a PluginContext (the run’s config, the plugin’s options and the registry), and every built-in plugin class takes it as its first constructor argument. resolve() takes the context as its third argument. Third-party plugins must accept context.

  • LM Studio settings hub_search, quantization, auto_download and serialize_generate moved from backend to backend.options and are validated by the LM Studio backend; unknown keys raise ConfigError.

  • Backends take a PluginContext instead of a BackendConfig, and generate(snapshot, generation) receives the whole GenerationConfig.

  • Message holds content parts (TextPart, AudioPart); Message(role=..., text=...) still works and Message.text joins the text parts. Snapshot ids of text-only conversations are unchanged; audio is detected by part type, no longer by an audio: text prefix.

  • Partitioners return a PlayerSet with one PlayerSpan per player. Absence policies and labels read the spans instead of re-tokenising, so custom partitioners work in ExplainRunner, and text outside every span is kept. Rendered messages keep their original whitespace instead of collapsing it to single spaces.

  • GenerationConfig, DedupConfig and KvCacheConfig live in mllm_shap.domain.config (still importable from mllm_shap.pipeline.config); the runtime no longer imports the pipeline.

  • import mllm_shap no longer reads [tool.logging] from the nearest pyproject.toml. Set MLLM_SHAP_LOGGING_CONFIG or call mllm_shap.configure_logging().

  • The orchestrator logs run start, base generation, scheduled coalitions and run duration at INFO.

  • kv_cache.enabled defaults to false; new kv_cache.max_entries.

  • New backend.timeout_s (default 120) and backend.max_retries (default 3); the HTTP client caps connections at explanation.max_inflight.

  • Run archives store generation text in the SQLite row (no blobs/ directory), use WAL journaling and carry user_version 3 and manifest schema version 3. Archives written by 0.1.17 raise ManifestVersionError; re-run explain to rebuild them.

  • Coalition jobs are rendered lazily while the scheduler consumes them.

  • explain_config_from_yaml() raises ConfigError (not pydantic’s ValidationError) for settings that contradict each other, and for an empty document or a YAML root that is not a mapping (earlier a TypeError).

  • The transformers extra requires transformers>=4.56; the build backend is uv_build 0.12.

Added

  • Message(speaker=...) names who wrote a message when people share a role. It is part of the snapshot id and of snapshot JSON ("speaker"), backends send "<speaker>: <text>", and labels and to_dict report it.

  • explanation.players: messages (MessagePartitioner) makes each message one player labelled "<role>[/<speaker>] <n>: <text>"; plugin option roles keeps other roles always present.

  • value_fn: logprob scores the mean log probability of each coalition’s own generated tokens. The transformers and api backends return logprobs (SupportsLogprobs); other backends raise ConfigError. Archives store them (user_version and manifest schema 4), so older archives need a new explain.

  • mllm-shap command line (Command line): explain (JSON result, optional PNG chart), reanalyze (--strict refuses runs scored with backend embeddings), plugins and config --example / --check.

  • The command line uses argparse only. It exits with 2 for invalid input (arguments, --log-level, configuration, snapshot, unreadable input files or archives) and with 3 when the backend is unavailable or the connection fails. Any other exception propagates with its traceback.

  • mllm_shap.ConversationSnapshot.from_json() and mllm_shap.ConversationSnapshot.to_json() read and write snapshots as {"turns": [{"messages": [{"role": ..., "text": ...}]}]}, with an optional "snapshot_id" when the id is not the content-derived one, so the id (and the run_id) survives a round trip. Messages with several content parts are refused, like audio.

  • mllm_shap.ExplainRunOutput.to_dict() returns a JSON-compatible result with a stable key order, including one label per player (text and character span); ExplainRunOutput.player_set holds the explained players as a PlayerSet.

  • configure_logging() accepts level= and json=True to log the pipeline’s stage events to stderr without a TOML file. Events are plain lines or one JSON object per line, and the package logger then stops propagating to the root logger. json=True with a config_path raises ValueError. config/logging.example.toml is a starting point for MLLM_SHAP_LOGGING_CONFIG.

  • explanation.budget.evaluations sets an absolute budget instead of budget.fraction: that many coalition evaluations in total, the empty and grand coalitions included, at any number of players.

  • Each estimator applies its floor to budget.evaluations and the ceiling 2 ** n: 2 for mc, 2n + 2 for complementary and neyman_cc (which also round an odd count down to whole pairs). exact accepts only fraction: 1.0, and setting both keys raises ConfigError.

  • max_players refuses budget.evaluations above 2 ** max_players. Configurations without evaluations serialise as before, so their run ids do not change.

  • explanation.plugin_options: per-plugin options keyed by <group>/<name> (for example absence_policies/pad: {placeholder: "<unk>"}) or by the bare plugin name when only one plugin of the run has it. Built-in plugins declare their options in a PluginOptions model; unknown keys, wrong types and keys that name no plugin of the run raise ConfigError before the run starts, naming the group, plugin and key.

  • Error hierarchy rooted at MllmShapError: PluginNotFoundError, PluginConflictError, PluginLoadError, ConfigError, ArchiveNotFoundError and ArchiveCorruptError join the existing errors.

  • Config cross-validation: estimator: exact requires budget.fraction: 1.0 and neyman_cc rejects include_minimal_masks: true.

  • Capability protocols SupportsKvCache, SupportsAclose, FittableValueFunction and RequiresEmbeddings; any value function with fit is fitted on the run corpus, not only TF-IDF. GenerationConfig.compute_embeddings asks a backend for model embeddings.

  • default_registry() registers all eight plugin groups (estimands and renderers included) and loads third-party entry points; registering a name twice raises PluginConflictError unless replace=True. registry can be passed to render_attribution() to reuse custom renderers.

  • Top-level exports for every built-in plugin class, the absence policies, TokenPartitioner, all errors, the observability and progress protocols and display_attribution_html.

  • ExplainRunner(backend=...) reuses one backend across explain calls; close(), aclose() and with / async with blocks close it.

  • New explanation.max_players (default 24): a run whose budget plans more than 2 ** max_players coalitions (budget.fraction * 2 ** n for n players, whatever the estimator) raises ConfigError.

Removed

  • generation.precompute_base (the base generation always runs; remove the key from YAML configs), CoalitionJob.utility, CoalitionRecord.utility, SystemRolesSetup, CooperativeGame.empty_value and masking.prefix_tree.PrefixTreeStub.

Documentation

  • New “How a run works” page (stages, scheduler counters, archives, reanalyze, progress and telemetry, explain vs explain_sync), “Choosing an estimator”, and “Coming from MLLM-Shap-legacy” guides; citing and contributing pages.

  • Configuration reference and the public-exports table are generated from the installed package; every runnable guide snippet is executed on each docs build from the page source.

  • Theory pages state the efficiency convention sum(values) = grand_value - base_value (exact on the full coalition table only), the stratified partial-sample estimand, and corrected DOIs (Štrumbelj & Kononenko 2014, Young 1985).

  • make docs builds with -W --keep-going -n; make -C docs linkcheck checks external links.

Tooling

  • Upgrades and releases: make upgrade-check and make upgrade (scripts/upgrade.py) list and apply new package versions, prek hook revisions, workflow action pins, the uv version and Obsidian plugins, except the pins in publish.yaml. scripts/pr_policy.py --bump derives the next version from the commits since the last tag.

  • Benchmarks use pytest-benchmark: make bench writes a JSON report and make bench-regression fails when a median is more than 25% slower than the committed tests/benchmarks/baseline.json. New benchmarks cover exact, stratified and complementary aggregation, the scheduler, TF-IDF scoring and an end-to-end mock run.

  • make mutate runs mutmut on the estimation package.

  • Tests that build a wheel or reach the network are marked slow and run in make tests-all only.

  • Practical tests: make practical FEATURE=<id|glob|all> runs about 30 scenarios (marker practical) on Qwen/Qwen2.5-1.5B-Instruct and SmolLM2-360M-Instruct on a GPU, with rubric checks and a logit judge; make check excludes them.

  • Deep practical tier: make practical TIER=deep runs nine deep.* scenarios on Qwen/Qwen2.5-3B-Instruct under an 8 GB VRAM cap. They cover error versus budget against a 10-player exact reference, estimator stability, Banzhaf, a counterfactual pair per value function, a 10-turn three-speaker thread, the KV cache, archive reanalysis and CLI parity.

0.1.17 (2026-10-07)

Fixed

  • Exact, Monte Carlo, complementary and Neyman attributions now satisfy efficiency on the full coalition table (budget.fraction: 1.0); approximate budgets converge to it as coverage grows. Earlier versions omitted the grand coalition (exact, Monte Carlo) and the \((N, \emptyset)\) pair (complementary, Neyman), which under-reported every player by its \(S = N\setminus\{i\}\) marginal. Re-run explanations produced by 0.1.x.

  • Coalition payoffs stay aligned with their masks when kv_cache.enabled reorders jobs for prefix reuse; earlier versions could score a coalition with another coalition’s generation.

  • Monte Carlo partial samples use a stratified mean of observed marginals instead of an un-normalised weighted sum.

  • Complementary and Neyman aggregation pair masks by complement, independent of evaluation order.

  • Neyman phase one no longer stalls on cells that cannot be filled, and phase two now runs inside ExplainRunner and spends the remaining budget. The allocation uses the complement column \(n - j\).

  • InferenceScheduler surfaces the first coalition failure and cancels the remaining jobs.

Changed

  • run_id includes a digest of the configuration and model id. Re-running into an existing archive raises ArchiveExistsError unless explanation.archive.on_exists is overwrite.

  • Manifest schema version 2 records the estimator, value function, normalizer, absence policy, partition, backend, model, player ids and the full config. Archives also carry SQLite user_version 2; older archives raise ManifestVersionError on load.

  • reanalyze() re-aggregates with the archived estimator, refuses a mismatched estimator, absence policy or partition with ReanalyzeMismatchError, and reproduces explain values for the same value function.

  • Value functions score records built from the generated text in both explain and reanalyze (generation_record_from_text()).

  • hash_mask() is stable across processes.

  • Complementary and Neyman estimators raise ValueError for the banzhaf estimand; earlier versions returned complementary Shapley values under a Banzhaf label.

Added

Removed

  • InferenceScheduler.run_stream (use run_iter()) and the mllm_shap.backends.mock.generation module.

Earlier releases

  • 0.1.16 (2026-07-19): coalition progress callbacks on the scheduler and runner; BackendConfig.serialize_generate for LM Studio; DeletePolicy renders the empty coalition; a Neyman infinite loop at small player counts is fixed.

  • 0.1.15 (2026-07-15): runtime micro-benchmarks and the make bench and make bench-regression targets.

  • 0.1.14 (2026-07-15): Extending mllm-shap guide; renderers load lazily, so import mllm_shap works without [viz].

  • 0.1.13 (2026-07-15): mllm_shap.viz with the token_text and token_bar renderers and the [viz] extra (on PyPI; no v0.1.13 git tag).

  • 0.1.12 (2026-07-15): ApiBackend for OpenAI-compatible HTTP APIs and the [api] extra.

  • 0.1.11 (2026-07-15): TransformersTextBackend, prefix KV caching and the [transformers] extra.

  • 0.1.10 (2026-07-15): LmStudioBackend and the [lmstudio] extra.

  • 0.1.9 (2026-07-15): reanalyze(), stage telemetry and PerfSummary.

  • 0.1.8 (2026-07-14): ExplainRunner, the orchestrator and the mock backend.

  • 0.1.7 (2026-07-14): TF-IDF, embedding and logprob value functions, presentation normalizers and GenerationRecord.

  • 0.1.6 (2026-07-14): the mc, complementary and neyman_cc estimators.

  • 0.1.5 (2026-07-13): ExactEstimator and run_iter().

0.1.4 (2026-07-13)

Added

Documentation

  • Theory and usage guides for the runtime archive, dedup, and scheduler.

  • API reference page for mllm_shap.runtime.

  • Example notebook examples/runtime_core.ipynb.

0.1.3 (2026-07-12)

Added

Documentation

  • Theory and usage guides for coalition masking and absence policies.

  • API reference page for mllm_shap.masking.

  • Example notebook examples/masking_views.ipynb.

0.1.2 (2026-07-12)

Added

Documentation

  • Sphinx theory pages for cooperative games and estimands (Shapley vs Banzhaf).

  • Usage guide for estimand aggregators, results, and manifests.

  • Expanded API reference for public modules.

  • Example notebook examples/estimands_toy_game.ipynb.

  • Furo theme, getting-started page, embedded notebook rendering, and API module layout.

Tooling

  • make notebooks target to execute example notebooks in place before commit.

0.1.1 (2026-07-12)

Added

0.1.0 (2026-07-12)

Added

  • Initial PyPI package layout and mllm_shap import surface.

  • Logging bootstrap from [tool.logging] in pyproject.toml via logging518.