Alignment

Spectrogram-guided alignment of transcript tokens to speech (audio extra).

@domain alignment @see [[knowledge/architecture/alignment]]

Spectrogram-guided alignment of transcript tokens to speech (SGPA).

A CTC model aligns the transcript’s characters to frames; each token boundary then moves to the quietest, steadiest point of the waveform near it (RMS energy and spectral flux), so a segment cut does not split a sound.

@domain alignment @see [[knowledge/architecture/alignment]] @tested-by [[tests/alignment/test_sgpa.py]]

mllm_shap.alignment.sgpa.DEFAULT_MODEL: str = 'facebook/wav2vec2-base-960h'

English character-level CTC model used when no model is given.

mllm_shap.alignment.sgpa.MIN_DURATION_S: float = 0.05

Shortest segment the aligner emits, in seconds.

class mllm_shap.alignment.sgpa.CtcModel(*args, **kwargs)

Bases: Protocol

Character-level CTC acoustic model.

sample_rate: int

Sample rate the model expects.

blank_id: int

Id of the CTC blank label.

vocab: Mapping[str, int]

Label id of every character the model emits.

log_probs(waveform: NDArray[float32]) → NDArray[floating]

Return [frames, labels] log-probabilities for a mono waveform.

class mllm_shap.alignment.sgpa.Wav2Vec2Ctc(model_id: str = 'facebook/wav2vec2-base-960h', revision: str | None = None, device: str = 'cpu')

Bases: object

A Hugging Face Wav2Vec2ForCTC checkpoint (transformers extra).

vocab: Mapping[str, int]

blank_id: int

sample_rate: int

log_probs(waveform: NDArray[float32]) → NDArray[floating]

Run the model once over waveform.

class mllm_shap.alignment.sgpa.SpectrogramGuidedAligner(model: CtcModel | None = None, energy_weight: float = 0.8, flux_weight: float = 0.2, separator: str = '|')

Bases: object

Align transcript tokens to speech and refine their boundaries.

align(transcript: str | Sequence[str], audio: AudioPart, attach_audio: bool = False) → list[AudioSegment]

Return one segment per token of transcript the model can spell.

A string is split on whitespace. Tokens without a character in the model’s vocabulary (punctuation, for example) have no segment.

Raises:

ValueError – If no token has a character the model knows, or the audio is too short for the transcript.

The sgpa_segments partition: one player per spoken word of an audio message.

@domain alignment @see [[knowledge/architecture/alignment]] @tested-by [[tests/alignment/test_sgpa_players.py]]

class mllm_shap.alignment.partitions.SgpaOptions(*, model_id: str = 'facebook/wav2vec2-base-960h', revision: str | None = None, device: str = 'cpu', energy_weight: Annotated[float, ~annotated_types.Ge(ge=0)] = 0.8, flux_weight: Annotated[float, ~annotated_types.Ge(ge=0)] = 0.2, text_roles: list[~typing.Literal['user', 'assistant', 'system']]=<factory>, transcript: Literal['send', 'align_only']='send')

Bases: PluginOptions

Options of the sgpa_segments partition.

model_id: str

Hugging Face id of the Wav2Vec2 CTC model.

revision: str | None

Hub commit to load; pin it for reproducible players.

device: str

Torch device the model runs on.

energy_weight: float

Weight of RMS energy in the boundary cost.

flux_weight: float

Weight of spectral flux in the boundary cost.

text_roles: list[Literal['user', 'assistant', 'system']]

Roles whose text-only messages also give one player per word, so typed and spoken words are explained in one run; none by default.

transcript: Literal['send', 'align_only']

send gives the model each audio message’s transcript with its audio; align_only uses the transcript to build the players and sends audio only, so values reflect what the model hears, not what it reads.

model_config = {'extra': 'forbid', 'frozen': True, 'strict': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mllm_shap.alignment.partitions.SgpaSegmentPartitioner(context: PluginContext | None = None, aligner: SpectrogramGuidedAligner | None = None)

Bases: object

Make each spellable word of every audio message’s transcript a player.

A message qualifies when it has an AudioPart and text: the text is the transcript the audio is aligned to. Each player’s span covers the word in the text and, through samples, its stretch of audio, so absence policies mask both. Text-only messages give word players when their role is in text_roles and no players otherwise.

options_model

Options accepted from explanation.plugin_options.

alias of SgpaOptions

property name: str

Return the registered partition identifier.

partition(snapshot: ConversationSnapshot) → PlayerSet

Align every audio message and return its words as players.

Raises:

ValueError – If no message has audio, or an audio message has no transcript text.

Transcript tokens aligned to audio time ranges.

@domain alignment @see [[knowledge/architecture/alignment]] @tested-by [[tests/alignment/test_segments.py]]

class mllm_shap.alignment.segments.AudioSegment(token: str, token_index: int, start_time: float, end_time: float, ctc_start_time: float, ctc_end_time: float, confidence: float, boundary_refined: bool, sample_rate: int, start_sample: int, end_sample: int, audio: bytes = b'')

Bases: object

One transcript token and the stretch of audio that speaks it.

token: str

Transcript token as given to the aligner.

token_index: int

Position of token in the transcript’s token list.

start_time: float

Start in seconds.

end_time: float

End in seconds; greater than start_time.

ctc_start_time: float

Start of the token’s first character frame, before boundary refinement.

ctc_end_time: float

End of the token’s last character frame, before boundary refinement.

confidence: float

Mean model probability of the token’s characters, in [0, 1].

boundary_refined: bool

Whether both edges moved to a low-energy point of the waveform.

sample_rate: int

Sample rate of the source waveform.

start_sample: int

First sample of the segment in the source waveform.

end_sample: int

Sample after the last one; greater than start_sample.

audio: bytes

The segment as WAV bytes when the aligner attached it, else empty.

property duration: float

Length in seconds.

CTC forced alignment (Viterbi over the blank-extended target sequence).

@domain alignment @see [[knowledge/architecture/alignment]] @tested-by [[tests/alignment/test_ctc.py]]

mllm_shap.alignment.ctc.BLANK_STATE: int = -1

Path value of a frame aligned to the blank label.

class mllm_shap.alignment.ctc.TargetSpan(start: int, end: int, confidence: float)

Bases: object

Frames [start, end) that one target label occupies.

start: int

First frame of the label.

end: int

Frame after the last one.

confidence: float

Mean probability of the label over its frames.

mllm_shap.alignment.ctc.forced_align(log_probs: NDArray[floating], targets: Sequence[int], blank: int = 0) → NDArray[int64]

Return, per frame, the index into targets it aligns to, or BLANK_STATE.

The most probable monotonic path over targets with optional blanks between labels and a mandatory blank between two equal neighbours (standard CTC).

Raises:

ValueError – If log_probs has too few frames for targets.

mllm_shap.alignment.ctc.target_spans(path: NDArray[int64], log_probs: NDArray[floating], targets: Sequence[int]) → list[TargetSpan]

Return the frame span and mean probability of every target in path.

Encoded audio bytes to mono float waveforms and back.

@domain alignment @see [[knowledge/architecture/alignment]] @tested-by [[tests/alignment/test_io.py]]

mllm_shap.alignment.io.require(module: str) → ModuleType

Import an optional dependency of the audio or transformers extra.

Raises:

ImportError – Naming the extra to install.

mllm_shap.alignment.io.decode_audio(data: bytes, sample_rate: int | None = None) → tuple[NDArray[float32], int]

Decode data to a mono float32 waveform, resampled to sample_rate.

Channels are averaged. Without sample_rate the file’s own rate is kept.

Raises:

ValueError – If data is not a format soundfile can read.

mllm_shap.alignment.io.encode_wav(waveform: NDArray[floating], sample_rate: int) → bytes

Encode a mono waveform as 16-bit PCM WAV; NaN becomes 0, values clip to ±1.