Benchmarks¶

Cross-client benchmark meta-analysis (v1.5): pooling fitted MMMs and archived platform lift studies from many client projects into industry-vertical benchmarks, without any one client’s results being recoverable by another client. That last clause is why this is nine modules and not a groupby. K-anonymity alone is not privacy — given an exact cell mean, n-1 colluding contributors recover the held-out brand’s number by arithmetic — so every value that crosses the export boundary is a clipped simple statistic released under Laplace noise, charged to a per-contributor ε ledger whose annual cap is a hard gate rather than a report. The meta-model posterior, whose sensitivity to any single contributor has no clean bound, stays internal and is never released directly. The layering enforces this at one boundary: contribute and ingest run project-side, store, meta_model and publish run in the central benchmark tenant, and only publish’s ledgered releases reach another client. Design: technical-docs/benchmark-meta-analysis-design.md; worked walkthrough: nbs/demos/benchmark_framework.ipynb.

Cross-client, privacy-preserving benchmark meta-analysis.

Pools findings across client projects — fitted MMMs and platform lift studies — into industry-vertical benchmarks, without any client’s results being recoverable by another client. Design: technical-docs/benchmark-meta-analysis-design.md and the whitepaper technical-docs/benchmark-meta-analysis-whitepaper.docx; the working walkthrough is nbs/demos/benchmark_framework.ipynb.

Layered so privacy is enforced at a single boundary: contribute and ingest run project-side and emit summary records; store, meta_model and publish run in the central benchmark tenant; only publish’s ledgered DP releases ever reach another client.

Imports are lazy (PEP 562): the meta-model pulls in PyMC only when fitted.

class mmm_framework.benchmarks.BenchmarkEstimand(name, description, scale='log', requires_finance_basis=False, performance=True, provenance=('model', 'experiment', 'calibrated_model'), notes='', reference_point=<factory>)[source]

Bases: object

One entry in the benchmark estimand catalog.

scale is the pooling scale. Ratio quantities pool on the log scale: effects are multiplicative across brands, and a symmetric-normal meta-model on the raw scale gets dragged by the right tail.

name: str

description: str

scale: str = 'log'

requires_finance_basis: bool = False

performance: bool = True

provenance: tuple[str, ...] = ('model', 'experiment', 'calibrated_model')

notes: str = ''

reference_point: dict[str, float]

__init__(name, description, scale='log', requires_finance_basis=False, performance=True, provenance=('model', 'experiment', 'calibrated_model'), notes='', reference_point=<factory>)

class mmm_framework.benchmarks.BenchmarkStore(path=':memory:')[source]

Bases: object

SQLite-backed contribution store with an audit trail.

__init__(path=':memory:')[source]

close()[source]

Return type:: None

add_contribution(record, *, now_ts=None)[source]

Return type:: None

add_contributions(records, *, now_ts=None)[source]

Return type:: None

contributions(*, vertical=None, channel_code=None, estimand=None, contributor_hash=None)[source]

Return type:: list[ContributionRecord]

cells()[source]

All contributions grouped by benchmark cell.

Return type:: dict[CellKey, list[ContributionRecord]]

delete_contributor(contributor_hash, *, now_ts=None)[source]

Consent revocation: remove every record from one contributor.

Already-published vintages are immutable (published noise cannot be un-released); deletion takes effect from the next vintage. The audit entry is the record consent requests are answered from.

Return type:: int

record_release(release, *, now_ts=None)[source]

Return type:: None

ledger()[source]

Return type:: EpsilonLedger

save_vintage(vintage, *, now_ts=None)[source]

Return type:: None

get_vintage(vintage)[source]

Return type:: PublishedVintage | None

latest_vintage()[source]

Return type:: PublishedVintage | None

list_vintages()[source]

Return type:: list[str]

audit_log(*, action=None, contributor_hash=None)[source]

Query the audit trail (consent/deletion requests are answered here).

Return type:: list[dict[str, Any]]

class mmm_framework.benchmarks.CellKey(**data)[source]

Bases: BaseModel

The unit of gating and publication.

model_config: ClassVar[ConfigDict] = {'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

vertical: str

channel_code: str

estimand: str

schema_version: str

label()[source]

Return type:: str

class mmm_framework.benchmarks.ContributionRecord(**data)[source]

Bases: BaseModel

One (contributor × channel × estimand) summary crossing the boundary.

Summaries, not draws: mean/sd/quantiles on the pooled scale cap the information a contribution carries. sd doubles as the known-SE the meta-model consumes as data, so the ingest SE floor must already be applied.

schema_version: str

taxonomy_version: str

estimand: str

channel_code: str

vertical: str

scale: Literal['log', 'linear']

mean: float

sd: float

quantiles: dict[float, float]

provenance: Provenance

covariates: dict[str, Any]

contributor_hash: str

consent_version: str

source_id: str | None

cell()[source]

Return type:: CellKey

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

exception mmm_framework.benchmarks.EligibilityError[source]

Bases: ValueError

A gate refused the contribution; the message says which and why.

class mmm_framework.benchmarks.EpsilonLedger(releases=None)[source]

Bases: object

Append-only record of every number that crossed the boundary.

Composition is basic (sum of ε per member across releases). Derived and carried-forward releases enter at ε = 0 — post-processing is free.

__init__(releases=None)[source]

append(release)[source]

Return type:: None

spent_by_member(vintage_prefix=None)[source]

Per-contributor ε spend, optionally within one annual vintage prefix (e.g. "2026" sums 2026Q1..Q4).

Return type:: dict[str, float]

worst_case(vintage_prefix=None)[source]

Return type:: float

would_exceed(members, epsilon, cap, vintage_prefix=None)[source]

Hard annual gate: would charging epsilon to members push any of them past cap? The caller must then freeze (carry forward), not release.

Return type:: bool

class mmm_framework.benchmarks.IngestResult(records=<factory>, rejects=<factory>)[source]

Bases: object

Accepted contribution records plus the per-study reject audit.

records: list[ContributionRecord]

rejects: list[RejectedStudy]

reject_counts()[source]

Return type:: dict[str, int]

__init__(records=<factory>, rejects=<factory>)

class mmm_framework.benchmarks.LiftStudyRecord(**data)[source]

Bases: BaseModel

One archived platform lift study, as an archive actually holds it.

Optional fields are optional because real archives omit them; the ingest gates in ingest decide whether an omission is fatal (it usually is). contributor_hash is the salted, tenant-stable brand identifier — the raw brand/tenant id never appears in this schema.

study_id: str

contributor_hash: str

vertical: str

channel_code: str

form: StudyForm

audience: str | None

quarter: str | None

point: float | None

ci_low: float | None

ci_high: float | None

ci_level: float | None

incremental_conversions: float | None

se_conversions: float | None

lift_pct: float | None

se_lift_pct: float | None

baseline_conversions: float | None

baseline_rel_se: float | None

value_basis: str | None

value_per_conversion: float | None

value_rel_se: float | None

spend_in_study: float | None

impressions_m: float | None

attribution_window: str | None

preregistered: bool | None

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.MetaModelResult(idata, verticals, contributors, contributor_vertical, moderators, reference=<factory>, has_provenance_bias=False)[source]

Bases: object

Fitted meta-model plus the indexing needed to read it.

idata: Any

verticals: list[str]

contributors: list[str]

contributor_vertical: list[str]

moderators: list[str]

reference: dict[str, float]

has_provenance_bias: bool = False

max_rhat()[source]

Return type:: float

population(vertical, prob=0.95)[source]

Population mean and heterogeneity for one vertical, with the two intervals every consumer must distinguish: the CI for the mean and the prediction interval for a new brand (tau included).

Return type:: dict[str, float]

moderator_summary()[source]

Return type:: DataFrame

shrinkage_table()[source]

Per-contributor pooled posterior vs vertical mean (internal view).

Return type:: DataFrame

__init__(idata, verticals, contributors, contributor_vertical, moderators, reference=<factory>, has_provenance_bias=False)

class mmm_framework.benchmarks.ModeratorSpec(name, covariate, kind='binary', value=None, reference=None)[source]

Bases: object

One meta-regression moderator built from a covariate.

kind="binary" codes covariates[covariate] == value as 1; kind="continuous" codes covariates[covariate] - reference (with reference defaulting to the pool mean), so the vertical means stay interpretable at the reference operating point.

name: str

covariate: str

kind: str = 'binary'

value: Any = None

reference: float | None = None

__init__(name, covariate, kind='binary', value=None, reference=None)

class mmm_framework.benchmarks.PrivacyPolicy(**data)[source]

Bases: BaseModel

Every privacy knob in one place, so the whole posture is reviewable.

k_min: int

spend_dominance_max: float

precision_dominance_max: float

epsilon_per_release: float

epsilon_year: float

clip_lo: float

clip_hi: float

deviation_clip: float

contrast_clip: float

se_floor: float

min_refresh_churn: int

property clip_width: float

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.Provenance(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: str, Enum

How a contributed number was produced.

MODEL = 'model'

EXPERIMENT = 'experiment'

CALIBRATED_MODEL = 'calibrated_model'

class mmm_framework.benchmarks.PublishedCell(**data)[source]

Bases: BaseModel

A client-visible benchmark cell, assembled only from ledgered releases.

vertical: str

channel_code: str

estimand: str

n_band: str

center: float

spread: float

noise_scale: float

epsilon_spent: float

derived: bool

offsets: dict[str, float]

stale_since: str | None

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.PublishedVintage(**data)[source]

Bases: BaseModel

The full published artifact for one vintage (client-safe).

vintage: str

cells: list[PublishedCell]

suppressed: list[SuppressedCell]

epsilon_per_release: float

worst_case_annual_epsilon: float

frozen: list[str]

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.ReleaseRecord(**data)[source]

Bases: BaseModel

One entry in the ε ledger: a number that crossed the boundary.

epsilon == 0 marks a post-processed (derived) or carried-forward release — free under DP, recorded for lineage.

release_id: str

vintage: str

name: str

statistic: str

value: float

epsilon: float

laplace_scale: float

members: tuple[str, ...]

vertical: str | None

estimand: str | None

model_config: ClassVar[ConfigDict] = {}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.StudyForm(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: str, Enum

The estimand form a lift study was archived as (whitepaper §2.5).

IROAS = 'iroas'

ABSOLUTE_LIFT = 'absolute_lift'

RELATIVE_LIFT = 'relative_lift'

ATTRIBUTED_ROAS = 'attributed_roas'

BRAND_LIFT = 'brand_lift'

class mmm_framework.benchmarks.VerticalPriorPreset(vertical, channel_code, estimand, mu, sd, scale, experimental_share, anchored, source_vintage)[source]

Bases: object

A default-prior preset for one (vertical × channel × estimand) cell.

vertical: str

channel_code: str

estimand: str

mu: float

sd: float

scale: str

experimental_share: float

anchored: bool

source_vintage: str

__init__(vertical, channel_code, estimand, mu, sd, scale, experimental_share, anchored, source_vintage)

mmm_framework.benchmarks.all_estimands()[source]

Return type:: list[BenchmarkEstimand]

mmm_framework.benchmarks.budget_band(annual_media_spend)[source]

Coarsen an exact annual budget to its order-of-magnitude band.

Return type:: str

mmm_framework.benchmarks.build_contribution(*, estimand, channel_code, vertical, draws, contributor, provenance=Provenance.MODEL, covariates=None, consent=False, consent_version='consent-v1', source_id=None, policy=None)[source]

Build one contribution record from posterior draws on the pooled scale.

draws must already be on the catalog scale for the estimand (log for ratio quantities). Catalog rules are enforced here: a finance-basis estimand without a declared value_basis covariate is rejected, not defaulted.

Return type:: ContributionRecord

mmm_framework.benchmarks.check_model_eligibility(results, *, channel, min_weeks=52, min_contraction=0.1, contraction=None)[source]

Raise EligibilityError unless a fitted model may contribute.

results is an MMMResults-like object (approximate, converged, panel). contraction is the channel’s prior→posterior contraction from mmm_framework.diagnostics.learning.parameter_learning(); pass it when a prior-predictive group is available — None skips that gate (the caller accepts the echo-chamber risk knowingly, not by default).

Return type:: None

mmm_framework.benchmarks.contributions_from_model(results, *, vertical, contributor, channel_draws, covariates=None, consent=False, contraction_by_channel=None, min_weeks=52, policy=None, source_id=None)[source]

Extract every eligible contribution from one fitted model.

channel_draws maps channel_code -> {estimand_name: draws} with draws already on the catalog scale (e.g. log-elasticity draws from an analyzer or estimand evaluation). Gates run per channel; one ineligible channel does not block the others, but a model-level gate (approximate fit, failed convergence) blocks everything.

Return type:: list[ContributionRecord]

mmm_framework.benchmarks.contributor_hash(tenant_id, salt)[source]

Salted, tenant-stable pseudonym. The raw tenant id never leaves.

Return type:: str

mmm_framework.benchmarks.experimental_share(records)[source]

Precision-weighted share of non-model evidence in a cell.

Return type:: float

mmm_framework.benchmarks.fit_meta_model(records, *, moderators=(ModeratorSpec(name='retargeting', covariate='audience', kind='binary', value='retargeting', reference=None), ModeratorSpec(name='log_spend', covariate='log_spend', kind='continuous', value=None, reference=None)), include_provenance_bias=True, draws=1000, tune=1000, chains=4, random_seed=11, progressbar=False)[source]

Fit the hierarchical random-effects pool over one estimand’s records.

All records must share a scale (enforced); mixed-estimand input is allowed only when the caller has a reason to pool them (normally: don’t). delta_m is added when both model-family and experimental provenances are present.

Return type:: MetaModelResult

mmm_framework.benchmarks.gate_cell(cell, records, policy)[source]

Apply the three per-cell gates: k, spend dominance, precision dominance.

Spend dominance is skipped (None) when no record carries a spend covariate; precision dominance always applies because the meta-model’s pooled mean is precision-weighted.

Return type:: CellGateAudit

mmm_framework.benchmarks.ingest_studies(studies, policy=None)[source]

Run the ingest gates over an archive; nothing is silently dropped.

Return type:: IngestResult

mmm_framework.benchmarks.laplace_release(value, sensitivity, epsilon, rng)[source]

Laplace mechanism for a bounded statistic. Clip before calling — without hard bounds the sensitivity is unbounded and no finite noise gives a guarantee.

Return type:: LaplaceRelease

mmm_framework.benchmarks.n_band(n)[source]

Band a contributor count for publication (exact counts difference).

Return type:: str

mmm_framework.benchmarks.normalize_study(study, policy)[source]

Route one archived study: a contribution record, or a reject with reason.

Return type:: ContributionRecord | RejectedStudy

mmm_framework.benchmarks.placement(client_value, cell, *, policy=None, n_replicates=400, rng=None)[source]

Percentile placement against a published cell — as a range, honestly.

At small cells and tight budgets the release noise exceeds the between-brand spread, so a point percentile would be noise theater. The placement is reported as the range across replicate releases (an analysis over hypothetical noise draws — nothing extra is published) plus a usability verdict with the cell size a stable placement would need.

Return type:: dict

mmm_framework.benchmarks.publish_vintage(store, *, vintage, policy=None, rng=None, contrasts=(ContrastSpec(name='retargeting_offset', covariate='audience', level_hi='retargeting', level_lo='prospecting'),), now_ts=None)[source]

Publish one vintage from the store’s current contributions.

vintage sorts lexicographically (“2026Q3”) and its leading 4 characters are the annual accounting prefix for the ε_year gate.

Return type:: PublishedVintage

mmm_framework.benchmarks.vertical_prior_preset(cell, *, vintage, experimental_share, floor_sd=0.5, anchor_threshold=0.3, scale='log')[source]

Map one published cell into a default-prior preset.

The predictive width is sqrt(spread² + noise²) — a new client is a draw from the population, not a re-measurement of its mean. Unanchored cells are floored at floor_sd regardless of how tight the pool looks.

Return type:: VerticalPriorPreset

Privacy¶

Privacy machinery: cell gates, DP releases, and the ε ledger.

Doctrine (whitepaper §4–§5, demonstrated in the companion notebook):

k-anonymity alone is not privacy — n-1 colluding contributors invert an exact cell mean to machine precision. The Laplace noise is the mechanism that makes the goal statement (“no client’s results recoverable by another client”) true; the gates only decide which cells are worth releasing.
DP applies to clipped simple statistics (per-contributor means, MADs, contrasts), never to posterior functionals — the sensitivity of a partially-pooled, precision-weighted posterior mean has no clean bound.
Every boundary crossing lands in one ledger. A contributor’s privacy loss is the sum of ε over every release it appears in; the annual cap is a hard gate, not a report.

mmm_framework.benchmarks.privacy.GENERIC_SUPPRESSION = 'insufficient contributors'¶: Generic client-facing suppression reason. Anything more specific (“dominance 67%”) would itself disclose cell structure.

class mmm_framework.benchmarks.privacy.PrivacyPolicy(**data)[source]¶

Bases: BaseModel

Every privacy knob in one place, so the whole posture is reviewable.

k_min: int¶

spend_dominance_max: float¶

precision_dominance_max: float¶

epsilon_per_release: float¶

epsilon_year: float¶

clip_lo: float¶

clip_hi: float¶

deviation_clip: float¶

contrast_clip: float¶

se_floor: float¶

min_refresh_churn: int¶

property clip_width: float¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

mmm_framework.benchmarks.privacy.n_band(n)[source]¶

Band a contributor count for publication (exact counts difference).

Return type:: str

mmm_framework.benchmarks.privacy.per_contributor_means(records, policy)[source]¶

Clipped unweighted mean per contributor — the DP release primitive.

Unweighted within contributor and clipped to the policy range, so the sensitivity of the cell mean is exactly clip_width / n.

Return type:: dict[str, float]

mmm_framework.benchmarks.privacy.gate_cell(cell, records, policy)[source]¶

Apply the three per-cell gates: k, spend dominance, precision dominance.

Spend dominance is skipped (None) when no record carries a spend covariate; precision dominance always applies because the meta-model’s pooled mean is precision-weighted.

Return type:: CellGateAudit

class mmm_framework.benchmarks.privacy.LaplaceRelease(value, scale)[source]¶

Bases: object

A noised statistic plus the scale it was noised at.

value: float¶

scale: float¶

__init__(value, scale)¶

mmm_framework.benchmarks.privacy.laplace_release(value, sensitivity, epsilon, rng)[source]¶

Laplace mechanism for a bounded statistic. Clip before calling — without hard bounds the sensitivity is unbounded and no finite noise gives a guarantee.

Return type:: LaplaceRelease

class mmm_framework.benchmarks.privacy.EpsilonLedger(releases=None)[source]¶

Bases: object

Append-only record of every number that crossed the boundary.

Composition is basic (sum of ε per member across releases). Derived and carried-forward releases enter at ε = 0 — post-processing is free.

__init__(releases=None)[source]¶

append(release)[source]¶

Return type:: None

spent_by_member(vintage_prefix=None)[source]¶

Per-contributor ε spend, optionally within one annual vintage prefix (e.g. "2026" sums 2026Q1..Q4).

Return type:: dict[str, float]

worst_case(vintage_prefix=None)[source]¶

Return type:: float

would_exceed(members, epsilon, cap, vintage_prefix=None)[source]¶

Hard annual gate: would charging epsilon to members push any of them past cap? The caller must then freeze (carry forward), not release.

Return type:: bool

Schema¶

Schemas for the cross-client benchmark system.

Two record families cross this module:

LiftStudyRecord — the archive-side schema for a platform lift study (whitepaper §2.5). This is what an ingest job receives; most archives violate it somewhere, which is the point of the ingest gates.
ContributionRecord — the only thing allowed to cross the privacy boundary from a project into the central benchmark store. Summaries on the pooled (log) scale, never draws, never raw data.

Everything downstream (store, gates, meta-model, publication) speaks these types. A bad record fails loudly at construction, the same doctrine as the garden contract.

mmm_framework.benchmarks.schema.SCHEMA_VERSION = 'bench-contrib-v1'¶: Version stamped on every contribution record. Bump on any change to the fields or their semantics; the store partitions cells by it, so records made under different schema versions never pool together silently.

mmm_framework.benchmarks.schema.DEFAULT_TAXONOMY_VERSION = 'chan-tax-v1'¶: Version of the channel taxonomy contributions are coded against. A re-mapping of what “digital video” means requires a bump plus a back-mapping table — records under different taxonomies never pool.

class mmm_framework.benchmarks.schema.Provenance(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶

Bases: str, Enum

How a contributed number was produced.

MODEL = 'model'¶

EXPERIMENT = 'experiment'¶

CALIBRATED_MODEL = 'calibrated_model'¶

class mmm_framework.benchmarks.schema.StudyForm(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)[source]¶

Bases: str, Enum

The estimand form a lift study was archived as (whitepaper §2.5).

IROAS = 'iroas'¶

ABSOLUTE_LIFT = 'absolute_lift'¶

RELATIVE_LIFT = 'relative_lift'¶

ATTRIBUTED_ROAS = 'attributed_roas'¶

BRAND_LIFT = 'brand_lift'¶

class mmm_framework.benchmarks.schema.LiftStudyRecord(**data)[source]¶

Bases: BaseModel

One archived platform lift study, as an archive actually holds it.

study_id: str¶

contributor_hash: str¶

vertical: str¶

channel_code: str¶

form: StudyForm¶

audience: str | None¶

quarter: str | None¶

point: float | None¶

ci_low: float | None¶

ci_high: float | None¶

ci_level: float | None¶

incremental_conversions: float | None¶

se_conversions: float | None¶

lift_pct: float | None¶

se_lift_pct: float | None¶

baseline_conversions: float | None¶

baseline_rel_se: float | None¶

value_basis: str | None¶

value_per_conversion: float | None¶

value_rel_se: float | None¶

spend_in_study: float | None¶

impressions_m: float | None¶

attribution_window: str | None¶

preregistered: bool | None¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.ContributionRecord(**data)[source]¶

Bases: BaseModel

One (contributor × channel × estimand) summary crossing the boundary.

schema_version: str¶

taxonomy_version: str¶

estimand: str¶

channel_code: str¶

vertical: str¶

scale: Literal['log', 'linear']¶

mean: float¶

sd: float¶

quantiles: dict[float, float]¶

provenance: Provenance¶

covariates: dict[str, Any]¶

contributor_hash: str¶

consent_version: str¶

source_id: str | None¶

cell()[source]¶

Return type:: CellKey

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.CellKey(**data)[source]¶

Bases: BaseModel

The unit of gating and publication.

model_config: ClassVar[ConfigDict] = {'frozen': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

vertical: str¶

channel_code: str¶

estimand: str¶

schema_version: str¶

label()[source]¶

Return type:: str

class mmm_framework.benchmarks.schema.RejectedStudy(**data)[source]¶

Bases: BaseModel

Ingest-gate rejection, with the reason kept for the internal audit.

study_id: str¶

contributor_hash: str¶

reason: str¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.CellGateAudit(**data)[source]¶

Bases: BaseModel

Per-cell gate outcome. internal_reason never publishes.

The published reason is deliberately generic: a suppression reason like “dominance 67%” would itself disclose that one brand carries the cell.

cell: CellKey¶

n_records: int¶

n_contributors: int¶

spend_dominance: float | None¶

precision_dominance: float¶

publish: bool¶

internal_reason: str¶

published_reason: str¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.ReleaseRecord(**data)[source]¶

Bases: BaseModel

One entry in the ε ledger: a number that crossed the boundary.

epsilon == 0 marks a post-processed (derived) or carried-forward release — free under DP, recorded for lineage.

release_id: str¶

vintage: str¶

name: str¶

statistic: str¶

value: float¶

epsilon: float¶

laplace_scale: float¶

members: tuple[str, ...]¶

vertical: str | None¶

estimand: str | None¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.PublishedCell(**data)[source]¶

Bases: BaseModel

A client-visible benchmark cell, assembled only from ledgered releases.

vertical: str¶

channel_code: str¶

estimand: str¶

n_band: str¶

center: float¶

spread: float¶

noise_scale: float¶

epsilon_spent: float¶

derived: bool¶

offsets: dict[str, float]¶

stale_since: str | None¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.SuppressedCell(**data)[source]¶

Bases: BaseModel

Client-visible record of a cell that did not publish.

vertical: str¶

channel_code: str¶

estimand: str¶

reason: str¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class mmm_framework.benchmarks.schema.PublishedVintage(**data)[source]¶

Bases: BaseModel

The full published artifact for one vintage (client-safe).

vintage: str¶

cells: list[PublishedCell]¶

suppressed: list[SuppressedCell]¶

epsilon_per_release: float¶

worst_case_annual_epsilon: float¶

frozen: list[str]¶

model_config: ClassVar[ConfigDict] = {}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

Estimand Catalog¶

The versioned catalog of benchmarkable estimands.

Only estimands defined to be comparable across brands may enter the pool. Average ROI is deliberately absent: it confounds margin structure, price point, market scale, and curve position (pitfalls doc §3). bench_iroas is quoted at a reference operating point and pooled with a spend moderator for the same reason — raw average pooling of iROAS is banned, moderated reference-point iROAS is not.

mmm_framework.benchmarks.estimands.CATALOG_VERSION = 'bench-est-v1'¶: Bump when the catalog’s definitions change; contributions carry it via the schema version, and cells never pool across catalog generations.

class mmm_framework.benchmarks.estimands.BenchmarkEstimand(name, description, scale='log', requires_finance_basis=False, performance=True, provenance=('model', 'experiment', 'calibrated_model'), notes='', reference_point=<factory>)[source]¶

Bases: object

One entry in the benchmark estimand catalog.

scale is the pooling scale. Ratio quantities pool on the log scale: effects are multiplicative across brands, and a symmetric-normal meta-model on the raw scale gets dragged by the right tail.

name: str¶

description: str¶

scale: str = 'log'¶

requires_finance_basis: bool = False¶

performance: bool = True¶

provenance: tuple[str, ...] = ('model', 'experiment', 'calibrated_model')¶

notes: str = ''¶

reference_point: dict[str, float]¶

mmm_framework.benchmarks.estimands.get(name)[source]¶

Look up a catalog entry; unknown names fail loudly.

Return type:: BenchmarkEstimand

mmm_framework.benchmarks.estimands.all_estimands()[source]¶

Return type:: list[BenchmarkEstimand]

Contribute¶

Project-side contribution extraction: gates, coarsening, consent.

Runs inside the client’s project and sees raw posteriors; only the summary ContributionRecord it emits ever leaves. Three responsibilities, in order:

Eligibility gates — approximate fits are excluded (their uncertainty is not calibrated and the meta-model consumes the SE as data); the posterior must actually have contracted from its prior (a prior-dominated read launders the default prior into the benchmark, which then becomes the next default prior — the echo-chamber loop); minimum panel length.
Coarsening at the source — covariates leave as bands/tiers, never exact values. This module is the single place the coarsening rules live.
Consent — no contribution without an explicit opt-in; the record carries the consent version it was made under so revocation can be honored retroactively (delete-by-contributor-hash on the store).

mmm_framework.benchmarks.contribute.CONTRIBUTION_QUANTILES = (0.05, 0.25, 0.5, 0.75, 0.95)¶: Quantiles carried on every contribution record.

mmm_framework.benchmarks.contribute.BUDGET_BANDS = ((1000000.0, '<$1M'), (10000000.0, '$1–10M'), (100000000.0, '$10–100M'), (inf, '>$100M'))¶: Order-of-magnitude annual media budget bands (coarse for privacy).

exception mmm_framework.benchmarks.contribute.EligibilityError[source]¶

Bases: ValueError

A gate refused the contribution; the message says which and why.

mmm_framework.benchmarks.contribute.contributor_hash(tenant_id, salt)[source]¶

Salted, tenant-stable pseudonym. The raw tenant id never leaves.

Return type:: str

mmm_framework.benchmarks.contribute.budget_band(annual_media_spend)[source]¶

Coarsen an exact annual budget to its order-of-magnitude band.

Return type:: str

mmm_framework.benchmarks.contribute.summarize_draws(draws, policy=None)[source]¶

Posterior draws → the summary that crosses the boundary.

The SE floor applies here too: a contribution claiming near-zero sd would own the precision-weighted pool exactly like a poisoned lift study.

Return type:: dict

mmm_framework.benchmarks.contribute.check_model_eligibility(results, *, channel, min_weeks=52, min_contraction=0.1, contraction=None)[source]¶

Raise EligibilityError unless a fitted model may contribute.

Return type:: None

mmm_framework.benchmarks.contribute.build_contribution(*, estimand, channel_code, vertical, draws, contributor, provenance=Provenance.MODEL, covariates=None, consent=False, consent_version='consent-v1', source_id=None, policy=None)[source]¶

Build one contribution record from posterior draws on the pooled scale.

Return type:: ContributionRecord

mmm_framework.benchmarks.contribute.contributions_from_model(results, *, vertical, contributor, channel_draws, covariates=None, consent=False, contraction_by_channel=None, min_weeks=52, policy=None, source_id=None)[source]¶

Extract every eligible contribution from one fitted model.

Return type:: list[ContributionRecord]

Ingest¶

Ingest gates and estimand normalization for archived lift studies.

Whitepaper §2.5 as code. The pool has exactly one experimental estimand: log incremental ROAS at a declared value basis (bench_iroas). Archives hold at least five things all called “the result of the test”; they are not interchangeable:

iROAS (conversion lift): pools directly, y = log(iROAS).
Absolute lift: converts via the declared value basis; the basis is an estimate, so its relative error enters the SE in quadrature. Treating the basis as exact understates every converted SE and overweights converted records in the precision-weighted pool.
Relative lift: needs the counterfactual baseline volume too; archived as “+12%” alone it has no path to a dollar scale and is rejected.
Attributed ROAS (pixel): a different estimand, not a noisier version of iROAS — rejected from the causal pool at any weight.
Brand lift (survey): different outcome, routed out of the ROI pool.

One addition beyond the reject rules: the SE floor. The meta-model’s mean is precision-weighted, so a study claiming near-zero SE — winner’s-curse selection, archival error, or deliberate poisoning — would own the pool.

mmm_framework.benchmarks.ingest.REASON_BRAND_LIFT = 'brand-lift: wrong estimand family'¶: Reject reasons (stable strings — the internal audit groups by them).

mmm_framework.benchmarks.ingest.se_from_ci(lo, hi, level)[source]¶

Recover the log-scale SE from a ratio-scale CI at its reported level.

Using the reported level matters: treating a 90% CI as a 95% one misstates every weight downstream.

Return type:: float

class mmm_framework.benchmarks.ingest.IngestResult(records=<factory>, rejects=<factory>)[source]¶

Bases: object

Accepted contribution records plus the per-study reject audit.

records: list[ContributionRecord]¶

rejects: list[RejectedStudy]¶

reject_counts()[source]¶

Return type:: dict[str, int]

__init__(records=<factory>, rejects=<factory>)¶

mmm_framework.benchmarks.ingest.normalize_study(study, policy)[source]¶

Route one archived study: a contribution record, or a reject with reason.

Return type:: ContributionRecord | RejectedStudy

mmm_framework.benchmarks.ingest.ingest_studies(studies, policy=None)[source]¶

Run the ingest gates over an archive; nothing is silently dropped.

Return type:: IngestResult

Store¶

The central contribution store: append-only, deduped, audited.

Everything right of the export boundary sees only what this store holds — contribution records, the ε ledger, and published vintages. Stdlib sqlite so the lean core stays dependency-free; pass ":memory:" (the default) for tests and notebooks, a path for the platform deployment.

Dedup rule: the natural key is (cell, contributor_hash, source_id) — a re-fit or re-ingest of the same source replaces its record rather than duplicating it, so frequently-refitting clients cannot dominate the pool. Distinct studies from one contributor keep distinct source_id values and legitimately coexist.

class mmm_framework.benchmarks.store.BenchmarkStore(path=':memory:')[source]¶

Bases: object

SQLite-backed contribution store with an audit trail.

__init__(path=':memory:')[source]¶

close()[source]¶

Return type:: None

add_contribution(record, *, now_ts=None)[source]¶

Return type:: None

add_contributions(records, *, now_ts=None)[source]¶

Return type:: None

contributions(*, vertical=None, channel_code=None, estimand=None, contributor_hash=None)[source]¶

Return type:: list[ContributionRecord]

cells()[source]¶

All contributions grouped by benchmark cell.

Return type:: dict[CellKey, list[ContributionRecord]]

delete_contributor(contributor_hash, *, now_ts=None)[source]¶

Consent revocation: remove every record from one contributor.

Already-published vintages are immutable (published noise cannot be un-released); deletion takes effect from the next vintage. The audit entry is the record consent requests are answered from.

Return type:: int

record_release(release, *, now_ts=None)[source]¶

Return type:: None

ledger()[source]¶

Return type:: EpsilonLedger

save_vintage(vintage, *, now_ts=None)[source]¶

Return type:: None

get_vintage(vintage)[source]¶

Return type:: PublishedVintage | None

latest_vintage()[source]¶

Return type:: PublishedVintage | None

list_vintages()[source]¶

Return type:: list[str]

audit_log(*, action=None, contributor_hash=None)[source]¶

Query the audit trail (consent/deletion requests are answered here).

Return type:: list[dict[str, Any]]

Meta Model¶

Bayesian random-effects meta-analysis over contribution records.

The pooling engine of the benchmark: per (vertical × channel × estimand) family, contributed means with known SEs are pooled hierarchically — between-contributor heterogeneity tau_v per vertical, moderators for market structure and curve position, and a provenance bias term delta_m identified by contributors who supply both a model read and an experimental read of the same channel.

The posterior is internal. Nothing here is released directly: the sensitivity of a posterior functional to one contributor’s data has no clean bound, so publication happens only through the clipped-statistic DP releases in publish. The posterior’s role is the internal benchmark, moderator science, and the shrinkage machinery of the internal view.

PyMC imports are local to fit_meta_model(), keeping the lean core’s import path free of the sampler stack.

class mmm_framework.benchmarks.meta_model.ModeratorSpec(name, covariate, kind='binary', value=None, reference=None)[source]¶

Bases: object

One meta-regression moderator built from a covariate.

name: str¶

covariate: str¶

kind: str = 'binary'¶

value: Any = None¶

reference: float | None = None¶

__init__(name, covariate, kind='binary', value=None, reference=None)¶

mmm_framework.benchmarks.meta_model.DEFAULT_LIFT_MODERATORS = (ModeratorSpec(name='retargeting', covariate='audience', kind='binary', value='retargeting', reference=None), ModeratorSpec(name='log_spend', covariate='log_spend', kind='continuous', value=None, reference=None))¶

audience scope is the largest hidden heterogeneity in lift archives, and log-spend is the curve-position term that makes an iROAS pool defensible at all.

Type:: Moderators applied by default to lift-study pools

class mmm_framework.benchmarks.meta_model.MetaModelResult(idata, verticals, contributors, contributor_vertical, moderators, reference=<factory>, has_provenance_bias=False)[source]¶

Bases: object

Fitted meta-model plus the indexing needed to read it.

idata: Any¶

verticals: list[str]¶

contributors: list[str]¶

contributor_vertical: list[str]¶

moderators: list[str]¶

reference: dict[str, float]¶

has_provenance_bias: bool = False¶

max_rhat()[source]¶

Return type:: float

population(vertical, prob=0.95)[source]¶

Population mean and heterogeneity for one vertical, with the two intervals every consumer must distinguish: the CI for the mean and the prediction interval for a new brand (tau included).

Return type:: dict[str, float]

moderator_summary()[source]¶

Return type:: DataFrame

shrinkage_table()[source]¶

Per-contributor pooled posterior vs vertical mean (internal view).

Return type:: DataFrame

__init__(idata, verticals, contributors, contributor_vertical, moderators, reference=<factory>, has_provenance_bias=False)¶

mmm_framework.benchmarks.meta_model.design_matrix(records, moderators)[source]¶

Build the moderator matrix; continuous moderators are centered on the reference point (recorded, so readouts quote the operating point).

Return type:: tuple[ndarray, list[str], dict[str, float]]

mmm_framework.benchmarks.meta_model.fit_meta_model(records, *, moderators=(ModeratorSpec(name='retargeting', covariate='audience', kind='binary', value='retargeting', reference=None), ModeratorSpec(name='log_spend', covariate='log_spend', kind='continuous', value=None, reference=None)), include_provenance_bias=True, draws=1000, tune=1000, chains=4, random_seed=11, progressbar=False)[source]¶

Fit the hierarchical random-effects pool over one estimand’s records.

Return type:: MetaModelResult

Publish¶

Vintage publication: orthogonalized DP releases, carry-forward, hard caps.

The release design is deliberately small and orthogonal per (vertical × estimand): a clipped population mean, a clipped mean absolute deviation (a naive DP variance is hopeless at k ≈ 5–10 cell sizes; the MAD’s sensitivity survives), and clipped contrasts for declared binary splits (channel, audience). Per-channel cell centers are then post-processing of the released mean and channel contrast — free under DP — so adding published cells never re-bills the same contributors.

Refresh discipline (whitepaper §5b):

Carry-forward is free. Republishing last vintage’s noised values costs nothing; only touching the data spends. The refresh trigger is membership churn — metadata the store knows without reading any value. (A value-dependent trigger leaks and must itself be accounted; that upgrade is a sparse-vector test, out of scope.)
ε_year is a hard gate, not a report. A release that would push any member past the annual cap is refused; the cell freezes at its last published values with the staleness disclosed. Exhaustion surfacing as staleness is the system working.

mmm_framework.benchmarks.publish.MAD_TO_SD = 1.2533¶

E|X - μ| = σ·√(2/π) for a normal, so σ ≈ 1.2533 × MAD (mean absolute deviation).

Type:: Normal-consistency factor

mmm_framework.benchmarks.publish.SPREAD_FLOOR = 0.05¶: Floor on the published spread so a small noised MAD cannot collapse the client band to a point.

class mmm_framework.benchmarks.publish.ContrastSpec(name, covariate, level_hi, level_lo)[source]¶

Bases: object

A clipped per-contributor binary contrast to release (e.g. audience).

name: str¶

covariate: str¶

level_hi: str¶

level_lo: str¶

__init__(name, covariate, level_hi, level_lo)¶

mmm_framework.benchmarks.publish.publish_vintage(store, *, vintage, policy=None, rng=None, contrasts=(ContrastSpec(name='retargeting_offset', covariate='audience', level_hi='retargeting', level_lo='prospecting'),), now_ts=None)[source]¶

Publish one vintage from the store’s current contributions.

vintage sorts lexicographically (“2026Q3”) and its leading 4 characters are the annual accounting prefix for the ε_year gate.

Return type:: PublishedVintage

mmm_framework.benchmarks.publish.placement(client_value, cell, *, policy=None, n_replicates=400, rng=None)[source]¶

Percentile placement against a published cell — as a range, honestly.

Return type:: dict

Prior Feedback¶

Prior feedback: published benchmarks → vertical default-prior presets.

The last stage of the loop, and the most dangerous one: the population posterior feeds back into ROI-parameterized default media priors for new clients in the same vertical. Two hard rules break the model → benchmark → prior → model echo chamber:

Prior width comes from the predictive distribution (between-brand τ and release noise included), never the posterior SE of μ — otherwise new clients get overconfident priors and a re-measurement of the pool’s mean masquerades as knowledge about their brand.
Experimental anchoring: a preset may only tighten beyond the floor width when the cell’s experimental share exceeds a threshold. A benchmark built purely from models is not allowed to sharpen the priors those models were fitted under.

mmm_framework.benchmarks.priors.DEFAULT_FLOOR_SD = 0.5¶: A preset may not tighten below this sd (pooled/log scale) unless anchored.

mmm_framework.benchmarks.priors.DEFAULT_ANCHOR_THRESHOLD = 0.3¶: Minimum weight of non-model contributions for a cell to count as anchored.

class mmm_framework.benchmarks.priors.VerticalPriorPreset(vertical, channel_code, estimand, mu, sd, scale, experimental_share, anchored, source_vintage)[source]¶

Bases: object

A default-prior preset for one (vertical × channel × estimand) cell.

vertical: str¶

channel_code: str¶

estimand: str¶

mu: float¶

sd: float¶

scale: str¶

experimental_share: float¶

anchored: bool¶

source_vintage: str¶

__init__(vertical, channel_code, estimand, mu, sd, scale, experimental_share, anchored, source_vintage)¶

mmm_framework.benchmarks.priors.experimental_share(records)[source]¶

Precision-weighted share of non-model evidence in a cell.

Return type:: float

mmm_framework.benchmarks.priors.vertical_prior_preset(cell, *, vintage, experimental_share, floor_sd=0.5, anchor_threshold=0.3, scale='log')[source]¶

Map one published cell into a default-prior preset.

Return type:: VerticalPriorPreset