Data Loader¶

The data loader module handles MFF (Master Flat File) format parsing, validation, and conversion to panel datasets for modeling.

Data Loading¶

Data loading and transformation utilities for MFF format.

Handles variable-dimension data, dimension alignment, and panel construction.

exception mmm_framework.data_loader.MFFValidationError[source]¶

Bases: Exception

Raised when MFF data fails validation.

mmm_framework.data_loader.coerce_numeric_column(series, *, column, decimal='.')[source]¶

Coerce an MFF value column to float, tolerating real-world formatting.

Client exports routinely carry currency/thousands formatting ("$1,234", "1.234,56") or accounting negatives ("(123)"). Previously these flowed straight into groupby().sum() and raised an opaque TypeError deep in the stack. This normalizes them up front: :rtype: Series

already-numeric columns pass through unchanged;
$ € £ ¥ ₹ and whitespace are stripped; (123) -> -123;
thousands/decimal separators are normalized per decimal ("." US, "," EU, or "auto") so non-US-formatted data ingests;
recognized null tokens (and genuinely empty cells) become NaN;
any remaining unparseable value raises MFFValidationError naming the column and a few offending samples (an actionable message, not a TypeError).

mmm_framework.data_loader.validate_mff_structure(df, config)[source]¶

Validate MFF dataframe structure against config.

Returns list of warnings (empty if clean). Raises MFFValidationError for critical issues.

Return type:: list[str]

mmm_framework.data_loader.validate_variable_dimensions(df, var_config, config)[source]¶

Validate that a variable’s data matches its configured dimensions.

Returns (is_valid, message).

Return type:: tuple[bool, str]

class mmm_framework.data_loader.PanelCoordinates(periods, geographies=None, products=None, channels=<factory>, controls=<factory>)[source]¶

Bases: object

Coordinate labels for panel dimensions.

periods: DatetimeIndex¶

geographies: list[str] | None = None¶

products: list[str] | None = None¶

channels: list[str]¶

controls: list[str]¶

property has_geo: bool¶

property has_product: bool¶

property n_periods: int¶

property n_geos: int¶

property n_products: int¶

property n_obs: int¶: Total number of observations.

to_pymc_coords()[source]¶

Convert to PyMC coordinate dictionary.

Return type:: dict[str, list]

__init__(periods, geographies=None, products=None, channels=<factory>, controls=<factory>)¶

class mmm_framework.data_loader.PanelDataset(y, X_media, X_controls, coords, index, config, media_stats=<factory>, explicit_nan_mask=None, spend_raw=None)[source]¶

Bases: object

Structured panel data ready for modeling.

All arrays are aligned to the same index structure.

y: pd.Series¶

X_media: pd.DataFrame¶

X_controls: pd.DataFrame | None¶

coords: PanelCoordinates¶

index: pd.MultiIndex | pd.DatetimeIndex¶

config: MFFConfig¶

media_stats: dict[str, dict]¶

explicit_nan_mask: dict[str, pd.Series] | None = None¶

spend_raw: dict[str, NDArray] | None = None¶

property n_obs: int¶

property n_channels: int¶

property n_controls: int¶

property is_panel: bool¶: True if data has geo or product dimensions.

to_numpy()[source]¶

Convert to numpy arrays for modeling.

Return type:: tuple[NDArray, NDArray, NDArray | None]

as_dataset()[source]¶

View this panel through the flexible role-tagged Dataset.

Additive adapter for the generalized data layer: the MMM roles are tagged (kpi→target, media→predictor, control→control) and every reader keeps working via the Dataset MMM views. See mmm_framework.dataset.

Return type:: Dataset

get_media_for_channel(channel)[source]¶

Get media series for a specific channel.

Return type:: Series

compute_spend_shares()[source]¶

Compute share of total spend for each channel.

Return type:: Series

summary()[source]¶

Generate summary string.

Return type:: str

__init__(y, X_media, X_controls, coords, index, config, media_stats=<factory>, explicit_nan_mask=None, spend_raw=None)¶

class mmm_framework.data_loader.MFFLoader(config)[source]¶

Bases: object

Loads and parses MFF format data into panel structure.

Handles: - Variable extraction by name - Dimension alignment (national to geo, etc.) - Missing value handling - Date parsing and frequency alignment

__init__(config)[source]¶

load(data)[source]¶

Load MFF data from DataFrame or file path.

Return type:: MFFLoader

Parameters¶

datapd.DataFrame or str: Either a DataFrame in MFF format or path to CSV/parquet file.

Returns¶

selfMFFLoader: For method chaining.

set_allocation_weights(dimension, weights)[source]¶

Set custom allocation weights for dimension disaggregation.

Return type:: MFFLoader

Parameters¶

dimensionDimensionType: The dimension to set weights for (GEOGRAPHY or PRODUCT).
weightspd.Series or dict: Weights indexed by dimension level (e.g., geo names). Will be normalized to sum to 1.

Returns¶

selfMFFLoader: For method chaining.

build_panel()[source]¶

Build complete panel dataset from loaded MFF data.

Return type:: PanelDataset

Returns¶

PanelDataset: Structured data ready for modeling.

mmm_framework.data_loader.load_mff(data, config, geo_weights=None, product_weights=None)[source]¶

Convenience function to load MFF data in one call.

Return type:: PanelDataset

Parameters¶

datapd.DataFrame or str: MFF data or path to file.
configMFFConfig: Configuration specifying variables and dimensions.
geo_weightspd.Series or dict, optional: Custom weights for geo allocation.
product_weightspd.Series or dict, optional: Custom weights for product allocation.

Returns¶

PanelDataset: Ready-to-use panel data.

Examples¶

>>> config = create_simple_mff_config(
...     kpi_name="Sales",
...     media_names=["TV", "Digital", "Social"],
...     control_names=["Price", "Distribution"],
...     kpi_dimensions=[DimensionType.PERIOD, DimensionType.GEOGRAPHY],
... )
>>> panel = load_mff("data.csv", config)
>>> print(panel.summary())

mmm_framework.data_loader.mff_from_wide_format(df, period_col, value_columns, geo_col=None, product_col=None)[source]¶

Convert wide-format data to MFF format.

Return type:: DataFrame

Parameters¶

dfpd.DataFrame: Wide format dataframe with one column per variable.
period_colstr: Name of the date/period column.
value_columnsdict: Mapping of column names to variable names. E.g., {“sales”: “Sales”, “tv_spend”: “TV”}
geo_colstr, optional: Name of geography column.
product_colstr, optional: Name of product column.

Returns¶

pd.DataFrame: Data in MFF format.

mmm_framework.data_loader.generate_complete_date_range(start_date, end_date, frequency)[source]¶

Generate a complete date range for the given frequency.

Return type:: DatetimeIndex

Parameters¶

start_datepd.Timestamp: First date in range.
end_datepd.Timestamp: Last date in range.
frequencystr: ‘W’ for weekly, ‘D’ for daily, ‘M’ for monthly.

Returns¶

pd.DatetimeIndex: Complete date range with no gaps.

mmm_framework.data_loader.build_complete_index(periods, geographies=None, products=None, period_col='Period', geo_col='Geography', product_col='Product')[source]¶

Build a complete index covering all combinations of dimensions.

This ensures we have entries for every date x geo x product combination, enabling proper detection of missing vs explicit NaN values.

Return type:: Index | MultiIndex

mmm_framework.data_loader.extract_with_nan_tracking(df, variable_name, cols, target_index, fill_value=0.0, preserve_explicit_nan=True)[source]¶

Extract a variable and track which values were explicitly NaN.

Return type:: tuple[Series, Series]

Parameters¶

dfpd.DataFrame: Raw MFF data.
variable_namestr: Name of variable to extract.
colsMFFColumnConfig: Column name mappings.
target_indexpd.Index or pd.MultiIndex: Complete index to align data to.
fill_valuefloat: Value to fill for missing rows (not present in source).
preserve_explicit_nanbool: If True, keep explicit NaN values; if False, fill them too.

Returns¶

valuespd.Series: Extracted values aligned to target_index.
explicit_nan_maskpd.Series: Boolean mask where True = value was explicitly NaN in source.

class mmm_framework.data_loader.RaggedMFFLoader(config)[source]¶

Bases: object

Loads MFF data with proper handling of ragged/sparse data.

Key behaviors: - Generates complete date range based on min/max dates in data - Missing date/dimension combinations filled with configured value (default 0) - Explicit NaN values in source data are preserved - Tracks which values were explicitly NaN for downstream handling

__init__(config)[source]¶

load(data)[source]¶

Load MFF data from DataFrame or file path.

Return type:: RaggedMFFLoader

build_panel()[source]¶

Build complete panel dataset with ragged data handling.

Missing dates/combinations are filled with zeros (or configured value). Explicit NaN values in source data are preserved.

Return type:: PanelDataset

mmm_framework.data_loader.load_ragged_mff(data, config)[source]¶

Load MFF data with ragged data handling.

Return type:: PanelDataset

Parameters¶

datapd.DataFrame or str: MFF data or path to file.
configMFFConfig: Configuration specifying variables and dimensions.

Returns¶

PanelDataset: Panel data with missing dates filled and explicit NaN preserved.

Examples¶

>>> config = MFFConfig(
...     kpi=VariableConfig(name="Sales", dimensions=[DimensionType.PERIOD]),
...     media_channels=[
...         VariableConfig(name="TV", dimensions=[DimensionType.PERIOD]),
...         VariableConfig(name="Digital", dimensions=[DimensionType.PERIOD]),
...     ],
...     fill_missing_media=0.0,
...     preserve_explicit_nan=True,
... )
>>> panel = load_ragged_mff("ragged_data.csv", config)
>>> print(f"Filled {(panel.X_media == 0).sum().sum()} missing values with 0")

Data Preparation¶

Data preparation utilities for BayesianMMM.

This module provides helper classes for preparing and standardizing data for use in Bayesian Marketing Mix Models.

class mmm_framework.data_preparation.ScalingParameters(y_mean, y_std, media_max, control_mean=None, control_std=None)[source]¶

Bases: object

Container for data scaling parameters.

These parameters are needed to transform predictions back to the original scale and for consistent predictions on new data.

Attributes¶

y_meanfloat: Mean of the target variable.
y_stdfloat: Standard deviation of the target variable.
media_maxdict[str, float]: Maximum adstocked value for each media channel.
control_meanNDArray | None: Mean of control variables (None if no controls).
control_stdNDArray | None: Standard deviation of control variables.

y_mean: float¶

y_std: float¶

media_max: dict[str, float]¶

control_mean: NDArray | None = None¶

control_std: NDArray | None = None¶

to_dict()[source]¶

Convert to serializable dictionary.

Return type:: dict[str, Any]

classmethod from_dict(data)[source]¶

Create from dictionary.

Return type:: ScalingParameters

__init__(y_mean, y_std, media_max, control_mean=None, control_std=None)¶

class mmm_framework.data_preparation.PreparedData(y, y_raw, X_media_adstocked, X_media_raw, X_controls, X_controls_raw, scaling_params, n_obs, n_channels, n_controls, channel_names, control_names, time_idx, geo_idx, product_idx, n_periods, n_geos, n_products, has_geo, has_product, t_scaled, seasonality_features=<factory>, trend_features=<factory>, geo_names=None, product_names=None)[source]¶

Bases: object

Container for prepared/transformed data.

This holds all the preprocessed data needed for model building, including standardized values, indices, and feature matrices.

Attributes¶

yNDArray: Standardized target variable.
y_rawNDArray: Raw target variable.
X_media_adstockeddict[float, NDArray]: Adstocked and normalized media data by alpha value.
X_media_rawNDArray: Raw media data.
X_controlsNDArray | None: Standardized control variables (None if no controls).
X_controls_rawNDArray | None: Raw control variables.
scaling_paramsScalingParameters: Parameters used for standardization.
n_obsint: Number of observations.
n_channelsint: Number of media channels.
n_controlsint: Number of control variables.
channel_nameslist[str]: Names of media channels.
control_nameslist[str]: Names of control variables.
time_idxNDArray: Time period index for each observation.
geo_idxNDArray: Geography index for each observation.
product_idxNDArray: Product index for each observation.
n_periodsint: Number of unique time periods.
n_geosint: Number of unique geographies.
n_productsint: Number of unique products.
has_geobool: Whether data has geography dimension.
has_productbool: Whether data has product dimension.
t_scaledNDArray: Time values scaled to [0, 1].
seasonality_featuresdict[str, NDArray]: Fourier features for seasonality.
trend_featuresdict[str, Any]: Features for trend modeling.

y: NDArray¶

y_raw: NDArray¶

X_media_adstocked: dict[float, NDArray]¶

X_media_raw: NDArray¶

X_controls: NDArray | None¶

X_controls_raw: NDArray | None¶

scaling_params: ScalingParameters¶

n_obs: int¶

n_channels: int¶

n_controls: int¶

channel_names: list[str]¶

control_names: list[str]¶

time_idx: NDArray¶

geo_idx: NDArray¶

product_idx: NDArray¶

n_periods: int¶

n_geos: int¶

n_products: int¶

has_geo: bool¶

has_product: bool¶

t_scaled: NDArray¶

seasonality_features: dict[str, NDArray]¶

trend_features: dict[str, Any]¶

geo_names: list[str] | None = None¶

product_names: list[str] | None = None¶

__init__(y, y_raw, X_media_adstocked, X_media_raw, X_controls, X_controls_raw, scaling_params, n_obs, n_channels, n_controls, channel_names, control_names, time_idx, geo_idx, product_idx, n_periods, n_geos, n_products, has_geo, has_product, t_scaled, seasonality_features=<factory>, trend_features=<factory>, geo_names=None, product_names=None)¶

class mmm_framework.data_preparation.DataPreparator(panel, adstock_alphas, seasonality_config=None, trend_config=None)[source]¶

Bases: object

Prepares panel data for Bayesian MMM.

This class handles all data preprocessing steps including: - Standardization of target and control variables - Adstock transformation and normalization of media data - Creation of seasonality features (Fourier terms) - Creation of trend features (spline/piecewise/GP) - Index creation for hierarchical dimensions

Parameters¶

panelPanelDataset: The panel dataset to prepare.
adstock_alphaslist[float]: Alpha values for geometric adstock.
seasonality_configSeasonalityConfig | None: Configuration for seasonality features.
trend_configAny | None: Configuration for trend features.

Examples¶

>>> from mmm_framework.data_preparation import DataPreparator
>>> preparator = DataPreparator(
...     panel=panel,
...     adstock_alphas=[0.0, 0.3, 0.5, 0.7, 0.9],
... )
>>> prepared = preparator.prepare()
>>> print(prepared.n_obs, prepared.n_channels)

__init__(panel, adstock_alphas, seasonality_config=None, trend_config=None)[source]¶

prepare()[source]¶

Prepare all data for model building.

Return type:: PreparedData

Returns¶

PreparedData: Container with all prepared data.

mmm_framework.data_preparation.standardize_array(data, epsilon=1e-08)[source]¶

Standardize an array to zero mean and unit variance.

Return type:: tuple[numpy.ndarray, float, float]

Parameters¶

dataNDArray: Input data array.
epsilonfloat: Small value added to std to prevent division by zero.

Returns¶

tuple: (standardized_data, mean, std)

mmm_framework.data_preparation.unstandardize_array(data, mean, std)[source]¶

Reverse standardization.

Return type:: numpy.ndarray

Parameters¶

dataNDArray: Standardized data.
meanfloat: Original mean.
stdfloat: Original standard deviation.

Returns¶

NDArray: Data in original scale.