Preprocessor

This module provides tools for preprocessing datasets, including merging, splitting data, and removing correlated features. Also, it recommends the best approaches for model choices based on the data.

datalizer.preprocessor.preprocess_data(X_train: DataFrame, y_train: DataFrame, merge_col: str, target_col: str, val: bool = False, test_size: float = 0.2, remove_corr: bool = False, corr_threshold: float = 0.95) → tuple

Merge the feature set (X_train) and target (y_train), optionally remove highly correlated features, and split the data into training and validation sets if specified.

Parameters:

• X_train (pd.DataFrame) – Feature DataFrame.

• y_train (pd.DataFrame) – Target DataFrame.

• merge_col (str) – Column name to merge on.

• target_col (str) – Target column name.

• val (bool, optional) – Whether to split the data into training and validation sets (default is False).

• test_size (float, optional) – Proportion of the dataset to include in the validation split (default is 0.2).

• remove_corr (bool, optional) – Whether to remove highly correlated features (default is False).

• corr_threshold (float, optional) – Correlation threshold for feature removal (default is 0.95).

Returns:

If val is True, returns X_train_split, X_val_split, y_train_split, y_val_split, and dropped_corr_feats. If val is False, returns X_processed, y_processed, and dropped_corr_feats.

Return type:

Touple

datalizer.preprocessor.recommend_approach(X_train_split: DataFrame, y_train_split: DataFrame) → dict

Recommends a modeling approach based on the characteristics of the dataset. It suggests appropriate models, checks for data imbalance, and recommends strategies to prevent overfitting.

Parameters:

X_train_splitpd.DataFrame

The feature dataset (excluding target column).

y_train_splitpd.DataFrame

The target DataFrame containing the target column.

Returns:

dict

A dictionary containing recommended models and strategies to prevent overfitting.