Feature Selection vs Feature Extraction

Feature Selection

Feature Selection involves selecting a subset of the original features from the dataset without transforming them. The goal is to keep the most relevant features that contribute the most to the predictive model.

• Works by choosing important features and discarding irrelevant or redundant ones
• Does not alter the original features
• Helps reduce overfitting, improve model interpretability, and decrease training time
• Common methods include filter, wrapper, and embedded techniques

Feature Extraction

Feature Extraction transforms the original features into a new space, creating new features by combining or projecting the original data. The aim is to capture the most important information in fewer dimensions.

• Creates new features from the original ones via transformations
• Reduces dimensionality by mapping data into a lower-dimensional space
• Helps with data compression, noise reduction, and improving model accuracy
• Common techniques include Principal Component Analysis (PCA), t-SNE, and autoencoders

Feature Selection Example: Using SelectKBest in Python

from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest, f_classif

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Select top 2 features based on ANOVA F-test
selector = SelectKBest(score_func=f_classif, k=2)
X_new = selector.fit_transform(X, y)

print("Original shape:", X.shape)
print("Reduced shape:", X_new.shape)

Feature Extraction Example: Using PCA in Python

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA

# Load dataset
iris = load_iris()
X = iris.data

# Reduce to 2 principal components
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

print("Original shape:", X.shape)
print("Reduced shape:", X_pca.shape)