Grid Search & Random Search

Grid Search

Grid Search exhaustively searches through a manually specified subset of the hyperparameter space. It tries every possible combination of the provided hyperparameters to find the best performing model based on a given evaluation metric.

• Systematic and comprehensive
• Computationally expensive, especially with many hyperparameters
• Easy to implement and interpret

Random Search

Random Search samples hyperparameter combinations randomly from a predefined distribution or range. Instead of trying every combination, it explores a wide variety of combinations, often finding good hyperparameters more efficiently.

• More efficient when the hyperparameter space is large
• Can discover better parameters in less time compared to Grid Search
• Does not guarantee finding the absolute best combination

Grid Search Example (Python, scikit-learn)

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Model and parameters
model = SVC()
param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['linear', 'rbf'],
    'gamma': ['scale', 'auto']
}

# Grid Search setup
grid_search = GridSearchCV(model, param_grid, cv=5)
grid_search.fit(X, y)

print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)

Random Search Example (Python, scikit-learn)

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from scipy.stats import randint

# Load data
iris = load_iris()
X, y = iris.data, iris.target

# Model and parameter distribution
model = RandomForestClassifier()
param_dist = {
    'n_estimators': randint(10, 200),
    'max_depth': randint(1, 20),
    'min_samples_split': randint(2, 20)
}

# Random Search setup
random_search = RandomizedSearchCV(model, param_dist, n_iter=10, cv=5, random_state=42)
random_search.fit(X, y)

print("Best parameters:", random_search.best_params_)
print("Best cross-validation score:", random_search.best_score_)