More Data, More Problems: Why Adding Features Can Make Your AI Brittle

It’s one of the first things you learn in machine learning: more data is good. It feels so intuitive, right? If you give your model more information to learn from, it should be able to make smarter predictions. So, we throw in every column we can get our hands on—user demographics, transactional data, sensor readings, you name it. We call this the "kitchen-sink" model.

But here’s the uncomfortable truth I’ve learned over the years: this instinct often backfires. Badly.

What looks like a sophisticated model in your Jupyter notebook can become an incredibly fragile, unpredictable mess in the real world. Every single feature you add is another string attached to an upstream data source. Another pipeline that can break. Another API that can change. A single missing value or a schema change in some obscure column can silently start wrecking your predictions, and you might not even notice until it's too late.

The real problem isn't just about complexity; it's about stability. When you flood a model with tons of features, especially ones that are redundant or just plain noise, the model gets confused. It struggles to figure out which features actually matter. This leads to a model that’s not just complicated, but brittle.

Let's break down exactly why this happens and see it in action. We'll build a couple of models to predict property prices and see why a lean, simple model often beats a bloated, "smarter" one in the long run.

---

First, Let's Build Our Playground: A Fake Real Estate Market

To really see what's going on, we need a controlled environment. So, let’s create a synthetic dataset for predicting house prices. This way, we know exactly which features are important and which are just noise.

Imagine we have 800 house listings. We’ll create:

1. True Signal Features: The stuff that really drives prices. Think sqft (square footage), bedrooms, and neighborhood. These are our superstars.
2. Correlated "Copycat" Features: These are features that basically repeat information we already have. For example, floor_area_m2 is just sqft in a different unit. total_rooms is almost the same thing as bedrooms. They don't add much new information.
3. Weak or Useless Features: Things that might have a tiny, tiny effect or none at all. Think door_color_code or the distance_to_nearest_mcdonalds.
4. Pure Noise: To simulate a real-world messy dataset, we’ll even add 90 columns of completely random numbers.

Finally, we’ll create our target: the price. We'll calculate it using a clear formula based on our "True Signal" features, with a little bit of randomness mixed in to make it realistic.

Here’s the code to set that up.

pip install seaborn scikit-learn pandas numpy matplotlib

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import seaborn as sns
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
import warnings

warnings.filterwarnings("ignore")
plt.rcParams.update({
    "figure.facecolor": "#FAFAFA",
    "axes.facecolor": "#FAFAFA",
    "axes.spines.top": False,
    "axes.spines.right":False,
    "axes.grid": True,
    "grid.color": "#E5E5E5",
    "grid.linewidth": 0.8,
    "font.family": "monospace",
})
SEED = 42
np.random.seed(SEED)

N = 800 # training samples

# ── True signal features ────────────────────────────────────
sqft = np.random.normal(1800, 400, N) # strong signal
bedrooms = np.round(sqft / 550 + np.random.normal(0, 0.4, N)).clip(1, 6)
neighborhood = np.random.choice([0, 1, 2], N, p=[0.3, 0.5, 0.2]) # categorical

# ── Derived / correlated features (multicollinearity) ───────
total_rooms = bedrooms + np.random.normal(2, 0.3, N) # ≈ bedrooms
floor_area_m2 = sqft * 0.0929 + np.random.normal(0, 1, N) # ≈ sqft in m²
lot_sqft = sqft * 1.4 + np.random.normal(0, 50, N) # ≈ sqft scaled

# ── Weak / spurious features ────────────────────────────────
door_color_code = np.random.randint(0, 10, N).astype(float)
bus_stop_age_yrs = np.random.normal(15, 5, N)
nearest_mcdonalds_m = np.random.normal(800, 200, N)

# ── Pure noise features (simulate 90 random columns) ────────
noise_features = np.random.randn(N, 90)
noise_df = pd.DataFrame(
    noise_features,
    columns=[f"noise_{i:03d}" for i in range(90)]
)

# ── Target: house price ─────────────────────────────────────
price = (
    120 * sqft
    + 8_000 * bedrooms
    + 30_000 * neighborhood
    - 15 * bus_stop_age_yrs # tiny real effect
    + np.random.normal(0, 15_000, N) # irreducible noise
)

# ── Assemble DataFrames ──────────────────────────────────────
signal_cols = ["sqft", "bedrooms", "neighborhood", "total_rooms", "floor_area_m2", "lot_sqft", "door_color_code", "bus_stop_age_yrs", "nearest_mcdonalds_m"]

df_base = pd.DataFrame({
    "sqft": sqft,
    "bedrooms": bedrooms,
    "neighborhood": neighborhood,
    "total_rooms": total_rooms,
    "floor_area_m2": floor_area_m2,
    "lot_sqft": lot_sqft,
    "door_color_code": door_color_code,
    "bus_stop_age_yrs": bus_stop_age_yrs,
    "nearest_mcdonalds_m": nearest_mcdonalds_m,
    "price": price,
})

df_full = pd.concat([df_base.drop("price", axis=1), noise_df, df_base[["price"]]], axis=1)

LEAN_FEATURES = ["sqft", "bedrooms", "neighborhood"]
NOISY_FEATURES = [c for c in df_full.columns if c != "price"]

print(f"Lean model features : {len(LEAN_FEATURES)}")
print(f"Noisy model features: {len(NOISY_FEATURES)}")
print(f"Dataset shape : {df_full.shape}")

Now we have two sets of features to build our models:

• Lean Model: Just the 3 superstar features.
• Noisy Model: All 100+ features—the good, the bad, and the ugly.

Let the games begin.

---

The "Who Gets the Credit?" Problem: Redundant Features

The first issue with our "kitchen-sink" model is something called multicollinearity. It sounds complicated, but the idea is simple.

Imagine two people, Alex and Ben, work on a project together. They contribute almost the exact same work. When it’s time for a performance review, how does your boss decide who gets the credit? It's tough. She might give Alex 60% and Ben 40% this time, and next time it might be the other way around. The credit assignment is unstable.

Your regression model faces the same dilemma. When you feed it sqft and floor_area_m2, it sees two features that move in perfect sync. They're telling the same story. The model doesn’t know which one to "reward" with a strong coefficient (weight). So, it just splits the credit arbitrarily.

This dilutes the importance of the actual signal and makes your model's logic confusing and unstable.

Let's look at just how correlated our "copycat" features are.

print("\n── Correlation between correlated feature pairs ──")
corr_pairs = [
    ("sqft", "floor_area_m2"),
    ("sqft", "lot_sqft"),
    ("bedrooms", "total_rooms"),
]
for a, b in corr_pairs:
    r = np.corrcoef(df_full[a], df_full[b])[0, 1]
    print(f" {a:20s} {b:20s} r = {r:.3f}")

fig, axes = plt.subplots(1, 3, figsize=(14, 4))
fig.suptitle("Weight Dilution: Correlated Feature Pairs", fontsize=13, fontweight="bold", y=1.02)

for ax, (a, b) in zip(axes, corr_pairs):
    ax.scatter(df_full[a], df_full[b], alpha=0.25, s=12, color="#3B6FD4")
    r = np.corrcoef(df_full[a], df_full[b])[0, 1]
    ax.set_title(f"r = {r:.3f}", fontsize=11)
    ax.set_xlabel(a); ax.set_ylabel(b)

plt.tight_layout()
plt.savefig("01_multicollinearity.png", dpi=150, bbox_inches="tight")
plt.show()
print("Saved → 01_multicollinearity.png")

As you can see, the correlations are nearly perfect. When your model sees this, it gets confused. This confusion doesn't just make the model hard to interpret; it makes it fragile, as we're about to see.

---

The Unstable Personality Problem: Why Retraining Is Risky

In the real world, you don't just train a model once. You retrain it periodically—maybe every week or every month—on new data. You expect its core logic to stay pretty consistent, right?

Well, with a "kitchen-sink" model, that's not what happens.

Let's run an experiment. We’ll pretend it's a new month, 30 times in a row. Each "month," we'll grab a random sample of our data and retrain both our Lean model and our Noisy model. Then, we'll track the coefficients (the weights) assigned to our three superstar features: sqft, bedrooms, and neighborhood.

Think of these coefficients as the model's "opinion" on how important each feature is. We want these opinions to be stable.

N_CYCLES = 30
SAMPLE_SZ = 300 # size of each retraining slice

scaler_lean = StandardScaler()
scaler_noisy = StandardScaler()

# Fit scalers on full data so units are comparable
X_lean_all = scaler_lean.fit_transform(df_full[LEAN_FEATURES])
X_noisy_all = scaler_noisy.fit_transform(df_full[NOISY_FEATURES])
y_all = df_full["price"].values

lean_weights = [] # shape: (N_CYCLES, 3)
noisy_weights = [] # shape: (N_CYCLES, 3) -- first 3 cols only for comparison

for cycle in range(N_CYCLES):
    idx = np.random.choice(N, SAMPLE_SZ, replace=False)
    X_l = X_lean_all[idx]; y_c = y_all[idx]
    X_n = X_noisy_all[idx]
    
    m_lean = Ridge(alpha=1.0).fit(X_l, y_c)
    m_noisy = Ridge(alpha=1.0).fit(X_n, y_c)
    
    lean_weights.append(m_lean.coef_)
    noisy_weights.append(m_noisy.coef_[:3]) # sqft, bedrooms, neighborhood

lean_weights = np.array(lean_weights)
noisy_weights = np.array(noisy_weights)

print("\n── Coefficient Std Dev across 30 retraining cycles ──")
print(f"{'Feature':<18} {'Lean σ':>10} {'Noisy σ':>10} {'Amplification':>14}")
for i, feat in enumerate(LEAN_FEATURES):
    sl = lean_weights[:, i].std()
    sn = noisy_weights[:, i].std()
    print(f" {feat:<16} {sl:>10.1f} {sn:>10.1f} ×{sn/sl:.1f}")

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
fig.suptitle("Weight Instability: Lean vs. Noisy Model (30 Retraining Cycles)", fontsize=13, fontweight="bold", y=1.02)
colors = {"lean": "#2DAA6E", "noisy": "#E05C3A"}

for i, feat in enumerate(LEAN_FEATURES):
    ax = axes[i]
    ax.plot(lean_weights[:, i], color=colors["lean"], linewidth=2, label="Lean (3 features)", alpha=0.9)
    ax.plot(noisy_weights[:, i], color=colors["noisy"], linewidth=2, label="Noisy (100+ features)", alpha=0.9, linestyle="--")
    ax.set_title(f'Coefficient: "{feat}"', fontsize=11)
    ax.set_xlabel("Retraining Cycle")
    ax.set_ylabel("Standardised Weight")
    if i == 0:
        ax.legend(fontsize=9)

plt.tight_layout()
plt.savefig("02_weight_instability.png", dpi=150, bbox_inches="tight")
plt.show()
print("Saved → 02_weight_instability.png")

The results are stunning.

Look at the charts. The green line (our Lean model) is pretty stable. Its "opinions" don't change much from one retraining to the next. But the red, dashed line (the Noisy model) is all over the place! Its coefficients for sqft and bedrooms jump around wildly.

The numbers tell the same story: the sqft coefficient is 2.6 times more unstable in the noisy model. This is a model with an unstable personality. You can't trust its reasoning because its reasoning changes every time you retrain it. This is a huge red flag for a production system.

---

Drowning in the Noise

So why is the noisy model so unstable? Because we've drowned it in useless information.

Think of it like trying to have a conversation at a loud party. The important signals (the person you're talking to) are drowned out by the background noise. Your model is trying to listen for the signal from sqft and bedrooms, but it's being distracted by 90 columns of random noise and a bunch of other weak features.

When we plot the correlation of every single feature with the house price, the problem becomes crystal clear.

correlations = df_full[NOISY_FEATURES + ["price"]].corr()["price"].drop("price")
correlations = correlations.abs().sort_values(ascending=False)

fig, ax = plt.subplots(figsize=(14, 5))
bar_colors = [
    "#2DAA6E" if f in LEAN_FEATURES else
    "#E8A838" if f in ["total_rooms", "floor_area_m2", "lot_sqft", "bus_stop_age_yrs"] else
    "#CCCCCC"
    for f in correlations.index
]
ax.bar(range(len(correlations)), correlations.values, color=bar_colors, width=0.85, edgecolor="none")

# Legend patches
from matplotlib.patches import Patch
legend_elements = [
    Patch(facecolor="#2DAA6E", label="High-signal (lean set)"),
    Patch(facecolor="#E8A838", label="Correlated / low-signal"),
    Patch(facecolor="#CCCCCC", label="Pure noise"),
]
ax.legend(handles=legend_elements, fontsize=10, loc="upper right")

ax.set_title("Signal-to-Noise Ratio: |Correlation with Price| per Feature", fontsize=13, fontweight="bold")
ax.set_xlabel("Feature rank (sorted by |r|)")
ax.set_ylabel("|Pearson r| with price")
ax.set_xticks([])

plt.tight_layout()
plt.savefig("03_snr_degradation.png", dpi=150, bbox_inches="tight")
plt.show()
print("Saved → 03_snr_degradation.png")

Look at that chart. A tiny handful of features (in green and orange) actually have a meaningful relationship with price. The vast majority, the huge sea of gray bars, is pure noise.

When you force your model to consider all that gray junk, you dilute the signal. The model starts finding spurious patterns in the noise, which contributes to the instability we saw earlier.

---

Every Feature is a Ticking Time Bomb

This brings us to the most critical point for anyone running models in production: every feature you add is another potential point of failure.

Data in the real world is messy. It "drifts." The way a feature is measured might change, a data pipeline might have a bug, or the underlying behavior it represents could shift.

Let's simulate this. We'll take a weak feature, bus_stop_age_yrs, which is included in our Noisy model but not our Lean one. Now, let's pretend the data source for this feature starts bugging out, causing its values to drift upwards. What happens to our predictions?

def predict_with_drift(model, scaler, X_base, drift_col_idx, drift_magnitude, feature_cols):
    """Inject drift into one feature column and measure prediction shift."""
    X_drifted = X_base.copy()
    X_drifted[:, drift_col_idx] += drift_magnitude
    return model.predict(scaler.transform(X_drifted))

# Re-fit both models on the full dataset
sc_lean = StandardScaler().fit(df_full[LEAN_FEATURES])
sc_noisy = StandardScaler().fit(df_full[NOISY_FEATURES])

m_lean_full = Ridge(alpha=1.0).fit(
    sc_lean.transform(df_full[LEAN_FEATURES]), y_all)
m_noisy_full = Ridge(alpha=1.0).fit(
    sc_noisy.transform(df_full[NOISY_FEATURES]), y_all)

X_lean_raw = df_full[LEAN_FEATURES].values
X_noisy_raw = df_full[NOISY_FEATURES].values

base_lean = m_lean_full.predict(sc_lean.transform(X_lean_raw))
base_noisy = m_noisy_full.predict(sc_noisy.transform(X_noisy_raw))

# Drift the "bus_stop_age_yrs" feature (low-signal, yet in noisy model)
drift_col_noisy = NOISY_FEATURES.index("bus_stop_age_yrs")
drift_range = np.linspace(0, 20, 40) # up to 20-year drift in bus stop age
rmse_lean_drift, rmse_noisy_drift = [], []

for d in drift_range:
    preds_noisy = predict_with_drift(
        m_noisy_full, sc_noisy, X_noisy_raw, drift_col_noisy, d, NOISY_FEATURES)
    # Lean model doesn't even have this feature → unaffected
    rmse_lean_drift.append(
        np.sqrt(mean_squared_error(base_lean, base_lean))) # 0 by design
    rmse_noisy_drift.append(
        np.sqrt(mean_squared_error(base_noisy, preds_noisy)))

fig, ax = plt.subplots(figsize=(10, 5))
ax.plot(drift_range, rmse_lean_drift, color="#2DAA6E", linewidth=2.5, label="Lean model (feature not present)")
ax.plot(drift_range, rmse_noisy_drift, color="#E05C3A", linewidth=2.5, linestyle="--", label='Noisy model ("bus_stop_age_yrs" drifts)')
ax.fill_between(drift_range, rmse_noisy_drift, alpha=0.15, color="#E05C3A")

ax.set_xlabel("Feature Drift Magnitude (years)", fontsize=11)
ax.set_ylabel("Prediction Shift RMSE ($)", fontsize=11)
ax.set_title("Feature Drift Sensitivity:\nEach Extra Feature = Extra Failure Point", fontsize=13, fontweight="bold")
ax.legend(fontsize=10)

plt.tight_layout()
plt.savefig("05_drift_sensitivity.png", dpi=150, bbox_inches="tight")
plt.show()
print("Saved → 05_drift_sensitivity.png")

The result is exactly what you'd fear.

The Lean model's predictions don't change at all. Why would they? It doesn't use that feature, so it's completely immune to any problems with it.

But the Noisy model? Its predictions start to go haywire. As the bus_stop_age_yrs data becomes more and more wrong, the model's prediction error (the RMSE) steadily climbs. We've introduced a vulnerability for no good reason. We added a feature that provided almost no value, but we inherited all the risk that comes with it.

This is the hidden cost of the "kitchen-sink" approach. Your model might look slightly more accurate on a static test set, but you’ve made it dramatically more fragile in the real, messy world of production data.

So, the next time you're tempted to dump every feature you have into a model, take a step back. Ask yourself: Is this feature adding real, unique signal? Or is it just adding noise, redundancy, and another potential time bomb to my production system? Often, less really is more.