Mean Squared Error (MSE) in Machine Learning: Formula, RMSE, MAE, and R-Squared

What Does MSE Stand For in Machine Learning?

MSE stands for Mean Squared Error. It is a regression metric equal to the average of the squared differences between a model’s predicted values and the actual values, written as MSE = (1/n) Σ (yᵢ − ŷᵢ)². Lower is better, zero means every prediction was exact, and the result carries the squared units of whatever you are predicting. MSE ranges from 0 to infinity, with no upper bound.

MSE is both an evaluation metric and a training loss: it is a function gradient descent minimizes when you train a regression network. This guide covers the MSE formula with a worked example and Python code, when to prefer MSE over MAE or RMSE, and the relationship to R-squared. It also shows why the squaring step matters, when MSE misleads you, and what to use instead.

TL;DR: MSE, RMSE, MAE, and R-Squared at a Glance

What Is the MSE Formula?

The mean squared error formula is:

MSE = (1 / n) × Σⁿᵢ₌₁ (yᵢ − ŷᵢ)²

Where:

• yᵢ is the actual value from the dataset
• ŷᵢ is the predicted value from the model
• n is the number of observations
• Σ sums the squared error across all n observations

The squaring step ensures both over-predictions and under-predictions contribute positively, and it weights larger deviations more heavily than smaller ones. The result is non-negative, has the units of the target variable squared, and equals zero only when every prediction is exactly correct.

Worked example

Suppose a model predicts house prices for four houses, in thousands of dollars:

Sum of squared errors = 100 + 100 + 400 + 100 = 700. Divide by n = 4, and MSE = 175.

Two things stand out. First, the units are thousands-of-dollars squared, which is not a quantity anyone can reason about directly, so you would report RMSE = √175 ≈ 13.2 thousand dollars instead.

Second, the single 20-unit error contributed 400 of the 700 total, more than the other three errors combined. That is the squaring step doing exactly what it is designed to do, and it is also the reason a single bad outlier can dominate the score.

MSE, RMSE, and MAE Formulas Side by Side

All three metrics start from the same per-observation error, yᵢ − ŷᵢ. They differ only in what happens to that error before averaging.

MSE = (1 / n) × Σⁿᵢ₌₁ (yᵢ − ŷᵢ)²

RMSE = √MSE = √[(1 / n) × Σⁿᵢ₌₁ (yᵢ − ŷᵢ)²]

MAE = (1 / n) × Σⁿᵢ₌₁ |yᵢ − ŷᵢ|

Applied to the four houses from the worked example (errors of −10, 10, −20 and 10 thousand dollars):

RMSE (13.2) sits above MAE (12.5) because the single 20-unit miss is squared before averaging, so it carries more weight in RMSE than in MAE. If all four errors had been the same size, RMSE and MAE would be equal.

Why MSE Squares the Errors

MSE answers one question: on average, how far off is this model, with big misses counting for disproportionately more than small ones? You could average the errors in two other ways, so the squaring step needs a reason.

Taking the raw average of (y − ŷ) fails immediately: over-predictions and under-predictions cancel, so a model that is wildly wrong in both directions can score zero. Taking absolute values fixes the cancellation and gives you MAE, which is a perfectly good metric.

Squaring does something MAE does not: it makes the error surface smooth and differentiable everywhere, which is what lets gradient descent optimize it directly. It also encodes a specific opinion that one 10-unit error is worse than ten 1-unit errors.

That opinion is a modelling choice. When it matches reality, MSE is the right metric. When your large errors come from measurement noise rather than from model failure, it is the wrong one, and the sections below cover what to use instead.

Figure 1: Mean Squared Error Calculation showing Actual vs Predicted values across five data points

How MSE Differs from MAE and RMSE

MAE (Mean Absolute Error) takes the absolute value of each error instead of squaring it. That keeps the metric in the original units and treats all errors linearly, so a single 10-unit error contributes the same as ten 1-unit errors. MAE is therefore more robust to outliers but harder to optimize with gradient methods because it is not differentiable at zero.

RMSE (Root Mean Squared Error) is the square root of MSE. It rescales MSE back into the original units of the target, which makes it easier to communicate. RMSE and MSE rank models identically (lower MSE always means lower RMSE), so they are interchangeable for model selection.

RMSE vs MSE: Which Should You Report?

Short answer: optimize MSE, report RMSE.

They contain identical information. RMSE is defined as √MSE, and because the square root is monotonic, any model that beats another on MSE also beats it on RMSE. There is no scenario where the two disagree about which model is better. The choice between them is entirely about what a number means to whoever reads it.

In the worked example above, MSE = 175 “thousand-dollars squared”, which means nothing to anyone. RMSE ≈ 13.2 thousand dollars puts the model’s error on the scale of about thirteen thousand dollars, which a product manager can act on.

Two details people get wrong. First, RMSE is not the average error, and it is not MAE. Because of the squaring inside, RMSE is always greater than or equal to MAE, and the gap between them widens as your errors become more uneven.

Comparing RMSE against MAE is a quick diagnostic: if RMSE is far above MAE, your error is concentrated in a few large misses rather than spread evenly. Second, taking the root does not make RMSE robust to outliers. It compresses the scale but the squaring already happened, so a single dominant outlier still drives the number.

For training, stick with MSE. The extra square root adds computation and changes the gradient scale without changing which minimum you reach.

MSE as a Loss Function: PyTorch and TensorFlow

PyTorch and TensorFlow both ship MSE as a built-in regression loss. Both snippets below reuse the four house prices from the worked example, so each prints the same 175.0.

import torch
import torch.nn as nn

criterion = nn.MSELoss()          # reduction="mean" by default
predictions = torch.tensor([210.0, 340.0, 295.0, 400.0])
targets     = torch.tensor([200.0, 350.0, 275.0, 410.0])

loss = criterion(predictions, targets)
print(loss.item())                # 175.0, matching the worked example above

import tensorflow as tf

mse = tf.keras.losses.MeanSquaredError()
print(mse([200.0, 350.0, 275.0, 410.0], [210.0, 340.0, 295.0, 400.0]).numpy())  # 175.0

# Or as a compile-time argument on a Keras model
model = tf.keras.Sequential([tf.keras.Input(shape=(1,)), tf.keras.layers.Dense(1)])
model.compile(optimizer="adam", loss="mse", metrics=["mae"])

Three things worth knowing when you use it as a loss rather than a report metric:

• The reduction argument matters. PyTorch’s nn.MSELoss defaults to reduction="mean", which averages over every element. Switching to reduction="sum" makes your effective learning rate scale with batch size, which can cause training instability that looks like a bad learning rate.
• Scale your targets. MSE gradients are proportional to the error magnitude, so an unscaled target in the tens of thousands produces enormous initial gradients. Standardize or normalize the target to keep those initial gradients small.
• Use Huber when the data is dirty. nn.HuberLoss and tf.keras.losses.Huber behave quadratically for small errors and linearly for large ones, which keeps MSE’s smooth gradients near the optimum while capping the influence of outliers.

Why MSE Matters in Machine Learning

Measuring Regression Model Accuracy

MSE provides a single, quantitative number that summarises how close predictions are to truth. Lower MSE means predictions are tighter around the actual values. Higher MSE means the model is off, sometimes systematically and sometimes only on a few high-leverage points.

Example use cases include:

• Stock price prediction. MSE measures how far model output is from realized prices. Squaring penalises blowups more than chronic small drift, which matters when a single bad day can wipe out a quarter.
• Sales forecasting. A retailer evaluates monthly forecasts with MSE to flag regions and seasons where prediction error is concentrated.
• Weather prediction. Temperature and rainfall forecasts can be scored with MSE against observed values before a model is released.

Why Gradient Descent Likes MSE

The training loop computes MSE on a mini-batch, takes the gradient with respect to model parameters, and updates the parameters with an optimizer like Adam or SGD. See the framework code above for the PyTorch and TensorFlow calls.

Three properties make MSE well behaved for gradient descent:

• Smoothness. The squared-error surface is differentiable everywhere, which keeps the gradient stable.
• Convexity in linear models. For linear regression, the MSE surface is convex, so gradient descent with a suitable learning rate converges to a global minimum.
• Strong penalty on outliers. The squared term keeps the model focused on large errors during training. This is helpful when large errors are costly and unhelpful when they are noise.

Insights MSE Provides About Model Performance

MSE alone tells you the average squared error. Pairing MSE with residual analysis reveals where the model struggles:

• A handful of large residuals dragging up MSE can point to outliers, leverage points, or missing features for a sub-population.
• A flat residual plot with high MSE points to underfitting.
• A residual plot that fans out with increasing predictions points to heteroscedasticity. One global MSE hides range-dependent error patterns in that case, so segmented metrics or residual plots are required to diagnose where the model is failing.

How to Calculate MSE: Step-By-Step

To compute MSE by hand:

1. Subtract the predicted value from the actual value for each observation.
2. Square each difference.
3. Sum the squared differences.
4. Divide by the number of observations.

Example:

• Actual values: [5, 7, 9]
• Predicted values: [6, 6, 10]
• Errors: [-1, 1, -1]
• Squared errors: [1, 1, 1]
• MSE: (1 + 1 + 1) / 3 = 1.0

Working Python Example with NumPy and Scikit-Learn

import numpy as np
from sklearn.metrics import mean_squared_error

y_actual = np.array([5, 7, 9])
y_predicted = np.array([6, 6, 10])

# Manual calculation
mse_manual = np.mean((y_actual - y_predicted) ** 2)

# Scikit-learn equivalent
mse_sklearn = mean_squared_error(y_actual, y_predicted)

rmse = np.sqrt(mse_sklearn)
# scikit-learn 1.4+ also ships root_mean_squared_error(y_actual, y_predicted)
print(f"MSE: {mse_manual:.4f}")
print(f"MSE (sklearn): {mse_sklearn:.4f}")
print(f"RMSE: {rmse:.4f}")

mean_squared_error is scikit-learn’s built-in implementation, and it rejects inputs of different lengths, empty arrays and NaN or infinite values before computing, so use it in production code rather than a hand-rolled version. On these three points the script prints 1.0000 for both MSE lines and for RMSE.

Common Pitfalls in MSE Calculation

• Outliers. A single 100-unit error contributes 10,000 to the sum of squares, which dwarfs ninety-nine 1-unit errors that total only 99. Inspect residuals before trusting MSE.
• Scale dependency. MSE is in the squared units of the target. Comparing MSE across models that predict different targets is meaningless without normalization.
• Train versus test split. Always report MSE on a held-out test set, not the training set, to detect overfitting.

What Is a Good MSE Value?

There is no universal threshold. The reason is in the units: MSE is measured in the squared units of your target variable, so its magnitude depends entirely on what you are predicting.

An MSE of 175 is excellent when predicting house prices in thousands of dollars and catastrophic when predicting a probability between 0 and 1. The number is not comparable across datasets, across target scales, or even across the same dataset after you change the units. MSE has no upper bound, so its range is 0 to infinity; that is another reason a raw MSE cannot be judged without a baseline.

What makes an MSE good or bad is always a comparison. Three that work:

1. Against the mean baseline. Compute the MSE of a model that ignores every feature and always predicts the training mean. Your model must beat it, and by a margin worth the complexity. This comparison is exactly what R-squared formalises: R² = 1 − (MSE / variance of the target), where that variance is the population form dividing by n, matching MSE’s own denominator. R² = 0 means you tied the mean predictor and negative R² means you lost to it.
2. Against the previous model. For an existing production system, compare against the incumbent’s MSE on the same held-out set. A 5% reduction that holds up on test data is a real result.
3. Against the cost of being wrong. Convert to RMSE, put it in target units, and ask whether an error of that size is acceptable to the business. If your demand forecast has an RMSE of 400 units and the warehouse holds 300, the model is not usable regardless of how the number compares to anything else.

The one absolute statement worth making: an MSE of exactly zero on training data is a warning sign. It points to a model that has memorised the training set or a feature that leaks the target.

How to Interpret MSE Results

High MSE

A high MSE indicates large prediction errors on average. Possible causes:

• Underfitting (model is too simple for the relationship)
• Inadequate feature engineering or missing predictors
• Data quality issues like mislabeled targets, missing values, or measurement noise

Low MSE

A low MSE indicates predictions are close to actual values. Always confirm the score on test data; a very low training MSE can hide overfitting. Cross-validation and a held-out test set give a more honest read.

Balancing MSE with RMSE and R-Squared

R-squared rescales error into a unitless fit score where 1 is perfect, 0 matches the mean predictor, and negative values mean the model is worse than the mean baseline. R-squared = 1 minus (MSE / variance of target). For a clear walkthrough see R-squared model accuracy.

RMSE rescales MSE back into the target units. A complete report includes all three, plus a residual plot for diagnostics.

Practical Applications of MSE

MSE in Forecasting and Time Series

Time-series models use MSE to track forecast error against realized values. A retail chain that forecasts monthly sales by region uses MSE to flag regions where forecast error has grown, which can signal missing seasonal or regional features. For drift over time see model vs data drift.

Pricing and Recommendation Models

E-commerce platforms use MSE to evaluate predicted optimal prices against actual customer behavior. Recommendation engines that predict ratings (Netflix-style 1-to-5 stars before they switched to thumbs) use MSE on the predicted rating vector against held-out ratings.

Computer Vision Regression Tasks

CNNs that regress bounding-box coordinates or pixel values use MSE on the coordinate vector. Object-detection losses like the Smooth L1 loss in Fast R-CNN combine MSE for small errors and MAE for large errors, which avoids exploding gradients while keeping smooth optimization.

Comparing Models Using MSE: Linear Regression vs Decision Trees vs Random Forests

A data scientist testing three regression models on housing-price prediction might see:

The Random Forest has the lowest MSE of the three on this dataset. Always confirm the ranking on a cross-validation split before deploying. Hyperparameter tuning (tree depth, learning rate, regularization strength) can lower MSE until it plateaus or test MSE starts to climb (overfitting signal).

MSE in Optimization Algorithms: Gradient Descent

Gradient descent uses MSE as the objective function for regression. For each mini-batch, the algorithm computes the gradient of MSE with respect to model parameters and steps in the negative-gradient direction. Over many iterations the parameters settle into a minimum.

Picture MSE as the height of a surface laid over every possible set of parameter values. Gradient descent is a ball that rolls downhill, and each step is an update to model parameters. With an appropriate learning rate, gradient descent on convex MSE surfaces (like linear regression) converges to the global minimum. Non-convex surfaces (like deep neural networks) have many local minima, and the optimizer settles into one of them.

MSE in Linear Regression

Ordinary least squares (OLS) linear regression picks the coefficients that minimize the residual sum of squares, Σ (yᵢ − ŷᵢ)². That sum is n times the MSE, and n is fixed for a given training set, so the coefficients that minimize one also minimize the other: OLS is the MSE minimizer for a linear model.

scikit-learn’s LinearRegression describes its fit the same way, as minimizing “the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation.” Because the MSE surface of a linear model is convex, gradient descent with a suitable learning rate reaches the same minimum MSE that the closed-form OLS solution gives.

MSE in Deep Learning Practice

For continuous-output deep networks, MSE is a default training loss. For image-to-image regression (super-resolution, denoising), MSE can be combined with perceptual losses, since pixel-wise MSE alone can produce blurry outputs. For tabular regression, MSE on its own can serve as the training loss.

Pro Tips for Using MSE Effectively

• Normalize features so that scales do not bias gradient updates. MSE is sensitive to feature scaling because parameters with larger feature scales also have larger gradients.
• Pair MSE with residual plots to catch heteroscedasticity, outliers, and systematic bias that the summary number hides.
• Combine MSE with domain knowledge. A model with low MSE that misses a known operational constraint (negative prices, missing-class predictions) is not deployable, even if the average error looks good.
• Use Huber loss when you need MSE-like smoothness for small errors and MAE-like robustness for large ones.

Advantages and Limitations of MSE

Advantages

• Penalizes larger errors heavily, which is the right behaviour when large errors are costly.
• Smooth and differentiable, which makes it well behaved for gradient descent.
• Convex in linear models, so gradient descent with a suitable learning rate reaches a global minimum.

Limitations

• Sensitive to outliers (a single outlier can dominate the metric).
• Squared units complicate direct interpretation (use RMSE to recover original units).
• Not the right metric for classification (use cross-entropy or F1, depending on the task; see F1 score).
• Not the right metric for highly skewed targets (consider log-transforming the target).

Where MSE Fits in LLM and Agent Evaluation

MSE is a numeric-output metric, which makes it a poor fit for LLM evaluation where outputs are free-form text. LLM evals instead use judge-based metrics such as groundedness, instruction adherence and toxicity, plus deterministic checks for things like schema conformance. See deterministic LLM evaluation metrics for a survey.

Future AGI has custom evals: write your own grading rule, choose an LLM judge or a deterministic check, map dataset columns to its inputs, set a pass/fail threshold, and run it as a CI gate. For true squared error over numbers an LLM extracts, write a Code eval that computes MSE and set the threshold you can tolerate.

The Future AGI Agent Learning Kit, an Apache-2.0 SDK for local-first testing, simulation, red teaming and optimization of AI agents, runs the same kind of checks from your own code. Future AGI’s built-in Numeric Similarity eval extracts the number from a model’s text answer and the reference, then scores the absolute or normalized difference between them, so you can check numeric answers that arrive inside sentences.

# Requires: pip install agent-learning-kit
from fi.evals import evaluate

result = evaluate(
    "numeric_similarity",
    expected_response="Forecast revenue: 2.4M USD for Q3.",
    output="The model forecasts 2.1M USD of revenue in Q3.",
)

print(result.score, result.reason)
# 0.8750000000000001 Numeric Diff: |2.1 - 2.4| = 0.300, Similarity: 0.875

In the open-source Future AGI platform, the Evaluations page lists each eval as one row with its eval type (Agent, LLM-as-Judge or Code), its output type and its tags. A custom eval you save there works like a built-in, so you can run it on a dataset, a trace or from the SDK by name.

Choosing Between MSE, MAE and Huber for Your Next Model

MSE is a default loss function for regression, in classical ML and in neural networks. It is smooth, well behaved for gradient descent, and the right call when large errors are disproportionately costly. Switch to MAE when outliers are noise rather than signal, Huber loss when you want a balance, and R-squared when you need a unitless score that translates across stakeholders. For classification and LLM outputs, MSE does not apply; use cross-entropy and LLM-as-a-judge metrics instead.

If part of your pipeline is an LLM that writes numbers in sentences, score those numbers with Future AGI’s Numeric Similarity eval instead of eyeballing them.