PatchTST: Transformers that Actually Work for Time Series

The Transformer Time Series Problem

Vanilla Transformers are notoriously terrible at raw time series forecasting. Unlike NLP, where a single word contains rich semantic meaning, a single time-series data point (e.g., a minute's closing price) contains very little meaning on its own. It's mostly noise.

Introducing PatchTST

We implemented PatchTST. Instead of feeding individual time steps into the Transformer as tokens, we group the time series into overlapping "patches" (e.g., blocks of 16 time steps).

# Conceptual Patching mechanism
def create_patches(time_series, patch_len, stride):
    patches = []
    for i in range(0, len(time_series) - patch_len + 1, stride):
        # A single token is now a patch of historical prices
        patches.append(time_series[i:i+patch_len])
    return torch.stack(patches)

This accomplishes two things:

It captures local semantic information (e.g., a localized trend or candlestick pattern) within a single token.
It drastically reduces the sequence length the attention mechanism has to process, cutting down memory usage.

This architecture significantly outperformed our baseline LSTM models on our volatility forecasting benchmarks.