Running Transformers in the Browser via WebAssembly

The Server Cost Problem

Server costs for our RAG application's text-embedding API were getting out of hand. Every keystroke from a user triggered an API call to embed their query, leading to massive AWS bills for our GPU instances.

WebAssembly and ONNX

We migrated the embedding generation entirely to the client side. We utilized Hugging Face's Transformers.js and the ONNX runtime compiled to WebAssembly (Wasm). This allows us to run standard transformer models directly in the user's Chrome or Safari browser using their CPU/GPU.

import { pipeline, env } from '@xenova/transformers';

# Ensure we use WebAssembly backend
env.backends.onnx.wasm.numThreads = 4;

# Load a quantized embedding model directly in the browser
const generateEmbeddings = await pipeline(
    'feature-extraction', 
    'Supabase/gte-small',
    { quantized: true }
);

const output = await generateEmbeddings("Convert this query to a vector!", {
    pooling: 'mean',
    normalize: true,
});
console.log(output.data);

The initial bundle size was an issue (loading a 120MB model in the browser is a terrible UX), but after applying aggressive INT8 quantization, we got the payload down to 30MB. The embeddings are now generated locally with zero latency, reducing our server inference costs to absolute zero.