TokenPad

Recipe · Tokens

Count tokens in JavaScript

Count OpenAI tokens in Node or the browser with gpt-tokenizer, loading only the encoding you need instead of the whole package.

The problem

You need an exact token count in a JavaScript or TypeScript codebase, either on a server before dispatching a request or in a browser to show a user what they are about to send.

The obvious import pulls in every encoding the package ships, which is megabytes you do not need.

Import one encoding, not the packagetypescript
// Importing the package root loads every encoding it ships.
// Import the single encoding you need instead.
import { encode, decode } from 'gpt-tokenizer/encoding/o200k_base';

export const countTokens = (text: string): number => encode(text).length;

console.log(countTokens('Summarise this ticket in two sentences.'));
Load it lazily in the browsertypescript
// The merge ranks are around a megabyte gzipped. Loading them on demand
// keeps them out of the initial bundle entirely.
let encoder: Promise<typeof import('gpt-tokenizer/encoding/o200k_base')> | null = null;

const getEncoder = () => {
  encoder ??= import('gpt-tokenizer/encoding/o200k_base');
  return encoder;
};

export async function countTokens(text: string): Promise<number> {
  const { encode } = await getEncoder();
  return encode(text).length;
}

// Optional: start the download during idle time so the first count is instant.
export function warm(): void {
  if ('requestIdleCallback' in window) {
    requestIdleCallback(() => void getEncoder(), { timeout: 2000 });
  }
}
Count a messages array with overheadtypescript
import { encode } from 'gpt-tokenizer/encoding/o200k_base';

interface Message {
  role: string;
  content: string;
}

const TOKENS_PER_MESSAGE = 3;
const TOKENS_PER_REPLY = 3;

export function countMessages(messages: Message[]): number {
  return messages.reduce(
    (total, m) => total + TOKENS_PER_MESSAGE + encode(m.role).length + encode(m.content).length,
    TOKENS_PER_REPLY,
  );
}

Why it is written this way

Why gpt-tokenizer rather than a WASM binding

It is pure JavaScript, so it works in Node, the browser and edge runtimes with no build step and no binary. The WASM alternatives are comparable in size and add a loading path that breaks in some serverless environments.

The encoding is what matters

o200k_base for GPT-4o and everything after it, cl100k_base for GPT-4 and GPT-3.5. The difference on the same text is commonly 10–20%, and larger on code and non-English content.

Server-side, import at module scope

The lazy pattern above is for browsers, where the megabyte matters. On a server, import normally at the top of the file so the ranks are parsed once at startup rather than per request.

What breaks the naive version

  • Importing from the package root. It pulls in every encoding and will dominate your bundle.
  • Calling encode inside a hot loop on large text. Encode once and reuse the array if you need both the count and the tokens.
  • Using it for Claude or Gemini and presenting the result as exact. Neither publishes a browser-capable tokenizer, so those numbers are estimates.

Check your numbers