TokenPad

Recipe · Safety

Redact PII before sending

Strip structured identifiers from the assembled prompt before dispatch, and restore them in the response if you need the original values back.

The problem

Personal data and credentials reach an API you do not control, usually not through a breach but through a retrieved document or a debugging paste.

Redacting the user message is not enough — the surprises are almost always in the retrieved context or the conversation history.

Redact with a restorable maptypescript
const PATTERNS: [RegExp, string][] = [
  [/[\w.+-]+@[\w-]+\.[\w.-]+/g, 'EMAIL'],
  [/\b(?:sk|pk|ghp|xox[baprs])[-_][A-Za-z0-9_-]{16,}\b/g, 'API_KEY'],
  [/\b(?:\d[ -]?){13,19}\b/g, 'CARD'],
  [/\b(?:\d{1,3}\.){3}\d{1,3}\b/g, 'IP'],
  [/\b[A-Z]{2}\d{2}(?:[ ]?[A-Z0-9]{4}){3,7}\b/g, 'IBAN'],
];

export interface Redacted {
  text: string;
  /** Placeholder to original, so a response can be rehydrated. */
  map: Map<string, string>;
}

export function redact(input: string): Redacted {
  const map = new Map<string, string>();
  let counter = 0;
  let text = input;

  for (const [pattern, label] of PATTERNS) {
    text = text.replace(pattern, (match) => {
      const token = `[${label}_${++counter}]`;
      map.set(token, match);
      return token;
    });
  }

  return { text, map };
}

/** Put the real values back into a response that referenced placeholders. */
export function restore(text: string, map: Map<string, string>): string {
  let out = text;
  for (const [token, original] of map) out = out.replaceAll(token, original);
  return out;
}
Redact the assembled prompt, not the user messagetypescript
// Wrong: only the part the user typed.
const messages = [
  { role: 'system', content: systemPrompt },
  { role: 'user', content: redact(userInput).text },
];

// Right: everything that reaches the model, including retrieved context
// and history — which is where the surprises usually are.
const assembled = [systemPrompt, ...retrievedChunks, ...history, userInput].join('\n\n');
const { text, map } = redact(assembled);

const response = await complete(text);
const answer = restore(response, map);

Why it is written this way

Redact last, immediately before dispatch

Anything assembled after your redaction step is unredacted. Doing it at the boundary is the only placement that catches retrieved documents, templates and tool results.

This is a net, not a guarantee

Pattern matching catches structured identifiers reliably. It cannot catch a name in a sentence, an address in prose, or a medical detail described in words. Treat it as removing the obvious, never as a compliance control.

Redaction usually costs fewer tokens

Random identifiers are what tokenizers handle worst — an API key can fragment into thirty tokens where a placeholder is one or two. On identifier-heavy payloads the redacted version is meaningfully cheaper.

What breaks the naive version

  • The card pattern matches any long digit run, so order numbers get redacted too. That is the safer direction for the error to fall.
  • Restoring placeholders into content shown to a user who should not see the originals. Restore only where the recipient is entitled to them.
  • Assuming a provider promise not to train covers you. Retention, your own logs and your contractual obligations to customers are separate concerns.

Check your numbers