Test MLX Tokenizer and Chat-Template Contracts on a Cloud Mac

If MLX suddenly starts including role labels in its replies after you move the same conversation to a cloud Mac, the model weights may not be at fault. A different tokenizer version, a changed chat template, or special tokens added twice can all alter the actual input. Establish a stable message-to-token conversion before comparing generated responses, so you can distinguish input drift from sampling randomness.

Define the input contract to validate

This walkthrough checks inference preprocessing only. It does not load model weights or measure tokens/s. Prepare a Python virtual environment and a verified local model directory containing the tokenizer configuration, vocabulary, and chat template required by that model. Do not copy these files from another model of the same size.

Each validation record captures three layers: the original message array, the rendered template text, and the complete token ID sequence. Comparing token counts alone is not enough: two sequences of the same length can still express different content. An identical ordinary vocabulary does not guarantee identical role boundaries or end markers, either.

A chat template is part of the model’s input protocol, not a display style. Successful tokenization only confirms that conversion completed—not that the model received the correct role structure.

Save the following script as check_prompt.py and run it in an environment with transformers and any dependencies required by the tokenizer. First run python -m pip freeze > requirements.lock.txt to capture the environment. This list records package versions; it does not guarantee identical installation results across platforms.

Fix the local files and message fixtures

The script explicitly disables network downloads and custom remote code. If the model requires a custom implementation, review it first and establish a separate execution procedure. Do not temporarily enable code execution just to work around an error.

The fixtures include Chinese text, a newline, angle brackets, and a multi-turn conversation. Each is tested with and without a generation prefix. They use the common alternating user/assistant structure; add fixtures for the system role and tool calls only after confirming that the model supports them.

import argparse
import json
from pathlib import Path
from transformers import AutoTokenizer

parser = argparse.ArgumentParser()
parser.add_argument("model_dir")
parser.add_argument("baseline")
parser.add_argument("--record", action="store_true")
args = parser.parse_args()

tokenizer = AutoTokenizer.from_pretrained(
    args.model_dir,
    local_files_only=True,
    trust_remote_code=False,
)
cases = [
    [{"role": "user", "content": "只回答数字：2+3\n<note>保留原文</note>"}],
    [
        {"role": "user", "content": "记住代号：实验甲"},
        {"role": "assistant", "content": "已记录。"},
        {"role": "user", "content": "代号是什么？"},
    ],
]
rows = []
for messages in cases:
    for generation in (False, True):
        rendered = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=generation,
        )
        ids = tokenizer.apply_chat_template(
            messages,
            tokenize=True,
            add_generation_prompt=generation,
        )
        encoded = tokenizer.encode(
            rendered, add_special_tokens=False
        )
        if ids != encoded:
            raise SystemExit("Tokenization paths differ")
        rows.append({
            "messages": messages,
            "generation": generation,
            "rendered": rendered,
            "ids": ids,
        })

path = Path(args.baseline)
if args.record:
    with path.open("x", encoding="utf-8") as stream:
        json.dump(rows, stream, ensure_ascii=False, indent=2)
else:
    expected = json.loads(path.read_text(encoding="utf-8"))
    if rows != expected:
        raise SystemExit("Prompt contract changed")

Create a baseline, but do not approve it automatically

For the first run, use python check_prompt.py ./model ./prompt-baseline.json --record. Recording uses exclusive creation mode: if the file already exists, the operation fails. This prevents evidence from being overwritten during troubleshooting. For subsequent checks, omit --record. The check passes only if the command produces no output and exits with a zero status code.

Review the JSON item by item: Is the user content complete? Do the role separators match the model’s documentation? Are previous assistant replies followed by appropriate boundaries? Does generation mode prepare the right position for the next response? Some templates ignore the generation-prefix flag, so the two settings must not be required to produce different results.

Keep the baseline, script, dependency list, and checksums of the tokenizer files used together in version control. Use synthetic data only. Do not include real conversations, access tokens, or internal documents in a baseline intended for commit.

Trace drift through the actual MLX call chain

Even after the script passes, check whether the application entry point uses the same template and parameters. Some calling layers accept message arrays; others accept already-rendered strings. Passing a rendered string through a chat wrapper again can apply the template twice. Capture the synthetic fixture’s final token IDs immediately before the generation function and compare them against the baseline one by one.

Avoid adding special tokens twice

Templates often include start or end markers already, which is why the script uses add_special_tokens=False when encoding the rendered text. Do not manually prepend an extra BOS. Nor should you accept the result simply because the decoded text “looks normal”: decoding options that hide special tokens can conceal duplicate markers in the input.

When a mismatch occurs, compare rendered first, then ids. If the text differs, start with the template and message preprocessing. If the text matches but the IDs differ, check the vocabulary, added tokens, and tokenizer implementation. Only after confirming that the actual inputs match should you investigate sampling parameters and the model-weight version.

Turn failures into reviewable checks

When adding fixtures for empty content, trailing spaces, emoji, or long Chinese text, document what each one validates rather than collecting random strings. Long inputs also require an application-level context-budget check: this script neither reserves output space for generation nor validates the truncation strategy.

Before handoff, confirm four things: offline execution downloaded no files; a regular check does not modify the baseline; differences in either template text or tokens stop the task; and dependency upgrades require human review. When updating the baseline, save a separate candidate file and submit the diff for review. Do not automatically record a new baseline in the failure path.

What you retain during the rental period is a rerunnable set of input evidence, not a screenshot of a single reply. When moving to another cloud Mac, you can first demonstrate that the model received the same input, then investigate why its response changed.

Frequently asked questions