Validate MLX fine-tuning datasets on a cloud Mac

Before starting an MLX fine-tuning run on a cloud Mac, check your data. Once the training and validation sets are uploaded, validation is the most useful first step: a line of incomplete JSON, an empty answer, or the same conversation appearing in both splits can make an experiment unsuitable for comparison. The following builds a repeatable check using Python’s standard library, without downloading a model or relying on the training environment.

Define the input contract instead of making the script guess

This article covers text-only conversations, with one object per line containing only a messages field. Each message contains only role and content. A single system message is allowed at the beginning; after that, messages must alternate between user and assistant, ending with an answer. Multimodal content, tool calls, and consecutive messages with the same role are outside this contract.

These are acceptance rules for this project, not a claim that every MLX training tool accepts only this structure. Before running the check, verify the data interface used by your current tool. If the source data uses different fields, write an explicit conversion step rather than letting the validator silently discard fields.

{"messages":[{"role":"user","content":"解释训练集与验证集的区别。"},{"role":"assistant","content":"训练集用于更新参数，验证集用于观察未参与更新的数据表现。"}]}

Save the two splits as train.jsonl and valid.jsonl, encoded as UTF-8 without a BOM. Blank lines count as errors; a final newline at the end of the file does not count as an extra blank line. Keep the original files read-only and save revisions separately so that data changes remain traceable.

Catch format errors line by line with the standard library

First, run python3 --version to confirm that the interpreter is available, then save the following code as check_dataset.py. The per-line limit is 256 KiB. This is the project’s input safety threshold, not the model’s context capacity. The script uses bounded binary reads so that an oversized line is not loaded into memory in full before being checked.

import hashlib
import json
import sys

LIMIT = 256 * 1024
seen = {}


def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("duplicate object key")
        result[key] = value
    return result


def reject_constant(value):
    raise ValueError("nonstandard JSON constant")


def validate(obj):
    if not isinstance(obj, dict) or set(obj) != {"messages"}:
        raise ValueError("expected messages-only object")
    messages = obj["messages"]
    if not isinstance(messages, list):
        raise ValueError("messages must be an array")
    roles = []
    for message in messages:
        if not isinstance(message, dict):
            raise ValueError("message must be an object")
        if set(message) != {"role", "content"}:
            raise ValueError("unexpected message fields")
        role = message["role"]
        content = message["content"]
        if not isinstance(role, str):
            raise ValueError("role must be text")
        if not isinstance(content, str) or not content.strip():
            raise ValueError("content must be nonempty text")
        roles.append(role)
    body = roles[1:] if roles[:1] == ["system"] else roles
    if len(body) < 2 or len(body) % 2:
        raise ValueError("expected complete conversation pairs")
    if body != ["user", "assistant"] * (len(body) // 2):
        raise ValueError("invalid role order")
    return messages


def inspect(path):
    count = 0
    with open(path, "rb") as stream:
        while True:
            raw = stream.readline(LIMIT + 1)
            if not raw:
                break
            count += 1
            try:
                if len(raw) > LIMIT:
                    raise ValueError("line exceeds byte limit")
                obj = json.loads(
                    raw.decode("utf-8"),
                    object_pairs_hook=unique_object,
                    parse_constant=reject_constant,
                )
                messages = validate(obj)
                canonical = json.dumps(
                    messages, ensure_ascii=False,
                    sort_keys=True, separators=(",", ":"),
                ).encode("utf-8")
                digest = hashlib.sha256(canonical).hexdigest()
                if digest in seen:
                    raise ValueError(
                        "duplicate conversation; first at " + seen[digest]
                    )
                seen[digest] = f"{path}:{count}"
            except (ValueError, RecursionError) as error:
                raise ValueError(f"{path}:{count}: {error}") from error
    if count == 0:
        raise ValueError(f"{path}: empty dataset")
    return count


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: check_dataset.py TRAIN VALID")
    try:
        counts = [inspect(path) for path in sys.argv[1:]]
    except (OSError, ValueError) as error:
        print(str(error), file=sys.stderr)
        raise SystemExit(1)
    print(json.dumps({"train": counts[0], "valid": counts[1]}))

Check for duplicates across both splits

Pass the training set first and the validation set second. The script shares a single hash index across both files, so duplicates within either split and duplicates across splits both cause the check to fail, with a reference to the first occurrence. It stops at the first error; fix that error and rerun the check. It does not print the full conversation.

python3 check_dataset.py train.jsonl valid.jsonl

Before calculating the hash, the script sorts object keys and removes JSON formatting differences, while preserving message order and the characters in the message text. Changing indentation or key order will not bypass detection; adding spaces to the message text may produce a different hash. The script does not normalize case or rewrite Unicode on its own, to avoid changing code, formulas, or the meaning of the original text.

Exact deduplication does not eliminate data leakage. Different question-and-answer pairs derived from the same document, or examples with the same question and a reworded answer, can still leak across splits.

If source document IDs are available, group and split by document before generating conversations. Without source IDs, at least add a duplicate-question check and a manual spot check. Passing the hash check is not proof that the validation set is independent. The hash index grows with the number of examples; very large datasets should use a disk-backed index instead.

Validate token lengths separately

The byte limit only guards against abnormal input; it cannot be converted into a fixed token count. Chinese text, code, and emoji have different encoding-to-tokenization ratios, and the chat template adds role markers and other special tokens.

For length checks, use the tokenizer supplied with the target model. Process each complete conversation with the chat template actually used by the training tool, then count the tokens. Record the model revision identifier, template, maximum sequence length, and number of over-limit examples. Do not just tokenize each content field separately and add the counts.

Explicitly decide whether over-limit examples will be rejected, split, or truncated. If you truncate, check whether the entire answer has been cut off and whether the loss mask still covers valid answer tokens. Reject any example with zero effective training tokens. No universal truncation command is provided here because templates and masking behavior differ across training interfaces.

Test the validator with invalid examples before connecting it to training

Start by copying an example to create four invalid cases: an empty answer, incorrect role order, duplicate object keys, and a conversation duplicated across files. Run each case separately and confirm that it returns exit code 1 and identifies the file and line number. Also test the success path with two distinct, valid conversations, so a validator that always reports errors does not go unnoticed.

When integrating the check into an experiment script, connect subsequent training steps with && or an explicit exit-code check. Training must not continue after validation fails. Once the data is finalized, save the SHA-256 hashes of both files, the example counts, the length-check results, and the validator version. This training run must read only that validated copy.

The goal is not a vague statement that “the data looks fine,” but reproducible evidence: identical input produces the same result, errors can be located, the split has a defensible basis, and length handling is documented. Complete those checks before spending time on training.

Frequently asked questions