Is tokenizers safe?
What tokenizers does
tokenizers is a Hugging Face library, with Python bindings over Rust, for training vocabularies and tokenizing text. It supports models such as BERT WordPiece and BPE. The PyPI metadata lists Nicolas Patry and Anthony Moi as authors, and the code lives in huggingface/tokenizers. It reads and writes tokenizer files on disk, and Tokenizer.from_pretrained loads a pretrained tokenizer from the Hugging Face Hub. Building from source needs a Rust toolchain, while PyPI ships prebuilt wheels. Before installing, prefer those wheels, and load pretrained tokenizers only from Hub repositories you know.
- Python filesystem read
What to do: Nothing here argues against installing it. Grant the capabilities it lists only if you expect the tool to need them.
tokenizers is a PyPI package analyzed by SkillTotal's deterministic static scanner. The scan found no malicious indicators. It can: filesystem read — capabilities are what the code can do, not a verdict on intent. Risk score 0/100 (low).
tokenizers 0.23.3
Get notified when this component changes
Change alerts aren't live yet. Leave your email and we'll let you know when they launch. We use it only for this and delete it on request.
Automated static-analysis result. It can contain false positives and false negatives, and is not a claim about the intent of tokenizers's authors. Report a false positive.
Behavioral traits
How this component maps to the CSA agentic threat model. Descriptive — it never affects the risk score.
Findings (1)
The component reads files from disk.
m.ParseFromString(open(filename, "rb").read())
with open(css_filename) as f:
Why it matters: Usually legitimate, but worth confirming it can't be steered into reading sensitive files.
Fix: Confirm which files are read and that paths cannot be influenced by untrusted input to reach sensitive locations.
Maintain tokenizers?
The free Claude Code plugin scans each npm, PyPI or MCP package with this report's open-source engine before the install command runs. It runs on your machine and needs no account.
Check your own component
Run the same evidence-backed scan on any MCP server, agent skill, or package.
Scan your own componentHow we determine this: deterministic static analysis (regex + AST), evidence-anchored, no code execution. Methodology →