Is tokenizers safe?

The component reads files from disk.

m.ParseFromString(open(filename, "rb").read())

with open(filename, "r") as f:

with open(self._model, "r") as model_f:

with open(args.input_file, "r") as f:

with open(args.input_file, "r", encoding="utf-8-sig") as f:

m.ParseFromString(open(filename, "rb").read())

with open(css_filename) as f:

Why it matters: Usually legitimate, but worth confirming it can't be steered into reading sensitive files.

Fix: Confirm which files are read and that paths cannot be influenced by untrusted input to reach sensitive locations.

The component writes or deletes files on disk.

with open(args.vocab_output_path, "w") as vocab_f:

with open(args.merges_output_path, "w") as merges_f:

remove(args.model)

os.remove(f"{args.model_prefix}.model")

os.remove(f"{args.model_prefix}.vocab")

Why it matters: Usually legitimate, but worth confirming the paths can't be controlled by untrusted input.

Fix: Confirm which files are written/deleted and that paths cannot be influenced by untrusted input.

The component makes outbound network requests.

from requests import get

response = get(args.model, allow_redirects=True)

Why it matters: Usually legitimate, but confirm the destinations are expected and no sensitive data leaves.

Fix: Confirm the destination hosts are expected and that no sensitive data is sent off-host.