Detection benchmark
How well the free engine actually detects AI-component attacks — measured, not claimed. Every number below is produced by running the open-source engine over a labeled corpus of malicious and benign-but-tricky samples. Detection is deterministic (regex + AST, never an LLM), so you can reproduce these exact numbers yourself.
Engine 0.34.1 · ruleset 33 · offline corpus of 25 malicious + 16 benign samples.
Recall by attack technique
Fraction of malicious samples of each technique that the engine flags (malicious verdict or high/critical risk).
| Technique | Recall |
|---|---|
| Credential / secret exfiltration | 100% |
| Decode-and-execute droppers | 100% |
| Deserialize-and-execute | 100% |
| Email (SMTP) exfiltration | 100% |
| Hidden-Unicode smuggling | 100% |
| Install-time droppers | 100% |
| Lethal trifecta (read + net + inject) | 100% |
| MCP tool poisoning | 100% |
| Prompt injection | 100% |
| Startup (.pth) persistence | 100% |
| shell-decode-exec | 100% |
| typosquat-dropper | 100% |
Coverage: OWASP class × language
Malicious-sample counts per OWASP LLM/AST class and language. The language axis is honest about scope: deep semantic detection is Python, Node/TypeScript, and shell today.
| Class | manifest/text | node | python | shell |
|---|---|---|---|---|
| AST01 | 3 | 4 | 7 | 2 |
| AST02 | 0 | 2 | 2 | 0 |
| AST04 | 3 | 0 | 0 | 0 |
| AST05 | 0 | 0 | 2 | 0 |
Reproduce it yourself
This is the whole point of a deterministic engine: no LLM, no black box, no “trust us”. Clone the public repo and run the same harness — you get the same numbers, byte for byte.
git clone https://github.com/pezhik/skilltotal
cd skilltotal
pip install -e ".[dev]"
python tests/manual_eval/efficacy.pyThe labeled corpus lives in tests/eval_corpus/ (sanitized, non-functional attack samples). Want to check us on your own corpus, or against another tool? Point the harness at a folder of positive/ and negative/ samples and compare. We publish our numbers; anyone can publish theirs.
Methodology & honest caveats
- Deterministic. Detection is regex + AST, never an LLM, and the scanned code is never executed — so the result is identical on every run and every machine.
- Curated corpus. These are sanitized samples that reproduce real attack shapes across OWASP classes; the corpus is intentionally hard on the negative side (benign-but-tricky code that naive scanners over-flag). It is a floor we hold at 100% / 0 FP, not a claim about every attack in the wild.
- Precision is measured on adversarial negatives. A false positive counts if a benign sample gets a malicious verdict, a high/critical risk level, or a synthesized exfiltration combo — the strictest reasonable bar.
- Language scope is explicit. Deep semantic detection today is Python, Node/TypeScript, and shell; other languages appear in the coverage matrix with their real (often zero) counts rather than being hidden.
See how the engine works or run a scan.