Detection benchmark
How well the free engine actually detects AI-component attacks — measured, not claimed. Every number below is produced by running the open-source engine over a labeled corpus of malicious and benign-but-tricky samples. Detection is deterministic (regex + AST, never an LLM), so you can reproduce these exact numbers yourself.
Engine 0.38.0 · ruleset 41 · offline corpus of 25 malicious + 16 benign samples.
Recall by attack technique
Fraction of malicious samples of each technique that the engine flags (malicious verdict or high/critical risk).
| Technique | Recall |
|---|---|
| Credential / secret exfiltration | 100% |
| Decode-and-execute droppers | 100% |
| Deserialize-and-execute | 100% |
| Email (SMTP) exfiltration | 100% |
| Hidden-Unicode smuggling | 100% |
| Install-time droppers | 100% |
| Lethal trifecta (read + net + inject) | 100% |
| MCP tool poisoning | 100% |
| Prompt injection | 100% |
| Startup (.pth) persistence | 100% |
| shell-decode-exec | 100% |
| typosquat-dropper | 100% |
Emergent-risk techniques
The most dangerous shapes are combinations — independent behaviors that co-occur to form an attack (read a secret and reach the network; an install hook and a decode-exec payload). SkillTotal synthesizes these as dedicated findings and maps each to the Cloud Security Alliance (CSA) agentic threat model — the same trait combinations shown on every report.
| Technique | Severity | CSA named risk |
|---|---|---|
| Credential exfiltration (sensitive-data read + network egress)MITRE ATLAS · Exfiltration | critical | Cross-request correlation blindness (read sensitive data then external comms) |
| Lethal trifecta (untrusted-instruction surface + file read + egress)MITRE ATLAS · Adversarial Perception Attacks, Exfiltration | high | Context manipulation escalated through tool egress |
| Install-time dropper (lifecycle hook + decode-exec/credential payload) | high | Supply-chain compromise via install-time execution or name confusion |
| Malicious-indicator convergence (>=2 distinct indicators) | high | Convergent malicious behavior across independent indicators |
See the full trait taxonomy & CSA / MAESTRO / MITRE ATLAS crosswalk in docs/trait-crosswalk.md.
Coverage: OWASP class × language
Malicious-sample counts per OWASP LLM/AST class and language. The language axis is honest about scope: deep semantic detection is Python, Node/TypeScript, and shell today.
| Class | manifest/text | node | python | shell |
|---|---|---|---|---|
| AST01 | 3 | 4 | 7 | 2 |
| AST02 | 0 | 2 | 2 | 0 |
| AST04 | 3 | 0 | 0 | 0 |
| AST05 | 0 | 0 | 2 | 0 |
Reproduce it yourself
This is the whole point of a deterministic engine: no LLM, no black box, no “trust us”. Clone the public repo and run the same harness — you get the same numbers, byte for byte.
git clone https://github.com/pezhik/skilltotal
cd skilltotal
pip install -e ".[dev]"
python tests/manual_eval/efficacy.pyThe labeled corpus lives in tests/eval_corpus/ (sanitized, non-functional attack samples). Want to check us on your own corpus, or against another tool? Point the harness at a folder of positive/ and negative/ samples and compare. We publish our numbers; anyone can publish theirs.
Methodology & honest caveats
- Deterministic. Detection is regex + AST, never an LLM, and the scanned code is never executed — so the result is identical on every run and every machine.
- Curated corpus. These are sanitized samples that reproduce real attack shapes across OWASP classes; the corpus is intentionally hard on the negative side (benign-but-tricky code that naive scanners over-flag). It is a floor we hold at 100% / 0 FP, not a claim about every attack in the wild.
- Precision is measured on adversarial negatives. A false positive counts if a benign sample gets a malicious verdict, a high/critical risk level, or a synthesized exfiltration combo — the strictest reasonable bar.
- Language scope is explicit. Deep semantic detection today is Python, Node/TypeScript, and shell; other languages appear in the coverage matrix with their real (often zero) counts rather than being hidden.
See how the engine works or run a scan.