Sectum AI vs NVIDIA Garak
TL;DR. Both products are Apache 2.0 open source and both target AI security, but they operate at different levels. Garak (NVIDIA/garak) is an LLM vulnerability scanner — 50+ probe modules covering prompt injection, jailbreaks, training-data extraction, hallucinations, encoding bypasses — that runs per-model. Sectum AI is a multi-tenant infrastructure verifier with a marker substrate, runs cross-tenant probes across 13 surfaces, and produces a tamper-evident, control-mapped audit pack. Both are useful, both are open source, and most prod AI teams should run both.
The two products
NVIDIA Garak (garak.ai, NVIDIA/garak)
Category: open-source LLM vulnerability scanner.
License: Apache 2.0. 6.9k GitHub stars.
Distribution: Python CLI (pip install garak). NVIDIA-maintained with active 2026 releases.
2026 capability (NVIDIA Garak Explained, Frank’s World 2026-02-03, Help Net Security):
- 50+ probe modules — prompt injection, jailbreaks, training-data extraction, package hallucination, toxic content, encoding bypasses, malware-gen attempts, XSS via LLM output.
- 28 detector types for data leakage, toxicity, policy violations.
- 23 generator backends — OpenAI, Anthropic, Hugging Face, local models via plugin architecture.
- 2026 release cadence: v0.15.0 (May 2026) added multi-turn GOAT probe + Agent-breaker probe + ModernBERT refusal detector + NeMo Guardrails server support; v0.14.1 (April 2026) added bootstrap confidence intervals for attack-success rates + WebSocket generator; v0.14.0 (February 2026) added JSON config support alongside YAML.
- JSONL + HTML reporting for CI integration.
Pricing: free / OSS.
Buyer: AI engineers, security teams running pre-prod LLM red-team scans in CI pipelines.
Sectum AI (sectum.ai)
Category: multi-tenant AI verification.
License: Apache 2.0 for the substrate, attack catalog, adapters, evidence chain, and sectum-ai verify. The evidence layer in the OSS produces the same artifacts the hosted Sectum Cloud does — by design.
Method: marker substrate. Provisions synthetic tenants on the customer’s AI stack, plants cryptographic canary markers (HARD_CANARY / ENTITY_CANARY / SECRET_CANARY), records a hashed ground-truth manifest, runs 11 cross-tenant probe classes across 13 surfaces, produces a tamper-evident evidence pack with a cryptographic chain of custody (RFC 3161 TSA + Sigstore Rekor + in-toto envelope).
For: CISOs, DPOs, and audit firms working on multi-tenant AI products. The flagship engagement is a GDPR Article 17 erasure attestation. See pricing.
The categorical difference: per-model probes vs. per-tenant-boundary attestation
| Garak | Sectum AI | |
|---|---|---|
| Unit of analysis | A single LLM endpoint at a time | A multi-tenant AI infrastructure |
| Method | Per-probe attack against the model + per-detector pass/fail | Marker substrate + manifest-grounded layered detection |
| Detection determinism | Detector heuristics + LLM-as-judge variants | Manifest-grounded: confirmed findings have zero false positives by construction |
| Output | JSONL + HTML report per scan | Tamper-evident audit pack (RFC 3161 + Rekor + in-toto + PDF + JSON) |
| Verification | Re-run Garak; trust the report | sectum-ai verify <pack> — third-party-verifiable without Sectum AI installed |
| Surfaces | Per-LLM probe surface | 13 surfaces (vector DB, RAG, caches, agents, MCP, fine-tunes, eval sets, search indexes, tracing, etc.) |
| Multi-tenant focus | Not specifically | The category |
| Flagship engagement | — | GDPR Art. 17 erasure attestation |
Both projects are healthy, well-maintained Apache 2.0 OSS. Garak’s strength is breadth and depth on per-model probing. Sectum AI’s strength is depth on the multi-tenant boundary across surfaces, with auditor-grade evidence.
Surface coverage
| Surface | Garak | Sectum AI |
|---|---|---|
| LLM endpoint (any of 23 backends) | ✓ (the primary unit) | ✓ (one of the probe surfaces) |
| Vector DB direct (cross-tenant integrity) | — | ✓ (Pinecone, pgvector, Weaviate, Chroma live adapters) |
| Semantic cache | — | ✓ (Class 4 + live Redis adapter) |
| KV cache (timing side channel) | — | ✓ (Class 5 — statistical Cohen’s d effect-size test) |
| Embedding inversion across tenants | — | ✓ (Class 6) |
| Agent tools / MCP | v0.15.0 added Agent-breaker probe (per-tool red-team) | ✓ (Class 7 — cross-tenant MCP confused-deputy + token passthrough) |
| Persistent agent memory | — | ✓ (Class 8) |
| LoRA / fine-tune cross-tenant influence | — | ✓ (Class 9) |
| Multi-turn benign extraction | ✓ (multi-turn GOAT probe v0.15.0) | ✓ (Class 10 — Silent Leaks / IKEA-style) |
| RAG poisoning | — (not a Garak focus) | ✓ (Class 3) |
| GDPR Article 17 erasure verification | — | ✓ (Class 11 — the Erasure Attestation engagement) |
| Observability backends (Langfuse / LangSmith / Phoenix) | — | ✓ (live adapters) |
| NeMo Guardrails server | v0.15.0 added integration | — (not a Sectum AI surface) |
| System-prompt-extraction probe | v0.15.0 added | — (not a Sectum AI focus) |
Garak owns prompt-level breadth across many backends. Sectum AI owns multi-tenant breadth across many surfaces. The two coverages run perpendicular and compound.
Evidence model
Both produce machine-readable output. The shapes differ:
Garak’s output:
- JSONL per probe run (machine-readable).
- HTML report (human-readable).
- Per-probe success-rate + (v0.14.1+) bootstrap confidence intervals.
- Re-runnable from saved configs.
Sectum AI’s output:
- Canonicalized run → SHA-256 digest.
- Digest timestamped by an RFC 3161 Time-Stamp Authority (a third-party TSA that attests the digest existed at this time).
- Digest recorded in a Sigstore Rekor transparency log entry with an inclusion proof.
- All wrapped in an in-toto attestation envelope.
- Bundled with the ground-truth manifest hash so the test conditions are provable after the fact.
sectum-ai verify <pack>validates the chain end-to-end, exits4and prints[FAIL]lines on any tampering — without requiring a Sectum AI installation.
The shapes serve different audiences. Garak’s report serves a security engineer reading findings; Sectum AI’s pack serves an auditor or DPO needing cryptographic chain of custody.
When to use Garak
- You’re running a single LLM endpoint and want broad coverage of per-model vulnerabilities (prompt injection, jailbreaks, training-data extraction, hallucinations, encoding bypasses).
- You want a tool that runs in CI on every commit / nightly build and produces actionable findings.
- You have multiple LLM backends (OpenAI + Anthropic + local) and want consistent probe coverage across them.
- You’re building on the NVIDIA AI stack (NeMo) and want the integrated experience.
- You want a strong open-source community (6.9k stars, NVIDIA maintainership, active releases).
When to use Sectum AI
- You operate a multi-tenant AI product and need to verify the tenant boundary holds across the 13-surface AI stack — not just the LLM endpoint.
- You need auditor-acceptable, tamper-evident evidence — a pack with a cryptographic chain of custody, not a JSONL report.
- You’re facing a GDPR Article 17 erasure obligation on a churned tenant.
- You need per-finding control mappings (OWASP LLM08:2025 / ATLAS / NIST AI RMF) in the evidence pack for audit or DPO use.
- You want manifest-grounded zero false positives on confirmed findings (the marker substrate guarantees this by construction).
Using both
The strongest AI security posture for a multi-tenant AI product runs both:
- Garak in CI / nightly — fast, broad LLM red-team across every LLM backend you call. Catches per-model vulnerabilities as they regress.
- Sectum AI on a release cadence and at every audit / Article 17 cycle — multi-tenant verification + tamper-evident attestation.
Both products being Apache 2.0 OSS means there’s no commercial lock-in either way. They live happily in the same pipeline; Garak’s JSONL output and Sectum AI’s audit pack address different consumers (engineer + auditor).
Honest positioning
Garak is the open-source standard for LLM vulnerability scanning. The 6.9k stars and NVIDIA backing speak for themselves. Sectum AI is not in that category — it focuses on multi-tenant verification and auditor-grade evidence, where Garak doesn’t compete. Most AI security programs at multi-tenant SaaS companies will benefit from running both.
Pricing
- Garak — free, Apache 2.0. Available on GitHub.
- Open Sectum (OSS) — free, Apache 2.0. Available on GitHub.
- Sectum Cloud — see pricing.