Sectum AI vs DeepTeam

TL;DR. DeepTeam is an open-source LLM red-teaming framework with a specific check — CrossContextRetrieval(types=["tenant", "user"]) — that targets cross-tenant retrieval leakage with a single prompt and an LLM-as-judge. Sectum AI is a different category: a multi-tenant infrastructure verifier that provisions synthetic tenants, plants cryptographic canary markers, runs a catalog of cross-tenant probes across 13 surfaces, and produces a tamper-evident, control-mapped evidence pack an auditor accepts. Both are useful and answer different questions.

The two products

DeepTeam (trydeepteam.com)

Category: open-source LLM red-teaming framework, built by Confident AI on top of their deepeval library.

License: Apache 2.0.

Distribution: an OSS CLI + library that runs locally, plus a commercial Confident AI platform (paid SaaS) that aggregates risk assessments and produces downloadable PDF reports.

DeepTeam simulates attacks against an LLM system — jailbreaks, prompt injection, multi-turn exploitation, PII probes — across 20+ single- and multi-turn attack patterns to surface 50+ vulnerabilities. It maps findings to frameworks including OWASP Top 10, OWASP_ASI_2026, NIST, MITRE, Aegis, and BeaverTails.

The specific check that overlaps with Sectum AI is CrossContextRetrieval(types=["tenant", "user"]): a single targeted prompt that asks the model whether it surfaces foreign-tenant or foreign-user context, scored by an LLM-as-judge as a binary outcome.

Sectum AI (sectum.ai)

Category: multi-tenant AI verification. Not LLM red-team, not a runtime guardrail, not GRC, not data-subject-rights workflow. The deliverable is auditor-acceptable, tamper-evident evidence that the multi-tenant boundary holds across every surface of the AI stack.

License: the marker substrate, attack catalog, adapters, evidence chain, and the sectum-ai verify command are Apache 2.0. The hosted Sectum Cloud is commercial.

Distribution: a sectum-ai CLI you run inside your stack (BYOC), a marker substrate SDK, and Sectum Cloud (hosted attestation, scheduled runs, managed audit-pack delivery). The evidence layer in the OSS produces the same artifacts the hosted product does — by design.

Sectum AI’s method is a marker substrate: it provisions synthetic tenants on the customer’s AI stack (Acme, Globex, Initech, Hooli by default), seeds them with realistic corpora and three classes of cryptographic canary markers, records a hashed ground-truth manifest, and runs a catalog of cross-tenant probes from each tenant’s session. Detection is layered: exact canary scan first (zero false positives), then semantic similarity, then a calibrated LLM judge — and every confirmed finding is traceable to a specific marker the manifest records.

The categorical difference

DeepTeam and Sectum AI operate at different units of analysis:

The unit-of-analysis difference shows up everywhere. A DeepTeam CrossContextRetrieval finding means one prompt’s judge said the model returned foreign context. A Sectum AI confirmed finding means one specific cryptographic marker, owned by a specific tenant and recorded in a hashed manifest, was observed in a specific other tenant’s session on a specific surface, at a specific time, in a run whose digest is timestamped by an RFC 3161 authority and logged in Sigstore Rekor. Those are different evidence shapes.

Method, in detail

DeepTeam’s CrossContextRetrieval

CrossContextRetrieval(types=["tenant", "user"]) is one of DeepTeam’s vulnerability detectors. The flow:

1. Generate an adversarial prompt designed to elicit cross-tenant or cross-user context.
2. Run the prompt against the customer’s LLM endpoint.
3. Pass the response to an LLM-as-judge.
4. Judge returns a binary verdict: did the response reveal foreign-context information?

This is fast (one prompt + one judge call), flexible (works against any LLM endpoint), and judgement-based. It does not depend on the customer’s data; the adversarial prompt is constructed from a generic template.

The trade-offs are inherent to the design: the judge can hallucinate, the prompt may not reach the surfaces where leakage actually happens (a stale vector index, a semantic cache, a tracing pipeline), and the binary verdict is not an attestation an auditor accepts as proof. It is a red-team finding, not a control attestation.

Sectum AI’s marker substrate

Sectum AI’s approach is a provisioned simulator. The workflow:

1. sectum-ai seed provisions synthetic tenants (default scenario: Acme, Globex, Initech, Hooli) and generates realistic corpora — HR records, sales pipelines, contracts, meeting notes — with deliberately overlapping organic entities (a shared vendor, a shared person name, shared compliance terms) so that the conditions of the Retrieval Pivot research (95.4% benign-query leakage on a shared vector index) are reproduced.

2. Three classes of canary markers are planted in document bodies, metadata, titles, and tags:

   • HARD_CANARY — a high-entropy unique token (SECTUM-CANARY-{base32(16 bytes)}). Detected by exact substring/regex match — zero false positives.
   • ENTITY_CANARY — a unique synthetic entity (fabricated person, codename, account number) owned by exactly one tenant. Detected by semantic similarity + exact match.
   • SECRET_CANARY — a fake-but-plausible secret (API-key-shaped, fake-SSN-shaped). Detected by exact + format detector.

3. A hashed ground-truth manifest records every marker — owner tenant, marker type, plaintext, planted locations — and is itself hashed into the run’s evidence chain. The test conditions are provable after the fact.

4. The probe catalog runs cross-tenant probes from each tenant’s session. The eleven implemented classes include the flagship rag-entity-bleed (the Retrieval Pivot — benign queries leaking across a shared index), tenant-boundary-fetch (direct ID-based BOLA), semantic-cache-contamination, kv-cache-timing (a statistical side-channel test), embedding-inversion, agent-tool-hijack (MCP confused-deputy + token passthrough), memory-contamination, lora-cross-tenant, ikea-extraction, rag-poisoning, and gdpr-erasure-verification (the Erasure Attestation engagement).

5. The detection pipeline is layered, cheapest-and-most-certain first: exact canary scan, then semantic similarity, then a calibrated judge. A confirmed finding is traceable to a manifest marker; an unverified candidate (a judge-only flag with no manifest tie) is recorded but excluded from the headline count. The zero-false-positive property holds for confirmed findings by construction.

6. The evidence chain canonicalizes the run, hashes it (SHA-256), submits the digest to an RFC 3161 Time-Stamp Authority, records it in a Sigstore Rekor transparency log entry, and bundles the run, the manifest hash, the TSA token, the Rekor inclusion proof, the in-toto attestation envelope, and a human-readable audit PDF into the evidence pack. The pack is independently verifiable: sectum-ai verify <pack> recomputes everything and reports VERIFIED or [FAIL] with reasons.

The cost is the up-front substrate provisioning (sectum-ai seed) and the broader integration surface (an adapter per backend family). The payoff is auditor-grade evidence with a chain of custody.

Surface coverage

DeepTeam’s CrossContextRetrieval is one prompt against one surface. Sectum AI’s catalog is 11 probe classes against 13 surfaces with live adapters for vector stores, caches, observability backends, agents, and MCP servers.

Evidence model

This evidence-model gap is why audit firms find Sectum AI useful. A SOC 2 auditor or a GDPR DPO asking “can you prove tenant A’s data didn’t reach tenant B?” is asking for an attestation, not a red-team report. DeepTeam answers “did our judge see leakage in this prompt?” — useful, but not the same shape as an attestation.

When to use DeepTeam

• You want a fast pre-prod red-team check that an LLM endpoint doesn’t return obviously-foreign context to a templated adversarial prompt.
• You’re already running DeepTeam for the broader vulnerability suite (jailbreaks, prompt injection, PII probes) and want CrossContextRetrieval as one more check in that suite.
• You need a PDF for an internal security review rather than an auditor or DPO engagement.
• Your AI surface is a single LLM endpoint — no vector DB, no caches, no agents, no MCP integrations.

When to use Sectum AI

• You operate a multi-tenant AI system and need to prove the tenant boundary holds across surfaces — vector DB, caches, agents, MCP servers, fine-tunes, eval sets, tracing pipelines.
• You have a GDPR Article 17 erasure obligation and need to attest that a churned tenant’s data has actually left every AI surface (for a DPO or regulator).
• You’re preparing for SOC 2 or ISO 27001 in a multi-tenant SaaS and the auditor will ask about logical-access / boundary-protection controls (SOC 2 CC6.x, ISO 27001 A.5.15 / A.8.3 / A.8.12) for the AI portion of the stack.
• You need independently-verifiable evidence — a pack a third party can verify without your vendor in the room.
• You want an open-source evidence layer — the same artifacts the hosted product produces.

Can you use both?

Yes — and often you should. The two are non-substitutable:

• DeepTeam in a CI pipeline gives you fast feedback on per-prompt LLM vulnerabilities (jailbreaks, prompt injection, the CrossContextRetrieval smoke test) at every commit.
• Sectum AI on a release cadence (or on-demand for a customer asking for an isolation attestation) gives you the deep, multi-surface, tamper-evident evidence the auditor or DPO needs.

DeepTeam in dev, Sectum AI at every audit cycle and every Article 17 ticket.

Pricing

• DeepTeam (OSS) — free, Apache 2.0. Available on GitHub.
• Confident AI platform — commercial; pricing not publicly listed.
• Open Sectum (OSS) — free, Apache 2.0. Available on GitHub.
• Sectum Cloud — see pricing.

References

• DeepTeam — homepage, docs, CrossContextRetrieval, GitHub, NIST AI RMF framework support, Apache 2.0 license.
• Sectum AI — GitHub, docs, attack catalog, evidence chain, sample evidence packs.
• Underlying research: OWASP LLM08:2025 — Vector and Embedding Weaknesses; Retrieval Pivot Attacks in Hybrid RAG (2026, 95.4% benign-query cross-tenant leakage on shared vector indexes); Silent Leaks (arXiv 2505.15420, 91% extraction via benign queries).