Security research

Ten places a deleted user still lives

Dmitry Maranik·October 17, 2026

A user asks to be forgotten. Your team runs the deletion, the row is gone from the database, the ticket is closed. In an AI stack, that is usually where the trouble starts. By the time the request arrives the user has been embedded, logged, cached, remembered, indexed, and backed up. The deletion code does not know about most of those copies. So when GDPR Article 17 asks whether the data is gone, the honest answer from most teams is that the delete ran, and they are not sure what it reached.

Delete is a verb with a scope

The clearest public example is small and recent. Langfuse, the LLM observability platform, lets you delete a user's traces by userId, which is the path its own data-deletion guidance points to for a data-subject request. That deletion cascades correctly: traces, their observations, their scores. It does not touch datasets, which teams build from real inputs and which can hold the same personal data. The docs implied the user was gone; one surface over, they weren't. The fix was a documentation note, now merged into Langfuse's official docs, saying so and telling you what else to delete.

The deletion worked where it was written and missed the places the data had been copied to. There was no bug in the delete. A feature made a second copy, and nobody went back and taught the erasure path about it. We see the same thing in cross-tenant bugs, where a check gets added to the main path and skipped on the read sibling next to it.

The ten hiding places

Here is where a deleted user's data survives in a typical multi-tenant AI stack, and why each one escapes the deletion path. This is the surface list behind the erasure check in our open-source catalog (Class 11).

The vector store. Embeddings of the user's documents live in a namespace or collection, often shared. Deletes are frequently soft: the vector is flagged and only physically removed at the next compaction. Orphaned collections from a renamed or migrated project keep serving queries nobody remembers creating.
Prompt and completion logs. Every request the user ever made was logged somewhere with its full input and output: an observability platform, an APM, a logs bucket. These stores are optimized for retention, not erasure, and their delete APIs usually operate on traces, not on people.
Fine-tuning datasets and adapters. If the user's content was exported into a training set, two copies now exist: the dataset file and the weights it produced. A deleted file does not un-train a model. A LoRA adapter that memorized a phrase will still complete it.
Eval golden sets and test fixtures. The best real examples get promoted into regression tests and quality benchmarks, then committed to a repo or an eval tool. They get copied because they are realistic, and realistic examples contain real people.
Semantic and application caches. A cache keyed on a prompt hash or an embedding does not know who asked. The cached answer, built from the user's data, keeps being served after the source is gone, sometimes to a different user whose question was merely similar.
Conversation and agent memory. Long-term memory exists to keep facts around across sessions. Unless someone built a "forget everything about this person" path, nothing in it expires on its own.
Backups and exports. Nightly snapshots of every store above, a CSV someone exported for an analysis, a dump taken before a migration. Regulators generally expect copies to be erased or put beyond use. Ops teams generally want backups immutable. You need a written answer for how those two meet, because an auditor will ask.
A provider's retention window. Inputs sent to a third-party model API may be retained by that provider for a period under its own policy, unless a zero-retention arrangement is in place. You cannot scan this from your side. The honest statement is "presumed retained for the documented window," not "deleted."
Tracing pipelines with embedded prompts. OpenTelemetry spans and their exporters often carry the prompt and the response as attributes, then fan out to several backends. Deleting from the first does nothing for the second and third.
Derived search indexes. A keyword or hybrid index built from the RAG corpus is a second copy with its own lifecycle. It gets rebuilt on a schedule. Removing one person from it happens only if someone asks.

Why this is a verification problem

None of the ten is a bug in the deletion code. In each case the system did what it was built to do, indexed or logged or cached the data, and the erasure path was never extended to reach that copy. Reading the deletion function will not find this, because the function is fine. What finds it is running the deletion and then asking each surface what it can still return about the user.

The check is the same one we use for tenant isolation, pointed at deletion instead. Plant markers for a synthetic subject on every surface you can reach, run your real deletion, then scan every surface for the markers again. Each surface gets one of three results: erased (there was a baseline and nothing comes back), residual (a marker survived, so you have a specific failure on a specific surface), or not covered (you could not check it, and you say so). That last rule is what keeps the result honest. A surface you did not scan is not covered, never erased, and a provider's retention window is not covered no matter what.

what a coverage table looks like (illustrative, not a measured run)

vector_db ERASED baseline 12 markers, 0 retrievable after deletion tracing ERASED traces by subject id: gone; content fingerprint: no hit eval_set RESIDUAL 2 markers still present in the golden set semantic_cache ERASED no cached answer surfaces the subject agent_memory RESIDUAL 1 memory entry still recalls the subject backup NOT_COVERED no per-subject erasure API on this backend subprocessor NOT_COVERED presumed retained for the provider's documented window

Three verdicts, one per surface. The two NOT_COVERED rows matter most. An auditor trusts a report that says what it could not check more than one that says everything is clean.

What to do with this on Monday

Write the list for your own stack, with the real product names in each slot. For each one, answer two questions: does our deletion reach it, and how would we know. If the answer to the second is "we would read the code," you do not know yet. The Langfuse gap was found by looking at what the deletion left behind, not by reading the deletion.

If you want the check automated, Sectum AI runs it: sectum-ai erasure plants markers for a synthetic tenant, re-scans after your deletion, and emits a per-surface verdict across the vector store, tracing, agent memory, semantic cache, model adapter, search index, eval set, and backups, signed into an evidence pack you can hand to a DPO or an auditor. A --subject mode verifies a real data-subject request by record id and by content fingerprint, with the fingerprints hashed in the attestation rather than stored. It is open source; point it at your own stack and see which rows come back red.

Explore the open source More research

Anchored on the public Langfuse documentation change (langfuse-docs PR #3064, merged 2026-09-17). The ten-surface list is the Class 11 erasure catalog in the Sectum AI open-source specification. The coverage table above is illustrative of the output format, not a measured run.