Document sources
A document source pulls files from a document store (a shared-folder store, a page store, a project tracker, a meeting store) into a namespace’s documents and keeps them in step, on any shared namespace or collection. This page is the design of sources of many items given as pasted links, per-user connected accounts, an org-wide crawl, and access that follows the store’s own permissions. The caller’s view of today’s sources is in knowledge: document sources; parsing is in document extraction. Optional provider notifications request the same sync path; source push describes their setup, authentication and lifecycle.
What changes
| Today | With this design |
|---|---|
A source holds 1 to 20 folders, projects, pages or databases, and no op changes its config. |
A source holds up to 300 items of mixed types, given as links or ids, and knowledge.source.update adds and removes them (items). |
| One shared credential per source, typed by an admin. | A shared credential, a person’s own connected account, or a delegated service account that reads many people’s drives (credentials). |
| Every reader of the namespace reads every synced document. | A source’s access is namespace or mirrored; a mirrored document is shown only to a reader the store says may read it (access). |
A linear source reads project documents only. |
Issues and comments are a later design of their own (issues). |
A recording source reads meeting-store transcripts (meet) or cloud-recording transcripts (zoom). |
Meeting notes and transcripts that the meeting store saves to the shared-folder store come in through a gdrive source; recording keeps zoom only (meetings). |
Any namespace
A source can be registered on any active shared namespace or collection; who may
register which kind is in credentials. add on an inactive namespace
answers NAMESPACE_INACTIVE, and add and update answer ERASURE_RUNNING (409,
retryable) while an erasure that will still run names the namespace, so no source writes
into a mirror the erasure is clearing. A source on a repository’s trunk code namespace
still tags each document repo:<id>, so the repository’s code.context and
loom_search legs read them as today; on any other namespace, a branch namespace
included, it does not. A person’s own place (a col-me-… collection, a personal-… or
u-… namespace, a namespace claimed at sign-up) holds no source (PERSONAL_NAMESPACE):
a user_id erasure erases what the person wrote, and a source’s documents are the sync
worker’s. The sync skips a source whose namespace is inactive; the cleanup of a stopped
source runs whatever its namespace’s state.
A source ends with its namespace: code.repo.remove, namespace.delete,
collection.delete and a namespace erasure each retire the namespace’s sources under
its source lease, clearing their credentials, as code.repo.remove does today, so no
source outlives its namespace with a sealed credential or writes back what an erasure
took.
A source’s sync state is keyed by its namespace and name, as today, and a mirrored
source’s by its mirror and source id, so the namespace’s erasure covers it and a later
source of the same name resumes none of an earlier one’s listing. It lives where
storage layout places it (loom.code_index_state today,
Postgres cursors from wave 3).
On the dashboard, the Library’s Sources button opens the sources of the namespace or collection in view; the repository card keeps its own.
Items and links
A source’s items are its things to read, of mixed types:
| Kind | Item type |
Fields | Reads |
|---|---|---|---|
gdrive |
folder |
id, recursive (default false) |
the folder’s files; with recursive, every subfolder’s too, to a depth of 8 and at most 500 subfolders per source (both not settings) |
gdrive |
file |
id |
one file of a kind include names |
gdrive |
shared_drive |
id |
every file of a shared drive |
gdrive |
filter |
any of name_contains, kinds, modified_after (a date), and optionally in (a folder id) |
the files that match, as a query the server builds |
gdrive |
meet_recordings |
none | the meeting store’s recordings folder of each account a connected or crawl source reads (meetings) |
notion |
page, database |
id |
as today’s page_ids and database_ids |
linear |
project |
id or slug_id |
the project’s documents, as today’s project_ids |
Every id keeps today’s alphabet check, since an id goes into a store query or a request
path. A filter never takes a raw query string: each field has a pattern (a name
fragment of printable characters, an ISO date, kinds from the include table), quotes
and backslashes are escaped, and server/clients/gdrive.py builds the query, so a
caller cannot widen a listing past what the fields say. include stays the source-wide
file-kind filter. meet_recordings is refused on a shared source, whose service
account has no recordings folder of its own.
A source holds at most 300 items (a wire contract, not a setting; a linear or notion
source keeps its 20 until its turn bounds what it lists), and at most
LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS documents (default 50,000). The bound counts the
source’s stored documents plus the new ones a turn writes (at most 200,000, since the
paged read of the stored copies costs about the square of the bound over 10,000). A turn
over it (a listing grown past it, an item swap whose new files are listed before the old
copies retire, or a lowered setting) holds back new documents only: it finishes its
listing, rewrites changed lists, retires, and lands its listing row, so its copies drain
back under the bound and no list goes stale; inspect and loom doctor show
too_many_documents meanwhile, with the count, for a namespace source; for a mirrored
source, its connected owner sees the code without the count, and a platform admin sees
both (exception 4). The read of a source’s stored copies goes in pages by source_uri,
so it never cuts one off. A listing asks the store about at most 50 parent folders per
query (not a setting), so a source of many folders lists in several queries; a recursive
folder’s subfolders are walked on every listing, so a new file in a new subfolder
arrives with the next turn, and files moved in with it (a move keeps a file’s change
time) with the next full listing; a walk the pass deadline cuts leaves the listing
incomplete, so the turn keeps its stamp. A walk cut at its depth or folder bound still
lets a full turn retire a file the store reports deleted or trashed, or one of a kind
include no longer names, but never one the listing merely lacks, since it may sit
below the cut.
A probe checks a bounded sample: every item while there are at most 20, otherwise the 20
items added last, and the answer says sampled: true with the number checked, which is
shown wherever the item count is.
On a mirrored source, the probe and the status hold back what depends on what a person can read:
- A mirrored source’s probe answers reachability, the number of items checked and codes, never a count, a sample of titles or a summary that names a file, since it lists as the owner or as crawled people; which items were checked goes only to a connected source’s owner and a platform admin.
- To anyone else, a mirrored source’s probe, status and
sync_nowanswer only source-level codes, whether the credential and its scopes work andowner_inactive, and depend only on source-level conditions, so a held-back code reads as a clean sync.rate_limited, which grows with how much a source lists, an item refusal, a per-file or per-item code (item <n>: http_404,meet_recordings: not found,too_large,extraction_failed:<code>), the oldest listing start and every last-sync time go only to the owner and a platform admin. - A refused item of the sample refuses
addorupdateonly on anamespacesource and on a connected source probed by its owner; on any other mirrored probe, a platform admin’s included, items never refuse, and only the source-level codes do. The next sync turn reports a refused item asitem <n>: http_404, to those who see per-item codes, and goes on with the rest.
The link parser
server/knowledge/domain/source_links.py turns a pasted link into an item, with no
request to the store:
| Link | Item |
|---|---|
https://drive.google.com/drive/folders/<id>, with or without u/<n>/ or mobile/ |
gdrive folder |
https://drive.google.com/file/d/<id>/… |
gdrive file |
https://docs.google.com/document/d/<id>/…, …/presentation/d/<id>/… |
gdrive file |
https://www.notion.so/<slug>-<32 hex>, https://www.notion.so/<workspace>/<32 hex>, https://<workspace>.notion.site/<slug>-<32 hex> |
notion page (database with ?v=, unless p= names a page) |
https://linear.app/<workspace>/project/<slug>-<id> |
linear project |
A shared drive’s own link has the folder form: a folder link names a folder, read
without its subfolders, even when it is a shared drive’s root; a recursive folder at a
drive’s root is turned into a shared_drive item by the probe. A link of another host,
with no id, or to something the sync does not read (a spreadsheet, a single issue)
answers unreadable with the reason, as does an open?id= link, which names a folder
or a file alike. The parser takes a list and answers each link, so a paste of fifty
links shows which three it could not read. It is knowledge.source.parse_links (viewer;
no store request, nothing written), which agents use to preview pasted links; add and
update take links directly. A page link that turns out to be a database is reported by
the probe. A project link carries the project’s short id, which add and update
resolve to the UUID the store’s API takes, in one lookup before the probe.
Wire shape
knowledge.source.add takes items, or links it parses (an unreadable link refuses
the call naming the first 10 with their reasons and counting the rest, BAD_LINK), and
still takes config, which it turns into items (folder_ids into folder items, and
so on), so an N-1 client keeps working. list and inspect answer items, and
config in today’s form when the items can be written in it, else {}; a mirrored
source answers config {} to everyone, and its audit rows carry every field of its
items and a crawl’s subjects as keyed hashes only (below). A filter has no owner field:
it would keep a person’s address in the registry, which outlives their erasure. Its
name_contains is an admin’s text and, like the source’s name, stays in the registry
and the audit log after an erasure.
knowledge.source.update takes add_items, remove_items (each also as links) and a
new include, probes what it adds with the stored credential (the same sample rule),
and writes the source at its next version under the namespace’s source lease. Its audit
row, and add’s, names the namespace, the name and the items added and removed, with a
crawl’s subjects and every field of a mirrored source’s items as keyed hashes (below).
Removing every item is refused (BAD_CONFIG); remove ends a source. Changing access
is a new source. For a crawl, update also takes subjects (the new list, probed
like added items) and directory_subject. Every update and rotate of a crawl needs
the same roles as add, since its credential reads every listed person’s drive: a
namespace admin without an org-wide role cannot change whose drives it reads, what it
reads of them (items, include) or the key it reads with.
One document, one owner
In a namespace, one document per address, as today, owned by one source: the source
whose source:<name> tag it carries, written by the sync worker (today’s ownership
check). Only the owner rewrites the document when its file changes and retires it on
positive evidence (today’s retirement rule), and a document another writer (a person, an
agent) wrote is never overwritten or retired by any source, as today. A mirrored
document also carries source-id:<id>, the id of the source that wrote it, and in a
mirror that tag, not the name, decides ownership: a later source of the same name owns
none of an earlier one’s documents, and the cleanup of a stopped mirrored source retires
by it. Only N writes a mirror, so every mirror document carries the tag. Inside a mirror
a document’s source_uri is its address with the source id appended
(<address>#<source id>; the address is the part before the last #, and citations and
answers use it), so each mirrored source writes its own copy with its own list under the
table’s key, heads and document machinery as they are, and no source’s write or
retirement touches another’s copy. The legs answer the address itself, dropping
duplicates after the check (below), and a get or download of an
address answers the namespace copy when the reader may see it, else the newest mirror
copy the reader may see. Only the owning source’s turn writes a mirror document’s list;
any other writer of a mirror row (a retention or tier move, a sweep, a reindex) writes a
tombstone or unread, never a list it carried. Only those writers and the erasure write
rows under a mirror’s name; no request does. How a stopped source’s documents and rows
are found and cleaned when several sources, builds or names meet is the implementing
pull request’s to specify.
In a namespace, a second source that lists a file at an address where a document is
counts it skipped, as today, and fetches no content. No source reads whether another
is live: an address is free exactly when no document is there. Once the owner’s document
is retired (by today’s retirement rule, its
remove with delete_documents), the next source that lists the file writes it as a
new file under its own tag at its next full listing
(knowledge_source_sync_full_interval_seconds). A kept document (one a remove without
delete_documents left) is skipped by every source until knowledge.delete removes it,
as today.
So a file that two namespace sources reach is written once and read through one
document. Between the owner’s retirement and the other source’s next listing of the
file, the document is missing; that gap is the cost of keeping one owner per address
there. A mirror pays duplicate chunks for a file two mirrored sources reach instead.
A mirrored predecessor never makes add of its name answer SOURCE_DELETE_PENDING,
since mirror ownership goes by source id, and it counts toward no source quota while it
waits; its pending retirement shows to a platform admin only. remove with
delete_documents retires the source’s documents, as today, and without it they stay as
kept documents, which add of the same name refuses to take over
(SOURCE_DOCUMENTS_KEPT, as today). A mirrored source’s documents are always retired
(below).
Credentials
A source has a credential_mode:
| Mode | Credential | Registers it | Writes to |
|---|---|---|---|
shared |
Today’s: a service-account key, an integration token, an API key | an admin of the namespace |
the namespace, or its mirror when access: mirrored |
connected |
The registering person’s own connected account for the kind’s store | a person with writer on the namespace; the source is theirs |
that namespace’s mirror |
crawl |
A service account with domain-wide delegation that reads, as each listed person, what that person can read | an admin of the namespace who also holds org_admin or platform_admin |
the namespace’s mirror |
The gate of add, update, check, sync_now and remove drops from admin to
writer. The handler then requires admin for a shared or crawl source. On a
connected source only its owner adds items or changes include, since the items read
with the owner’s account; its owner or an admin of the namespace may check, sync_now
and remove it, and only its owner, who sees them, removes its items. rotate stays
admin (for a crawl, also an org-wide role) and is refused for a connected source,
whose credential is its owner’s connection. Owners are compared as canonical subjects
(server/auth/aliases.py canonical_subject), so a person’s other spellings are the
same owner.
LOOM_KNOWLEDGE_SOURCES_MAX_PER_NAMESPACE (default 8) bounds a namespace’s shared and
crawl sources, as today. LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_PERSON (default 8)
bounds a person’s connected sources in one namespace, and
LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_NAMESPACE (default 64) all of a namespace’s,
so writers cannot fill an admin’s quota or the sync’s passes. A crawl’s credential reads
every listed person’s drive, which is why it needs an org-wide role; a turn stops a
crawl whose registrant no longer holds admin on the namespace and an org-wide role, as
the owner check below stops a connected source.
Connected accounts
A person connects their own account from the dashboard (Settings → Connected accounts)
or with loom connect gdrive|notion|linear. Connecting needs LOOM_PUBLIC_BASE_URL:
the redirect URI registered with the store is built from it, never from a request’s
Host.
-
The flow is bound to the browser that starts it, as SSO account linking binds its ticket (
_link_cookieinserver/auth/oauth.py). The dashboard’s page callsPOST /v1/connect/<kind>/startwith its bearer token, which must carry a sign-in session (sid), which a service-account token lacks; the cookie below, not the token, is what ties the connection to one browser. The server keeps a single-usestatein Keeper for 10 minutes, bound to the subject and to a random binding it sets in that browser as an HttpOnly,SameSite=Laxcookie, and answers the store’s consent URL, which the page opens. The store redirects toGET /v1/connect/<kind>/callback, which stores the connection only when the request carries the cookie whose binding thestatenames; the subject is the one recorded at start. A consent URL sent to someone else attaches nothing: their browser has no such cookie.loom connectopens the dashboard’s Connected accounts page for the kind, where the person signs in if they have not. PKCE where the store takes it;include_granted_scopes=false, so an earlier grant of the same client is not folded in. After consent succeeds, a callback whose HTTPAcceptheader prefers HTML returns a redirect to the fixed same-origin/settings/connected-accountspath. Other clients receive{"connected": true}as JSON. Both responses disable caching, carryReferrer-Policy: no-referrer, vary onAccept, and clear the browser binding cookie. The dashboard URL carries no code, state, token or caller-supplied return address. -
Scopes, the narrowest each store has:
gdrive:https://www.googleapis.com/auth/drive.readonly,openidandemail(which the token answer spellshttps://www.googleapis.com/auth/userinfo.email). The Drive scope is a restricted one, so the operator registers an OAuth client that the store treats as internal to the org’s workspace; a public client needs the store’s verification first.drive.filewith the store’s file picker reads only files picked in a Loom page, never a folder’s later files or the recordings folder, so it is not offered. The store’s device grant does not issue Drive scopes.notion: the public-integration OAuth, where the person picks the pages to share on the consent page. The operator registers read-content capability only, without write, comment or user-information capabilities. The OAuth flow has no documented scope parameter; token introspection can report an opaque scope, without a documented mapping to capabilities. Neither that value nor its absence proves the registration is read-only. The registration is the permission bound; a display email is optional because reading it needs a separate user-information capability.linear: an OAuth application with thereadscope andactor=user.
-
GET /v1/connect/<kind>/callbackexchanges the code. Each granted scope is mapped to its canonical name, and a granted scope outside the kind’s set refuses the connection. The callback stores inloom.connected_accountsthe canonical subject, the kind, the store account’s id, optional display email, scopes and refresh token. The store id and email describe the connected account; neither changes the person’s Loom identity or grants namespace access. A usable refresh token is required for ongoing sync, including fornotion; a missing token refuses the connection instead of falling back to storing an access token. The refresh token is sealed under the purposeconnected_accounts.credential, whichloom secrets resealcovers. An access token is held in memory only. The optional display email is sealed separately underconnected_accounts.email; neither value is copied into a source row. -
Refresh for a source retains its account generation and source binding across provider work. One reservation owns the account generation at a time. A successful rotation is sealed only while the owner, account fence and reservation remain current; source or requesting administrator loss then refuses delivery without discarding that rotation. Disconnect, replacement, erasure or deactivation cannot be undone by an older refresh. A cancellation or lost response does not prove the provider left its token unchanged, so a retry needs the provider’s documented recovery guarantee. Missing expiry metadata does not justify an invented lifetime or permanent access-token fallback.
-
Each native HTTP call receives an internal
CheckedAccesscheckout rather than an ordinary credential string. It retains the owner, source id and connection generation, rechecks standing after pacing and before dispatch, and rechecks after the bounded response stream closes. Its deadline covers those waits and the body read. It cannot be converted into a shared credential or serialized, and it forwards credentials only to the provider’s fixed native API paths, without redirects. A source changed during the request discards the result. -
Provider token contracts differ:
gdrivecan omit a replacement refresh token. Retain the existing refresh token in that case. Revoking a token affects the person’s grant across the OAuth project’s clients. Use a project dedicated to document connections; unrelated applications must not share that revocation boundary. Other affected connections stop when refresh is refused.notionrefresh replaces both access and refresh tokens. The authorization guide requires the pair, while the response reference permits a null refresh token; a reply without usable replacement material is unavailable. The documented REST flow supplies no token lifetime, PKCE or lost-response recovery guarantee to assume.linearrefresh replaces both tokens and renews the refresh token’s 30-day lifetime. Track access expiry fromexpires_in. The provider documents a 30-minute recovery window for a lost successful refresh response, excluding revoked or replaced credentials; this does not permit retrying another provider’s rotation.
These contracts follow the stores’ authorization references: Drive, page-store authorization, page-store refresh, capabilities, token introspection and tracker authorization.
-
A connection belongs to the subject that made it, and only that subject’s sources use it.
connection.listandconnection.disconnectact on the caller’s own; an erasure of the person deletes its connections and retires their sources (erasure). An org admin sees in admin Users which kinds each person has connected, never a token or the store account’s email. -
Connecting again with another store account replaces the connection and stops the sources that used the old one, as a disconnect does: their items were chosen for the old account.
-
Every path that ends a connection (
connection.disconnect, connecting again with another store account, a SCIM deactivation for every subject it revokes, an erasure of the person, a restore) ends it in one order: it unseals the credential into memory, clears the sealed credential and marks every source that uses it inactive (the part that fails closed), and only then revokes at the store with the token in memory, best effort, under a short timeout and outside any lease or claim, as SCIM revokes tokens past its leases; a revoke that hangs or fails leaves the credential cleared. Local disconnection is complete independently of remote revocation. The page store’s revocation reference documents access-token revocation, without promising refresh-token or whole-grant revocation. An access-token revoke therefore cannot be reported as invalidating its refresh family. If only refresh material is available, bounded refresh solely to obtain an access token for revocation may follow local inactivation; no replacement credential is persisted. An unavailable or unproven remote revocation remains so, and the person can remove the integration’s grant at the store. The sync worker then retires those sources’ mirrored documents. A turn also checks the owner: a source whose owner subject is revoked, deactivated, or no longer holdswriteron the source’s namespace is stopped (written inactive under the source lease,status: owner_inactive) and goes as a removed source does, so a missed hook fails closed.
A connected source reads the person’s own files, the files and folders shared with them, their recordings folder, and the pages and projects they can read. It reads no directory, so it expands no group (below).
Org-wide crawl
A crawl source’s credential is a service-account key whose client id the org’s
workspace administrator granted domain-wide delegation for drive.readonly,
admin.directory.group.member.readonly and admin.directory.user.readonly. Its config
holds directory_subject (a workspace administrator the directory reads impersonate)
and subjects: people (emails) and groups (group emails), whose members the directory
read lists at each turn, at most 2000 people (not a setting). For each person the sync
exchanges the key for a token that impersonates them (the JWT bearer grant with sub)
and lists the source’s items as them. A typical config is one meet_recordings item and
a filter with kinds: [docs], over a group: every member’s meeting notes, without
anyone connecting. Every list and members row the sync writes, a crawl’s expansion
included, leaves out the people an erasure excluded (erasure).
The probe checks that the delegation grants no write on what the crawl reads: it asks
for each of the three scopes, which must be granted, and for
https://www.googleapis.com/auth/drive, admin.directory.user and
admin.directory.group, which the store must refuse; a credential the store grants any
of them is refused (CREDENTIAL_REFUSED). It cannot prove the delegation grants nothing
else (a mail scope, say), and the page says so to the admin who registers it. The audit
rows of add, update and rotate record the configured subjects as keyed hashes (an
HMAC under the root secret, as SCIM’s audit rows hash a subject, so a list of the org’s
addresses does not reverse them). Audit rows also record the scopes granted and the
number of members resolved: the audit trail outlives erasure and keeps no crawled
person’s address.
Access follows the source
A source’s access:
namespace: every reader of the namespace reads every document, as today. The only mode for asharedsource of a store whose permissions the sync does not read (notion,recording), and the default forsharedgdriveandlinearsources.mirrored: a document is shown only to a reader the store says may read it. Required for everycrawland everyconnectedsource.
The model has four parts: mirrored documents live in a companion namespace no grant
opens; each document carries its permission list; every read through that namespace
checks the list and fails closed; and every complete listing reads the lists again. One
rule holds them together: nothing derived from a mirrored document (its address, title,
text, counts or existence) enters a Loom record outside the mirror or an answer to a caller. An op
that is not a checked leg (a source’s probe, status or remove, an erasure’s receipt)
therefore answers codes for what a mirror holds, never a count, title, file id or
address, and a code that depends on the mirror’s rows (a residue failure) reads as the
same code for any cause. The exceptions are these:
- A checked leg’s answer to its reader, every field of which depends, of the mirror, only on the items that reader was admitted to, and otherwise only on the request and on whether the leg’s reads succeeded.
- The rows a marked request leaves, and every read of them (its own answer falls under
exception 1, so a marked
memory.askstill answers itssynth_*fields to its caller):- Its spend (input, output and embedding tokens, and cost) depends only on what
exception 1 lets the answer depend on: no model or embedding call runs, or is
chosen, on a mirror candidate before the check. It counts in its namespace’s totals
and cap. Except to a verified platform admin without namespace-role restrictions,
it shows only in the per-namespace-day total and in
cost.budget_status’s live namespace or org total, and every othercost.summarydimension leaves it out. The fold carries ops, those tokens and cost only, and leaves marked requests out oferrorsand the latency figures. A viewer can still difference a live or daily total around another person’s marked request and so learn that request’s spend; that residual is accepted. Named-user caps also count marked spend. Their public status exposes the cap and whether the user or namespace ceiling is reached, but exact user spend and percentage arenullexcept to that operator. User warnings expose the selected user, cap, warning share and threshold crossing, with exact spend withheld from every recipient, including an operator’s configured inbox or webhook. Namespace warnings retain the allowed namespace total. These policy signals expose bounds; an admin changing caps can infer tighter bounds. That residual is accepted, not a claim of full noninterference or a grant to marked work, models or outcome fields. - Its work figures (statements, rows and bytes read, chunks injected,
llm_calls,response_tokens, the stored duration), itsmodeland the outcome fields a model call can change (success,error_code,status_class) go to a platform admin only, and the error-rate and model metrics leave marked requests out. The stored argument map keeps only numeric statement/read/injection work fields. Result identifiers, document tags, argument digests and error prose are omitted.
- Its spend (input, output and embedding tokens, and cost) depends only on what
exception 1 lets the answer depend on: no model or embedding call runs, or is
chosen, on a mirror candidate before the check. It counts in its namespace’s totals
and cap. Except to a verified platform admin without namespace-role restrictions,
it shows only in the per-namespace-day total and in
- A connected source’s per-item and per-file codes and
too_many_documents, to its owner, and its listing age, to its owner as a code (fresh or stale). - A platform admin’s view of a mirror’s sources (every mirrored source’s status, the
probe’s sampled items and their codes, and the oldest listing start), its sync and
spend, and its erasure counts, as codes and counts, through
loom doctor,knowledge.source.inspectand the erasure ops.
The rule covers Loom’s records, application logs, spans, metrics and caller-visible answers. Database-native query, error, mutation, text, profile and trace logs remain privileged deployment infrastructure, under the same trust as the plaintext tables and backups. They can retain mirrored addresses, counts and values quoted in errors after erasure; the database operator controls their access and retention, and any export of them keeps that privileged boundary. This is the existing database-log boundary and erasure limit. The SQL surface and analyst must refuse access to these logs. Loom’s statement-capture files, provider and extractor telemetry, and caller-readable exports remain subject to the mirror rule.
A deployment with auth off (LOOM_AUTH_REQUIRED=false) has no reader identity to check.
It refuses to add a mirrored source with AUTH_REQUIRED, which wins over
MIRRORED_SOURCES_OFF, the answer of every deployment until
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED is set. That setting turned off again once
mirrors exist makes the mirror legs answer nothing and stops the sync apart from the
cleanup of a stopped source. Auth turned off once mirrors exist does the same, and the
gate still refuses every mirror name; the cleanup only deletes.
The mirror
A mirrored document is not stored in the source’s namespace but in that namespace’s
mirror: a namespace named acl-<24 random hex>, made under the namespace’s source lease
with its first mirrored source and recorded in loom.knowledge_mirrors (org, namespace,
mirror). The gate admits no token to a name of that form (acl-[0-9a-f]{24}):
check_authorized and CachedGate refuse the name before any other rule, the auth-off
return included, and namespace.register and namespace.restore refuse the form.
erasure.request refuses an acl- name in its scope for every caller but a platform
admin with auth on, and erasure.export for every caller; with auth off both refuse it
for everyone. So no token searches, reads or exports a mirror, and the threat model’s
rows hold for every token. A mirror has no directory row and appears in no namespace
listing or usage view; a deleted name keeps its mirror and counts as in use
(namespace_census.in_use, with the same code as any other reason, whatever the mirror
holds) until a namespace erasure, or the mirror-erase command, writes its
loom.knowledge_mirrors row deleted. A namespace erasure does that last, after the
mirror’s rows (the row holds names only), so an erased name can be registered again and
gets a new mirror. A name that has ever had a mirror is always erased as a purge, a new
owner’s erasure included, since the registry keeps every mirror it had. The migration
that enables the refusal first looks for namespaces of the reserved form, registered or
holding rows, and refuses to run, naming them, if it finds any.
A knowledge read taught the mirror adds a leg over the namespace’s mirror, with the
check below, while the namespace’s source list holds an active mirrored source (a fact
that list shows every viewer); a mirror with no active mirrored source in that list runs
no leg. Every other read path reads the namespace and finds nothing of the mirror. So a
new op, a context-pack leg or an export that nobody taught the check shows no mirrored
document; an N-1 replica has no mirror leg, and the rollback procedure keeps it off a
mirror that holds rows. A permission column on loom.documents read by every path would
make the opposite the default; sharing rejected
that for collections, since one missed path leaks.
Search discovers chunk keys and scores without returning chunk text. Each leg checks candidates in batches of at most 64 until its admitted pool fills or its stream ends; denied candidates consume no pool position or fusion rank. Content reads bind the exact admitted namespace, address, chunk index, version and revision under the requested embedding model. A missing key is omitted without substituting a later version. Duplicate or unexpected rows discard all mirrored legs. Permissions are checked again after content is fetched. Listing evidence and any group membership relied on must still be fresh before the content leaves the checked leg. Independent email, owner, domain or public grants can admit a document whose group evidence expired.
A marked private-first search gathers both possible passes from its authorized namespace list and visibility settings before choosing which to return. It refreshes all retained mirror permissions and reranks before applying the first-pass similarity threshold. A provisional private hit that loses permission therefore cannot suppress the public fallback. No reranking provider receives the provisional pools.
-
Legs with the check:
knowledge.search,get,outline,list,download,synthesize,knowledge.inspect(titles),knowledge.source.inspect(a mirrored source’s documents), thesearch.queryknowledge and repository-docs legs,code.context’s docs section,context.pack,memory.stack,memory.askand the assistant (which calls the same handlers under its answer token, delegated from the caller’s). Aget,outlineordownloadof a mirrored address the reader may not see answersfound=false, as for an absent one. -
Nothing to read: decision mining and dataset extraction skip mirrored documents, since a decision row is a memory every reader of the namespace reads and a dataset is read by
knowledge.querywithout the check. -
A request that touches a mirror is marked. Nothing a marked request writes outside the mirror depends on what the mirror holds, except as exception 2 allows: no cache shared across readers (the repository-docs listing cache,
memory.ask’s precedents, the provider response and embedding caches) is read or written with text a mirror leg gave, and no field or row of itsagent_events, trace, exposure, metric or span records, of recall credit (it writes none toloom.memory_usage_events), nor a generation’s captured text, depends on the mirror. Every request that runs a mirror leg is marked, whether or not anything was admitted, since marking on admission would tell existence. The costs are accepted:knowledge.readsnames no address a marked request read, a namespace document’s included;- every marked knowledge read drops out of
cost.summarybyuser_id,operationandagent; - a marked request writes no recall credit or exposure row, so the usage signals
(
recall_count,last_recalled_at, the usage totals) leave out every marked read; - the assistant’s support check reads a marked call as unknown, so a memory it cites
from one gets the reason that the check could not run (
unchecked), notunsupported; - the assistant’s step list cannot say a marked tool call failed;
- every session whose request runs a mirror leg is flagged, and a prompt-recall
context.packruns one, so while a namespace lists an active mirrored source every Loom hook of a session over it stops after its first recall, reads included, and a dashboard view of a flagged answer sends noui.vitals(the session rule).
The assistant front end’s token record of a marked answer is a spend row under exception 2 that only the deployment’s operator reads; the conversation itself lives with the caller (the dashboard’s browser storage, or the message list an SDK or HTTP caller sends).
-
The flag. The answer of every marked request carries
mirrored: trueas its first field or line, whether or not anything was admitted, for the same reason as the mark, and each admitted item carries it again, so a truncated copy still carries it. Every assistant answer on a namespace with an active mirrored source (theassistant.askop and the chat route,POST /v1/assistant/chat/completions, which carries the flag in its stream’s first event or its response’s first field) is marked and flagged, since its answer is model text written from tool results and need not carry the field the tools answered. A mirror leg answers nothing to a client that does not declare it honours the flag. Loom’s plugins and CLI declare it, and so do the dashboard, which shows the document, and the assistant, whose answer token declares the flag only when its caller’s request did: a caller that does not declare gets no mirrored text through the assistant, and the assistant writes nothing from that turn into anything it shares. An SDK declares it only when the app built on it opts in. -
The session rule keeps a session that saw a mirrored answer from sending it on (the session rule).
-
The sync and the workers write nothing outside the mirror. The mirrored sync’s spend charges no namespace’s cap, and
LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASSbounds it. A mirrored source’s sync state, the sync’s audit rows, error text and per-file wait state are filed under the mirror’s namespace and go with it when it is erased; whatever of a mirror leaves it (a source’s status, a worker’s pass notes, logs, spans and metrics, the extractor service’s logs and traces) carries codes only (exceptions 3 and 4 apart). Worker pass summaries exclude mirrored turns’ completed-source counts, document figures and cut flags. A failed mirrored turn contributes onlySYNC_FAILED, for every cause; returned turn details remain private. Mirrored per-file failures emit no log record, since the number of records would disclose a document figure. A sync turn enters its private diagnostic scope before taking its source lease or listing files; retirement enters it before reading its queue. That scope suppresses client and provider instrumentation, database work figures, shared metric calls and log records, including their volume. Provider health and embedding caches receive no private contribution. The extractor receives a generic filename; its service emits no per-request access, stderr or fault logs. A pass that touched private work records codes without duration, outcome figures or failure counters, including when invoked from the CLI. Its retry history cannot advance a later ordinary pass’s diagnostic streak. Ordinary work keeps its own cost accumulator; private spend is written directly under the mirror, without entering the shared event queue or its statistics. -
No scope reaches a mirror. Every statement a request runs that reads a table holding mirror rows other than by a namespace the caller names leaves out the
acl-[0-9a-f]{24}form; the workers named here are the exception, since they must reach the mirror’s rows.resolve_org_scope(its auth-off filter included),every_namespace_scopeandrollup_member_namespaces(server/auth/context.py) leave it out for a request’s caller; the sync, the erasure saga, the mirror-erase command (which finds mirrors by theacl-form in every store, not only by their registry rows, so a restore that loses aloom.knowledge_mirrorsrow orphans nothing) and the maintenance workers (reindex and embedding-model rebuilds, tier and retention moves, the blob and originals sweeps, which keep a mirror’s rows referenced) reach a mirror through the stores, never the gate, under the rule of the bullet “the sync and the workers write nothing outside the mirror”. Everysql_v1_view (each replaced by a migration), the analytics export’s statements and the org-wide reads oferasure.export(Shown.clausecarries the form) and of auser_iderasure’s steps and census leave it out too; a saga whose scope holds onlyacl-names is not found for anyone but a platform admin; every scope a saga answers to anyone else, and every audit row a saga writes, leavesacl-names out (a mirror-only saga’s audit rows show only that a platform admin erased something); soaudit.list, the insights, usage and cost reads, the SQL surface and a person’s export name no mirror.check_authorizedrefuses the form before its auth-off return, so no id-addressed check (row_reach) oraudit.listof a named namespace opens one. The deployment-wide admin reads (/v1/admin/, outside the gate) leave a mirror’s ledger rows out whole: until the fold below carries a marked request’s spend, a mirror’s spend is in no fleet or consumer total. -
Private events have separate physical storage. Ordinary events stay in
agent_events; marked requests and mirror workers writeagent_events_private. The ordinary table’s CHECK excludes marked rows and mirror namespaces. An analyst keeps ordinary signed, namespace-scoped reads and cannot read the private table or its physical metadata. Provisioning verifies the CHECK on every writer, drains the finite set of prepared INSERTs and refuses any existing private contamination before granting access. Checked per-writer HTTP sessions read the native access catalog and prove that policies cannot hide the process or event populations. Unknown visibility, writer routes or metadata refuse. It preserves existing rows for operator resolution. Administrative application usesloom ch-setup --apply-analyst-sql; unchecked analyst SQL files refuse. -
Private event reads are folded and masked. The analytics export and
sql_v1_usage_dailyunion both stores and fold marked spend into a namespace/day row with agent and operation-.sql_v1_agent_eventsand the signedanalysis_agent_eventsview mask every field that can depend on an admitted item before predicates run: model, tokens, cost, work,llm_calls, success, error code and status class. Trusted budget and history-policy reads cover both streams too. User erasure scrubs both stores underagent_events.scrub; the private store has the ordinary ledger’s retention and tiering policy. The storage cutover preserves ordinary analysis across builds. The assistant boundary describes the view and key-hiding requirements. Exception 2 accepts this fold besidecost.summary’s.
History predicates cannot use a marked call’s outcome or projected arguments. A
potentially matching marked event in a rule’s identity, operation and time scope makes
that leaf unknown before success or argument filters run. This includes an unresolved
argument reference and a negated condition. Breadth rules check marked participation
under the same principal pin because marked reads write no exposure row. The compiler
reads coarse participation and the ordinary aggregate in one statement. Protected
unknown carries one fixed reason and no count or timestamp; every affected enforcing
operation refuses, reads included. Shadow rules retain their non-enforcing behavior.
policy.check uses the same protected-history decision. Ordinary trace-backend errors
keep their existing read-availability policy.
A checked answer retains its access proofs through required policy witnesses and other work that must finish before publication. All provider work for its complete ordinary alternative finishes before cost and event snapshots. After required tail work, publication renews the exact private dependencies of the already selected frame and checks every retained deadline after the last renewal await. This boundary does not render, rerank, call a provider or choose an ordinary alternative. A lost proof refuses the whole frame; removing a citation from private prose is insufficient. After awaiting the response headers, publication renews access again before sending the body. A checked-proof or publication failure after headers aborts with a constant failure and sends no unchecked body. Discarded candidate widths and optional sections retain no publication guard; returned metadata, counts and cursor lookahead retain their contributing proofs. A raw download owns the same required marked accounting attempt as a registered operation. Cancellation and disconnects still propagate through owned cleanup. These checks end at the server’s response-send boundary.
The assistant’s incremental stream has a terminal accounting boundary because final
usage is unavailable before its early text. Its server-owned first event is
data: {"mirrored":true} when marked. Partial text remains provisional until the
response closes and joins its producers, captures final usage and confirms its
required private cost row before [DONE] or a terminal error. Disconnect cleanup
retains and joins that required delivery. Whole answers and checked source frames
keep the accounting-before-publication boundary above. The answer token’s distinct
answer_mirrors bit fixes mirror admission before model work, so a source activated
during a negative answer cannot introduce an unmarked tool leg. Missing admission is
false; ordinary delegation drops it. See the
assistant boundary.
A multi-entry MCP batch refuses every response-bearing marked entry with the same instruction to send it individually, including denied and empty entries. Notifications remain silent. It discards each marked entry’s complete frame before buffering it or awaiting another entry. Single calls and one-entry arrays retain checked reads. Each entry gets fresh local state before routing or screening; the enclosing HTTP request keeps a monotonic mark through later ordinary entries, streaming and cleanup. Middleware task boundaries do not clear that owner.
A mirrored source’s documents are never kept past it: remove retires every one it owns
(delete_documents is implied), and so do a disconnect, a SCIM deactivation, the owner
check and every retirement of a namespace’s sources. Each of them marks the source
inactive (in the request, or in the turn for the owner check) and leaves the deleting to
the sync worker, so no request’s work depends on how much the mirror holds. The worker
retires a stopped source’s documents by its source-id: tag outside the content bound,
with a per-pass bound of its own, so a large removal does not stall other sources’
content. In a mirror, ownership and every per-source document read go by the
source-id:<id> tag, not by name. The retention overrides give each mirror its
namespace’s retention_days and warm_days.
Stopped-source deletion has a durable source-id queue in code_index_state under the
mirror, with keys knowledge_source_retire:<source id>. A queue row contains only the
original namespace, source name, kind and registration id. The request writes it under
the sources lease before inactivating the source. A replacement registration can reuse
the name while the old queue row and copies remain; cleanup uses the row’s mirror and
source-id ownership, independently of the current mirror registry.
The knowledge_source_retire worker runs even with authentication or syncing off. It
shares the sync cadence, with its own deadline, cursor and
LOOM_KNOWLEDGE_SOURCE_RETIRE_DOCUMENTS_PER_PASS bound. Its cursor contains registration
metadata only. A queued source still active under the same id is left alone, so a crash
between enqueue and inactivation deletes nothing. Completion requires both no live owned
documents and no pending or parked publication while holding the source’s turn lease.
Clearing the queue writes an empty version row; erasure removes it with the mirror’s
other state. Cleanup exports no document counts or cut flags.
A namespace source and a mirrored source that reach the same file own two documents, one in the namespace and one in its mirror. The namespace copy is readable by every reader, because an admin registered a source that makes it so; every leg answers the address once, dropping duplicates after the check. When both copies matched and the reader may see the namespace copy, the hit is the namespace copy’s, with its own chunks and scores and no item flag. When only a mirror copy the reader is admitted to matched, the hit is that copy’s, with the item flag. Otherwise the mirror copy is dropped.
The session rule
A session-content op records a session’s text, or a figure derived from it, that a hook
or client sends without the agent choosing it; an op the agent calls with text it chose
is the reader’s own write. Each op declares
its class in the op registry (server/operations/registry.py), and an op both a hook
and an agent call is session content. They include conversation.append (which the
CLI’s ingest-transcript calls), conversation.save_message, the title and
metadata of conversation.create and conversation.update,
memory.set_from_messages, session.brief_save, observe.record,
observe.crystallize, working_memory.set, working_memory.append, run.begin,
run.complete, code.outcome, cost.record_turns and ui.vitals, and every read a
hook sends with a session’s prompt, files or transcript (context.pack, code.search,
code.impact, memory.for_file, namespace.validate). Framework callbacks also send
the latest user message through memory.search and persist conversation items through
memory.set, so both are session content. Its coalesced delivery through
memory.bulk_set has the same class. Automatic tool interception also sends
code.test_gate, code.symbol, code.enclosing and tracker.duplicate_check with
session-derived inputs. The rule decides; the
registry’s classes and a test over what the shipped hooks send pin the full set.
Once a session has seen the flag, no Loom hook sends any op from it, reads included, and
no client sends a session-content op on its own; the calls the agent makes go on,
dual-use ops (code.search, context.pack) included. A session is also flagged when a
subagent or a child process it started saw the flag: the child records it in the session
state it shares with the parent, which the parent’s hooks read, since the child’s final
text, reworded, need not carry mirrored: true. A session resumed, forked or compacted
from a flagged one is flagged, and so is one started from a flagged session’s process
(claude -p or codex exec run through its shell): the CLI and the hooks set an
inherited marker in the environment and read it, and keep the flag in the session’s own
state beside what they keep for resume and fork. A worker an orchestrator starts outside
that process tree (through tmux, ssh or a job queue) inherits nothing; what its own
session records is the orchestrating agent’s choice, out of scope (threat
model). The CLI’s ingest-transcript refuses a transcript that carries
the flag, or whose session’s state records it.
The refusal: from the first time LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED is set until
it is off again and the mirror-erase command has run, every session-content op refuses a
client that does not declare the flag, in every org, whatever mix of clients and tokens
a session runs; the declaration is the client’s promise, carried by the transport and
never by an op argument a model fills in. The exact transport declaration is one
X-Loom-Session-Protection: v1 header; missing, repeated or unknown values declare
nothing. An assistant answer token carries its caller’s declaration in the Boolean
session_protection claim the bridge fixes at delegation. An answer or its descendant
cannot replace that claim with its own header, and ordinary delegation does not inherit
the claim. A client that does not make
it (a third-party MCP client connected straight to Loom) is refused like any other. The
condition is the one row of loom.knowledge_mirror_latch or the process’s own switch: a
process refuses while its switch is on, row or no row, and writes the row whenever it
reads the switch on and finds it absent, so neither a lost write nor a restore of a copy
without it lifts the refusal. Only the mirror-erase command deletes the row, last, once
no namespace of the acl- form holds rows in any store, and it refuses while any live
process reports the switch on. The check fails closed: a process that cannot read the
row refuses, and the first mirrored add waits until every live process reports the
switch on, as the schema check does. Since the refusal lifts only then, it says nothing
of when another org’s mirrors went.
A process with the switch on confirms durable protection before startup can admit
clients or start workers. Its existing instance heartbeat reasserts that state, so a
lost row is repaired without waiting for a session-content call. Startup refuses an
unconfirmed write. The common gate runs after token verification. HTTP checks before
reading the operation body; MCP checks after resolving the operation and before
content validation, identity defaults, history, replay or implicit session runs. Its fixed SESSION_PROTECTION_REQUIRED refusal is HTTP 409 and is not
retryable without changing the client or completing the documented cleanup.
The latch is one fixed id = 0 KeeperMap row on the default Keeper ensemble, with
set_at and no TTL, tenant identity or content. It is deployment protection, so tenant
erasure never removes it. A fresh negative check uses a checked HTTP session under the
service principal. It validates the local latch and lease engines, primes the latch,
and brackets an acknowledged nonempty strict write to a unique expiring lease key and
the latch read with the same live Keeper client identity. Every proof SELECT returns
its execution’s server UUID, including an empty latch, and the barrier INSERT computes
the writer’s UUID into its own lease row. The final exact holder and write UUID must
match. This matters because a retried initial HTTP session creation can leave the same
session name on two backends; the name alone does not pin execution. Query caches are
disabled within this protocol. Any missing metadata, changed identity, uncertain write,
expired nonce or cleanup failure refuses the negative decision. Only its exact own
lease is disposable; a cleanup fault leaves an expiring row and never admits a request.
A positive repair computes its fixed row key in the INSERT from a check of the writing
server’s UUID against the validated backend. A differently routed write refuses; an
acknowledged strict insert or duplicate is followed by a fresh validated positive read.
No negative result is cached. The named-session transport alone is not a Keeper proof.
Mirrored sources are off for a deployment until an operator turns them on, after every Loom client and hook that sends session content honours the flag. Each pull request that adds a mirror leg, writer, client or hook lists what it writes and tests each against these rules.
The permission list
Each mirrored document carries its permission list in an access_json column of its row
(loom.documents; under the Postgres authority it rides in the head’s row jsonb, with
no Postgres migration), written by the turn of the source that owns the document:
emails: the people the store grants any role on the whole file, lower-cased, and, by credential mode:connected: the owner, whose account listed the file in that turn, entered as the source’sowner_subject(anownerentry, never an email, and never the connection’s store email), since the owner chose the item and can open it. When the store will not give the file’s list, the list is the owner alone, notunread.crawl: a person the crawl listed the file as, only when they own it or the permissions read as them name them, never from a read by an item or folder id or from a shared-with-me entry, which the store also gives a link-only share someone opened. An unreadable list holds the file’s owner if a crawled person owns it, and is otherwiseunread.shared: nobody more; its service account is no reader, so its unreadable lists areunread.
groups: group emails, which a crawl’s directory read expands intoloom.knowledge_group_members(one row per source and group, with its members andread_at). A connected orsharedmirrored source reads no directory and expands no group, so a file shared with its readers only through a group shows only to the people whose own account listed it.domainsandanyone: a domain or public permission the store marks discoverable. A link-only share (one the store marks not discoverable) is left out: it lets someone who holds the link open the file, not find it.unread: no current list is known; such a list carries noowner,emails,groups,domainsoranyone, and admits nobody.
For gdrive the list is the file’s permissions, read with the listing, or with
permissions.list when the listing leaves it out; for linear, the active members of
every team the project belongs to, a member of unknown state admitting nobody; for a
connected notion source, the owner alone, so only the owner sees a page. The list
lives and dies with its document: a retirement’s tombstone carries no list, and no read
op returns the column.
Who may read
A reader sees a mirrored document when all of these hold:
- The reader holds
vieweron the source’s namespace (the ordinary gate). - The document’s owning source (the registry row whose
source_idmatches itssource-idtag) is active and has a last complete listing that started withinLOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS(default 3600) of now, read from its row inloom.knowledge_source_listings. - The person the token reads as (below) is the list’s
ownerentry, compared throughcanonical_subject, or one of that person’s verified emails is in the list’semails, or in the members of one of itsgroupsfrom a members row of the document’s owning source read withinLOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS(by itssource-idtag), or that email’s domain is in itsdomains, or itsanyoneis set.
The reader’s verified emails are its subject’s directory emails that
VERIFIED_EMAIL_SQL admits (an SSO account’s or a linked one’s,
server/core/grant_sql.py), and nothing else: a subject that merely spells an address
is not one, and neither is a connection’s store email. The predicate’s passwordless
arm covers nonempty contacts from verified SSO provisioning or operator linking.
Password signup hashes the supplied password before storing its claimed address;
ordinary namespace registration cannot write a contact address or password hash.
The internal reader resolver retains the uncached canonical subject independently of
verified emails and reads actual active contact rows for email proof.
Signed hosted-domain evidence lives in sso_links.hosted_domain and issuer, written
after the final SCIM check under the account ownership lock. A domain matches only
through a current linked row naming the active passwordless account’s contact, with the
configured issuer and its signed hosted-domain claim. Empty defaults, missing claims,
issuer changes and generic link changes deny domain access. Neither a linked password
account’s email nor a bare verified address proves domain membership. These identity
inputs alone expose no mirrored document.
A request keeps a monotonic mirror mark in its existing work bag, shared with child
legs. An unbound direct helper serves no mirrored content. Active registrations, mirror
bindings and listing starts are read in sequence before candidate permissions. A failed
unconditional read marks mirror_unread; a failed candidate read invalidates the whole
mirror plan without that marker. Checked adapters cover search, get, outline, download
and listing. Other readers, marked-request sinks and session protections remain
prerequisites to enabling mirrored sources.
documents.access_keys and its bloom index select coarse candidates under the
ClickHouse authority. Owner and group buckets retain candidates needing identity
resolution; Unicode buckets retain matches whose database and Python normalization
can differ. Every candidate still needs the current strict permission check. Under
the Postgres authority these keys do not filter candidates: an unpublished head may
widen permission. Both chunk-frequency views exclude mirrors, and a lexical-statistics
rebuild refuses their names before reading or writing statistics. Mirrored lexical
scores use fixed corpus constants.
The checked list projection reads title, description, tags, blob metadata and the stored preview only at an admitted live document version and revision. Under Postgres, the current head supplies these fields and computes only the first 220 content bytes in the database, including for encoded text; no full body crosses into the list reader. Chunk counts belong to that same storage copy and revision. They count distinct embedding-model and section-index pairs, including sections awaiting embeddings. A Postgres count also requires the head’s recorded chunk version; a current revision without recorded chunks counts zero. Permission and serving-state proofs must remain current through the list’s final read before any count or lookahead is returned. Metadata discovery groups complete public address families before a public cursor or limit. Postgres reads internal pages of 64 rows, releasing the connection before permission checks; it reads every page without a candidate cutoff. Ordinary shadow checks read bounded batches independently of public filters and cursors. They cover the ordinary listing’s published rows and its unpublished Postgres heads; requesting superseded rows also lets an ordinary tombstone shadow a mirrored copy. Mirrored candidates remain live-only. Listing chooses the newest admitted complete copy of a public address, with a stable storage-identity tie break, before applying public tags, URI prefixes or the cursor. An ordinary copy wins its entire family before these filters. Selection retains only one current family, bounded shadow batches and the requested page plus one lookahead. After selected projections and counts are read, ordinary shadows, permissions, identity, document versions and recorded chunk serving state are checked again for every retained copy, including the lookahead. A lost proof drops the whole mirrored leg, so it cannot leave a stale cursor or reveal a removed row through the page’s continuation.
A token reads as one person, for both the owner entry and the emails, or as nobody:
- a person’s own session token (a primary token whose own
sidclaim names a sign-in session, with nocreatorclaim) reads as that person; a token minted for a subject without one reads as nobody; - a token minted for one person alone that says so (a
reads_as_creatorclaim, which only the person’s own primary first-party sign-in session can mint throughPOST /v1/me/personal_tokens.create, for the dashboard’s own-agent connection flow; delegation never copies it) reads as its creator; - the assistant’s answer token, delegated from the caller’s for that answer alone,
carries a reads-as claim that the bridge sets at delegation to the person the caller’s
token reads as, or to nobody, and reads as that; the check never infers it from the
token’s
jtior consumer, and no token delegated from it reads as anyone; - every other token (a service account made for tools, CI or a team, a bot, an end-user
token, whatever their
creator) and a request sent on behalf of another person (X-Loom-On-Behalf-Of) reads as nobody, so it sees onlyanyonedocuments.
The check fails closed. A stale listing, an unread list, an unexpanded or stale group,
a link-only permission, a permission the store marks deleted (its address may now belong
to someone else), a grant to one view of the file only (view), and a permission type
or role the sync does not know admit nobody. A missing registry or listing row admits
nobody. A leg whose per-request reads (the statements it issues whatever matched, the
reader’s canonical subject and verified emails among them) fail shows no mirrored
document and marks the answer degraded: mirror_unread; a read it issues only for
candidates (the lists, the members rows, the heads) that fails drops every mirrored
result without the marker, so the marker says nothing about what matched. Each leg reads
the registry row and the listing row first and then the members rows and the lists,
under sequential_reads(), so a lagging replica cannot pair a fresh listing row with an
older list. It computes the addresses the reader may see before its LIMIT, so a page
is never cut short by the check, and caches no admitted set. No score depends on a
document the reader may not see: a mirror’s chunks add nothing to any corpus statistic
(the lexical leg’s document counts, lengths and term frequencies), and a mirror leg
scores its lexical hits with fixed corpus constants, which depend on no document, so a
namespace fed only by mirrored sources still ranks by both legs. Under the Postgres
authority each leg judges the addresses it is about to answer by their heads, so a
narrowing or a retirement committed and not yet published already applies, and a missing
head drops the address; a leg that cannot read the heads shows no mirrored document. The
implementing pull request measures the leg on a mirror of a million documents under both
storage authorities and puts the figures here.
Every complete listing reads the lists again
Every complete listing of a mirrored source reads every file, metadata only (id,
revision, permissions): the store does not change a file’s modification time when its
sharing changes, so an incremental listing would miss a permission removed. A listing
that reads a list unlike the stored one rewrites the document’s row with the same
content and the new list, as knowledge.update_metadata without reindex does, so no
chunk or embedding is rebuilt; an unchanged list costs no write. A stored copy behind
its file’s revision (a fetch deferred, given up or over the size bound) takes only the
entries both its stored list and the new one hold, since its text was shared under that
list (entry by entry, so a person the stored list admitted only through a domain or
anyone is dropped, which fails closed); the content write that brings it up to date
carries the wider one. A behind copy that once reads unread stays so until that write.
A stored document the complete listing no longer has is rewritten unread, with every
entry cleared, then retired on positive evidence, as today. The listing row’s start
lands only once every changed list and every such unread rewrite of that listing is
written, so a row is never newer than the lists it vouches for. These rewrites are
outside the pass’s document cap; the permission reads they need stop at half of what is
left of the listings’ share when a turn starts them, so the turn’s new and changed files
still get time. Content is fetched only for a file whose revision changed, and a content
write carries the list of the latest listing that read the file, so a fetch the pass
bound deferred never writes an older list; a listing row lands only once every document
written during that listing carries a list it read.
A pass runs the listings first, rotating across the mirrored sources, and gives them at
most a third of the time the pass starts work in (a quarter of
knowledge_source_sync_max_pass_seconds) and their own document bound, while the
namespace sources then have the rest of the pass under theirs. Mirrored files consume
none of the namespace sources’ document bound. Their earlier work can reduce the time
left for namespace writes, so namespace progress can vary under the threat model’s
timing exclusion. A listing that does not finish goes on from its cursor in the next
pass, and until it completes the source’s listing row keeps its older start, so a large
source needs a stale bound longer than the time to list it all. inspect and loom doctor report each source’s oldest listing start, a mirrored source’s to a platform
admin only (its connected owner sees fresh or stale). A stopped sync hides every
mirrored document once the stale bound passes. The implementing pull request measures
the writes of a crawl over 2000 people and puts the figures here.
How lists from several accounts of one source, a person a crawl stops reading, a stopped source’s rows, and restores of either store are handled is the implementing pull request’s to specify against these four parts; each must leave no list or row that admits someone the store no longer names.
Threat model
| Threat | Boundary | Enforced by |
|---|---|---|
| A connected person leaks a private file into a team namespace | Every connected source is mirrored; its documents live in the mirror, and a teammate sees one only when the store names them. access cannot change. |
add and update; the mirror gate; test_document_source_leakage.py |
| A crawl’s registrant reads crawled people’s private files | No rule admits a source’s registrant or owner as such; a registrant reads a file only when the store names them or their own connected account lists it. | the access check |
| An admin points someone’s connected account at files they did not choose | Only a connected source’s owner adds its items or changes include. |
update |
| A link-only share becomes a search hit for everyone | A permission the store marks not discoverable admits nobody. | the access check |
| A person stops reading a file and Loom keeps showing it (stale ACL) | Every complete listing of a mirrored source reads every file’s list and rewrites a changed one; a source whose last complete listing is older than the stale bound admits nobody. | the sync’s mirrored listing; the access check |
| A person leaves a group and keeps reading through it (group churn) | Groups are read with every complete listing into a members row with its read_at; a stale or unread group admits nobody. |
the access check |
| Auth is switched off after mirrors exist | The acl- refusal runs before the auth-off return; mirror legs answer nothing and the sync skips mirrored sources while auth is off. |
check_authorized; the mirror legs |
| A person leaves the org and their connected sources keep syncing | SCIM deactivation and disconnect stop the sources and clear the credential, then revoke it at the store; each turn’s owner check skips a source whose owner is revoked, deactivated or no longer a writer. | server/auth/scim.py; the sync’s turn |
| A crawl reads more than its registrant meant, or can write | An org-wide role to register, update or rotate; the probe requires the store to refuse a write scope; subjects and scopes in the audit row. | the crawl probe; loom.audit_log |
| An id-addressed read confirms a mirrored document exists | get, outline, download and the inspects answer found=false or leave out an address the reader may not see. |
the mirror legs |
| A viewer or admin learns what a mirror holds from an op that is not a checked leg | A source’s probe, status and remove, and an erasure’s receipt, answer codes for a mirror; its counts go to a platform admin only. |
the source ops; the erasure handlers |
| A new read path shows mirrored documents to everyone | Mirrored documents live in a namespace no grant opens and no listing names; a path that does not read the mirror shows none, and no org-wide scope names an acl- namespace. |
check_authorized (acl-[0-9a-f]{24}); the scope helpers (the mirror) |
| An older build shows mirrored documents | N-1 has no mirror leg, but its org-wide reads return acl- rows and so hand out a mirror’s name, and its gate admits a wildcard admin token that names it. So no N-1 may serve while a mirror holds rows: the new modes wait for N everywhere, the rollback procedure erases the mirrors with every replica stopped first, and N reports an N-1 heartbeat while a mirror holds rows. An N-1 replica started without that procedure (a bare helm rollback) exposes mirror text to an org admin; that residual rests on the procedure. |
the rollback procedure; loom doctor |
| A mirrored document’s text reaches every reader through a derived row | Decision mining and dataset extraction skip mirrored documents; a request that touches a mirror writes nothing derived from it outside the mirror. | the sync; the mark |
| A store account is linked into another person’s Loom account, either way | The connection is stored only from the browser that started it, with that subject’s own dashboard session and the state cookie. A connection’s email is never a reader identity. |
the connect routes |
| A caller injects store query syntax through a filter | Filters are fields with patterns, escaped and built by the server; no raw query is taken. | server/clients/gdrive.py |
| A stolen connected credential changes the store | Only read scopes are accepted; the credential is sealed and never returned. | the connect callback; server/core/secrets_at_rest.py |
Out of scope: response timing, and when a pass reaches the namespace sources, which the
mirrored sources’ turns before them delay; a resource limit (memory, rows read) that a
mirror’s size can trip on a per-request statement, which like timing depends on how much
the mirror holds; how fast the shared pass gets to a connected owner’s own source, which
its fresh or stale code and per-item status reflect; that a namespace erasure is still
retrying, which a namespace admin sees whatever its cause; a worker an orchestrating
agent starts outside a flagged session’s process tree; a person who hands their own
reads_as_creator token to a team bot or a CI job, which then reads as them; a session
still open from before an operator lifted the refusal that saw mirrored text then; a
reader who already saw a document and kept its text in a record only they read, or one
their agent chose to write or sent in a call it chose (a follow-up memory.search whose
terms come from what it read, recorded in a trace log the namespace’s viewers read),
which is that reader’s write; that a person able to add a file where a mirrored source
lists can tell, by the file never appearing, that the source is over
LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS; the store itself reporting a wrong permission
list.
Meetings through the shared-folder store
Meet transcripts and notes are documents in Drive. A gdrive source reads them as
files: a meet_recordings item finds the recordings folder in an account’s own
Drive, and a folder item reads a supplied folder. Folder discovery uses the fixed
table of localized names. When no folder is found, the turn reports
meet_recordings: not found without naming an account; a folder item with the
folder’s link reads it instead. Individual document links use the same source.
A transcript Doc in the recognized English export layout is tagged
ingested_format: transcript. Its elapsed-time headings and speaker paragraphs
are retained unchanged; the document chunker uses those headings for timed
sections and citation context. The export supplies coarse elapsed points, rather
than a start and end time for each speaker turn. Loom does not infer missing
intervals or use the meeting-ended footer as an utterance endpoint. Edited,
untimed or unrecognized layouts, including meeting notes, remain ordinary Docs
with their content preserved.
The source keeps its existing connected-account and mirrored-access checks. A folder name, transcript shape or attendee paragraph does not grant access.
The recording source accepts Zoom only. A stored Meet source remains listed and
inspectable, and keeps its configuration and documents. Its check, rotate,
sync_now and update operations refuse with BAD_CONFIG and direct the caller
to gdrive. Its worker turn records provider_removed, contributes no pass
failure and preserves the cursor, pending full request and file waits. Removing
the source deletes documents only when requested. No stored configuration is
rewritten to a sentinel, and no native Meet API fallback remains.
Project-tracker issues
Feeding a project tracker’s issues and comments into the tracker plane is a design of its own, written in the pull request that builds it, after the items and access pull requests. That design has to answer what this page found:
- The tracker plane reads a team’s records live and copies none (tracker plane); issues with no team database would be kept by Loom, in its own tables, under the erasure census.
- The plane’s statements filter by
repos(and, in a standard namespace, byis_private = 0), never by org or namespace, so a Loom-kept table needs an (org, tracker namespace) predicate on every read. - A source serves only with
github_issues,github_prsandedges(record contract); the contract’ssource_idnames a repository; and tracker namespaces belong toLOOM_TRACKER_ORGalone. - An issue of a private team, or read through a person’s connected account, is private
(
is_private = 1). The contract derives an item’s id fromsource_id, which changes when an issue moves to another team; the design changes the contract to take the issue’s own UUID, so a moved issue keeps its item.
Surfaces
- Ops.
knowledge.source.addtakesitemsorlinks,credential_modeand, forcrawl,directory_subjectandsubjects;update,parse_links,connection.listandconnection.disconnectare new (a connection starts in the dashboard,POST /v1/connect/<kind>/start);listandinspectshow items, modes and, through the check, the documents each owns. A mirrored source shows its document counts to nobody but a platform admin (exception 4), apart from what a checked leg lists to an admitted reader, and a connected source’s items only to its owner (an admin can still remove the source). To a viewer who is neither the owner nor an admin, a mirrored source shows its kind, its owner as “a connected account” for a connected one, its item count and its last sync’s status, not its items,configorcreated_by; its error text carries codes only, for every viewer. Python and TypeScript SDKs are regenerated. - CLI.
loom knowledge source add|update|list --namespace <ns> --link <url>...(--credential-file,--connected),loom connect gdrive|notion|linear,loom connection list|disconnect,loom knowledge mirror erase --all(restores and rollback), andloom doctor’s line for the oldest listing start of each mirrored source (for a platform admin; a connected owner sees fresh or stale). - Dashboard. The Library’s Sources section: paste many links, a preview of what each resolves to and which could not be read, Check, Add, edit items, the access mode, and each source’s last sync (a mirrored source’s status code only). Settings → Connected accounts: connect, see, disconnect. The repository card keeps its own sources.
- Agents. Every op reaches an agent through
loom_op. The mentor reference gains a line: handed a store link and asked to bring it in, callknowledge.source.updatewithlinkson the caller’s connected source for that store, oraddone; with no connection,connection.listanswers the dashboard link<public base>/settings/connected-accounts?kind=<kind>to hand back.
Data model
ClickHouse, additive migrations:
| Table | Key | Holds |
|---|---|---|
loom.knowledge_sources (columns) |
(org, namespace, name) |
source_kind, credential_mode (default shared), access (default namespace), owner_subject, items_json, options_json (a crawl’s directory_subject and subjects), secret_enc |
loom.knowledge_mirrors |
(org, namespace, mirror) |
deleted (written once a namespace erasure or the mirror-erase command has erased the mirror); at most one live mirror per namespace |
loom.knowledge_mirror_latch |
one row | set_at: the deployment once had the switch on; the mirror-erase command deletes it last. It stays in ClickHouse and holds no memory content. |
loom.documents (column) |
as today; in a mirror source_uri is <address>#<source id> |
access_json: a mirrored document’s permission list (emails, owner, groups, domains, anyone, unread); empty elsewhere; under the Postgres authority it rides in the head’s row jsonb, no migration |
loom.knowledge_source_listings |
(org, namespace, source_id) |
name, started_at (the start of the source’s last complete listing), version; namespace is the mirror |
loom.knowledge_group_members |
(org, namespace, source_id, group) |
name, members, read_at, deleted, version; namespace is the mirror |
loom.connected_accounts |
(org, subject, kind) |
connection_id, account_id, workspace_id, account_email_enc, scopes, credential_enc, active, refresh_id, connected_at, disconnected_at, version |
This is an authentication control registry, keyed by canonical sign-in owner across
namespaces, like the sign-in and revocation registries. It is not a memory-content row:
namespace, agent, end-user user_id and session_id would falsely bind one provider
connection to one requesting session or namespace. Source registrations retain their
namespace and worker provenance; account changes use the authenticated actor in audit.
The explicit owner erasure axis covers sealed identity and credential history rather
than guessing a memory user_id from the sign-in subject. Auth invokes a required lifecycle
port; the HTTP and CLI composition roots bind its knowledge implementation before
accepting mutations. An unbound process refuses retirement rather than skipping it.
The account registry seals its optional display email in account_email_enc. Its
workspace_id distinguishes provider workspaces, connection_id fences replacement
and disconnection, and refresh_id reserves one rotating refresh. Source rows retain
that exact connection id with their canonical owner_subject; created_by remains
audit attribution and supplies no credential authority. Indexed mirror discovery retains
an exact source and account generation for each connected source. Preparing and refreshing
a read plan revalidates the owner’s standing without refreshing a provider token. The
existing document permission list still decides which readers may see each copy; a
verified reader need not own its source. A replacement source is not added to a retained
plan. Worker publication checks owner and source standing again after extraction and
embedding, before committing documents or a complete listing.
The browser routes and account operations implement the connection backend. The dashboard connection page, interactive CLI flow and crawl support require their own integration; the backend does not implement those surfaces. Connect routes are excluded from HTTP tracing before request attributes or headers can be captured.
Account restore cleanup discovers raw account and connected-source history, including
replaced registrations and source-only orphan owners. It stops every proven random
Keeper key under its exact owner lease, queues every historical source id, clears only
matching current registrations, and physically deletes account and source history with
raw residue verification. Ordinary disconnect still refuses another live generation.
The cleanup runs before publication in loom reconcile for either restored store. For a
ClickHouse-only deployment, stop every serving process, restore its configured stores,
then run python dev/restore_connected_accounts.py and repeat with --apply. The script
accepts only the configured deployment and preserves its authority and cell checks;
it does not attach cleanup to the generic loader’s arbitrary database or scratch target.
Recent worker activity refuses cleanup. Local completion precedes bounded best-effort
provider revocation outside leases, and revocation failure cannot restore credentials.
Complete the separate mirror-content restore cleanup before serving; account cleanup
alone does not prove every mirror document, chunk, head and original was erased.
A source N-1 can operate on (shared, access: namespace, with items today’s config
can express) keeps today’s columns as they are: kind, its credential in
credential_enc and its items in config_json, which N writes alongside items_json.
Every other source (mirrored, connected, crawl, or with an item today’s config
cannot express) is stored with kind empty and its kind in source_kind, its
credential in secret_enc (purpose knowledge_sources.secret, which
loom secrets reseal covers) and its items in items_json, with credential_enc empty
and config_json {}. N reads source_kind first and kind when it is empty, and
items_json first and config_json when it is empty (rolling
upgrades). The mirror’s documents and chunks live in
loom.documents and loom.chunks under the mirror’s namespace, so the document
machinery (heads under the Postgres authority, deliveries, originals, the blob sweep)
applies unchanged. Each new table is placed in storage layout by
its placement rule, in the pull request that adds it: loom.knowledge_mirrors and
loom.connected_accounts are registrations read by key, so they follow
knowledge_sources to Postgres in wave 3; the listing and members rows are read by the
mirror legs beside the lists in loom.documents, so they stay in ClickHouse. An
implementing pull request that needs another table (for a stopped source’s cleanup, or a
crawl’s exclusions) adds it with its placement and erasure classification.
Erasure and audit
What every implementing pull request must hold; each specifies its steps, leases, statements and census keys against these, and its tests pin each one:
- Every new table is classified in
server/erasure/store/census.py, and every table that can hold a person’s address or a document’s text is covered by an erasure. - A namespace erasure,
namespace.delete,collection.deleteandcode.repo.removeretire the namespace’s sources first, clearing their credentials, so no turn writes back what the erasure took. Anamespaceornamespace_listerasure covers each named namespace’s mirrors as more names of the same erasure: a scope that holds a mirror is erased as a purge, in one pass over the named namespaces and their mirrors, since no token reads or restores anacl-name, and the census counts it the same way. The last Loom step,knowledge_mirrors.forget, writes the mirrorsdeleted, so the name registered again gets a new mirror; the registry keeps them, so a resume or a replay after a restore still reaches them. Auser_iderasure touches no mirror row (the sync writes them as-), and its census and export name none. - Each namespace step resolves its mirrors once and persists
mirror_markedinloom.erasure_requestsand, under the Postgres authority,loom.erasure_ledgerbefore counting or deleting that scope. Failure-tail verification does the same; if marking fails it counts nothing. The mark only moves from false to true and survives registry loss, resume, abandonment and restore replay. A restore promotion preserves the receipt, checkpoint time and pending-replay fence. Marked checkpoints use the high bit of their UInt64 version; they supersede stale unmarked writers even after a merge, while timestamps order marked writes. - Marked
erasure.requestreplies carry codes only for every caller, since the idempotent answer can be replayed after a role change.erasure.statusanderasure.listcarry codes only unless the current caller is a flat platform admin; namespace and org admins receive zero counts, empty residue, no census and no verification timestamp. These placeholders never prove verified absence. Masking the whole receipt prevents a caller subtracting its namespace’s counts to discover a mirror’s size. Ordinary receipts keep their counts. Marked completion and abandonment audit rows persist no counts. Failed marked receipts and audit rows answerERASURE_FAILEDfor every cause and withhold step progress, so a residue failure reveals no special code or failed stage. Full internal evidence remains available for recovery and authorized platform-admin status reads. - A person’s connected accounts, the sources they own with every mirrored copy those wrote, and their entries in every crawl are erased when an org admin asks; afterwards no source row, list or members row names them, and a crawl reads them again only once an org admin readmits them. The pull request that builds connected accounts and crawls designs that erasure.
- Private canonical-owner cleanup uses the exact
(org, subject, kind)key independently of namespace and end-user erasure. Disconnect and deactivation fence the generation, clear the current credential, queue every owned source for retirement, then physically delete historical provider identity and credentials. They retain only an empty inactive canonical-owner marker. SCIM deletion removes that marker too. The canonical-owner census counts all historical rows, including sealed values; a failed delete, missing census answer or residue refuses completion. Theuser_iderasure scope continues to mean an end-user principal and never guesses a sign-in subject from an id or email. Source retirement remains queued until the existing mirror retirement worker finishes; account cleanup alone does not certify that every source copy or crawl entry is erased. erasure.requestrefuses anacl-scope for every caller but a platform admin with auth on, anderasure.exportrefuses one for every caller (an export would read the mirror’s chunks with no permission check); with auth off both refuse it for everyone.- Audit rows:
add,update,rotate,remove,sync_nowas today, plusupdate’s item changes;connectanddisconnect(subject, kind, scopes, never a credential); a crawl’s subjects (as keyed hashes), granted scopes and member count; a SCIM-driven stop names the sources it stopped.
Rolling upgrades
- Clients. A client built before
mirrored: truedeclares nothing, so a mirror leg answers it nothing until it is upgraded, and the refusal refuses its session-content ops; a third-party MCP client is refused the same way. Mirrored sources stay off (LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED) until the operator has rolled out the plugins, CLI, SDKs and hooks that honour the flag. - N-1 cannot operate on a source it must not. N-1 reads
kind, unsealscredential_encand readsconfig_json, and knows none of the new columns. A source stored withkindempty is one N-1 cannot use: itscheck,rotateand sync turn fail before any request and retire nothing, and N-1 syncs only sources on a trunk namespace. - The new modes wait for N everywhere.
addandupdateanswer 503OVERLOADED(retry_after_s) for a source stored withkindempty while any live process of any deployment on the ClickHouse service reports a schema version below the migration that adds the columns, or no heartbeat has been read, as rule 24 does for namespace guides (rolling upgrades). - Rollback and restores. N-1 has no mirror leg, but its gate admits a wildcard admin
token that names a mirror, its org-wide reads (insights, usage, cost, the analytics
export) include
acl-rows, and nothing on N-1 ages the lists. N reports the condition (loom doctor, and the schema heartbeat check above) when a process below the mirror migration heartbeats whileloom.knowledge_mirrorsholds a live mirror (one not writtendeleted), as a rolling rollback (a Helm rollback included) would start one. So the rollback procedure (upgrades) stops every replica, stops every mirrored source and erases every mirror (loom knowledge mirror erase --all) before any N-1 replica starts. A restore of either store runs the same command before the deployment starts, since a restored copy cannot be trusted to say who may read; owners connect and add their sources again. The implementing pull request adds both steps and specifies the command. - Ownership. In a namespace, one owner per address, found by its tag, is today’s model, so N-1 and N agree on whose a document is; N-1 has no mirror leg.
- Lists. N-1 never meets a list: the build that adds the column writes none and carries none forward, and the first build that writes one runs beside it (rule 53).
meet. N-1 still syncs a storedmeetsource on its turns; N skips it.- Erasure. N-1’s erasure misses the mirror and the new tables (rule 14): run again with N every erasure completed while an N-1 replica served.
Each implementing pull request adds its rule to rolling upgrades,
and pins this section’s N-1 behaviour with a test that runs today’s check, rotate,
remove and sync turn over a row N wrote with kind empty.
Breaks
- Mirrored sources need
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED, which an operator sets once every Loom client and hook honoursmirrored: true. - While the refusal holds, every session-content op, the hook-sent
reads included (
context.pack,code.search,code.impact,memory.for_file,namespace.validate), refuses a client that does not declaremirrored: truein every org: an SDK app that has not opted in, and a third-party MCP client, which cannot. recordingsources lose themeetprovider.- The gate of
add,update,check,sync_nowandremovetakeswriter, withadminchecked by the handler for shared and crawl sources;rotateis refused for a connected source.
Settings
| Setting | Default | Meaning |
|---|---|---|
LOOM_KNOWLEDGE_SOURCE_ACL_STALE_SECONDS |
3600 | a source’s listing or group members row older than this admits nobody |
LOOM_KNOWLEDGE_SOURCE_MAX_DOCUMENTS |
50000 | a source’s stored documents plus the new ones a turn writes, at most 200,000; at it a turn tries no new file |
LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_PERSON |
8 | a person’s connected sources in one namespace |
LOOM_KNOWLEDGE_CONNECTED_SOURCES_MAX_PER_NAMESPACE |
64 | a namespace’s connected sources |
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLED |
false | mirrored sources may be added; set once every Loom client and hook honours mirrored: true |
LOOM_KNOWLEDGE_SOURCE_MIRROR_DOCUMENTS_PER_PASS |
40 | documents a pass fetches, writes and retires for mirrored sources, apart from the namespace sources’ bound |
LOOM_CONNECT_GDRIVE_CLIENT_ID, _SECRET |
unset | the shared-folder store’s OAuth client; unset, those connections are off |
LOOM_CONNECT_NOTION_CLIENT_ID, _SECRET |
unset | the page store’s public integration |
LOOM_CONNECT_LINEAR_CLIENT_ID, _SECRET |
unset | the project tracker’s OAuth application |
The redirect URI to register with each store is
<LOOM_PUBLIC_BASE_URL>/v1/connect/<kind>/callback.
Rejected
- Mirrored documents in the namespace itself, filtered by every read. A missed path would show the document to every reader. The list sits on the document only inside the mirror, which no token opens.
- A namespace per permission list. A list changes with every share, which would move the document, its chunks and embeddings on each change.
- A mirror name derived from the namespace. Anyone could compute it, and a known name makes every error and refusal about it a probe.
- A permission list per reading account within a source. An account that read the list before a reader was removed would keep admitting them until its row aged out. Across sources each copy keeps its own list, bounded by the stale bound and dropped at its own source’s next listing.
- Resolving the store’s groups through Loom’s SCIM groups. They hold a service’s end
users (
user:<id>), not people’s emails. - A separate meeting-store path. The meeting store already saves transcripts and notes to the shared-folder store, with the participants’ permissions.
drive.filewith the picker. It cannot read a folder’s later files.- Incremental listings for mirrored sources. A sharing change does not move a file’s modification time.
Invariants
- A mirrored document is read only through a leg that applies the permission check; no
token reads or exports a mirror, and
erasure.requesttakes a mirror’s name only from a platform admin, with auth on. - Every
crawlandconnectedsource is mirrored. - Every permission the sync cannot trust admits nobody, and so does an
unreadlist (who may read). - Every complete listing of a mirrored source reads every file’s list again, and only a complete listing’s start vouches for the lists.
- No caller-supplied host or query string reaches a store.
- A connected source stops on every path that ends its connection or its owner’s standing (connected accounts), and its mirrored documents go with it.
- Nothing derived from a mirrored document leaves the mirror except as the four
exceptions allow, and no mirrored source exists until
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLEDis set. - N-1 never operates on a source it cannot honour: such a source is stored with
kindempty.
Source entry points
server/knowledge/domain/sources.py: kinds, configs, items.server/knowledge/domain/source_links.py: the link parser.server/knowledge/app/sources.py: the ops.server/knowledge/app/source_sync.pyand its kind modules: the sync.server/knowledge/app/source_access.py: the permission check and the mirror legs.server/knowledge/app/connection_routes.py: the browser consent routes.server/clients/gdrive.py,notion.py,linear.py,recording.py: the stores.server/clients/connected_oauth.py(OAuthClient): fixed-provider consent, code exchange, refresh, account metadata and revocation requests. Calls share the caller’s deadline, cap response bodies, refuse redirects and never retry. The caller consumes browser-bound state before exchange, authorizes the Loom owner/source and commits refresh generations; this HTTP helper neither supplies nor bypasses those checks.
Tests
Each implementing pull request lists its own; these pin the model:
- Unit:
- Items, links and the sources’ ops:
- The link parser on every link form and every unreadable one.
- Item validation.
- The query builder escaping every field.
- A non-owner admin’s
listandinspectof a connected source showingconfig{}. - A connected source’s items, and its document counts beyond those a checked leg
lists to that admin when admitted, hidden from an admin who is not its owner; a
non-owner admin’s
checkof a connected source answering no item ids. - A failed read of the reader’s canonical subject or emails giving the same marker whether or not a candidate exists.
- A mirrored source’s probe (
check, and theadd,updateandrotateprobes, by a non-owner admin and a crawl registrant), status,removeanswer and a namespace erasure’s receipt carrying no count, title, file id or address of what the mirror holds. - A crawl
updateprobe by any caller answering the same code whether or not the file exists. - A crawl’s status carrying no
item <n>code and nomeet_recordings: not found. - A non-owner admin’s
removeof mirrored sources of 10 and 10,000 documents leaving the same request-side figures. addof a removed mirrored source’s name answering as for a fresh name.- A mirrored
addduring an open namespace erasure refused withERASURE_RUNNING. - The mixed-version refusal, and N-1’s
check,rotate,removeand sync turn over a row N wrote withkindempty. - A crawl
updateby anadminwith an org-wide role who is not the registrant, adding afileorfilteritem, accepted whether or not any crawled person can read it.
- The mirror and its gate:
- The
acl-gate incheck_authorizedandCachedGate, with auth on and off, and for a platform admin on a read op. namespace.registerandnamespace.restorerefusing the form.erasure.requestanderasure.exportrefusing anacl-scope with auth on and off.row_reachandaudit.listrefusing anacl-name with auth off.row_reach,row_in_scopeandnamespace_for_idopening nothing of a mirror with auth off.erasure.listanderasure.statustreating anacl--scoped saga as not found for an org admin.erasure.statusanderasure.listby a viewer, and a saga that fails on mirror residue, showing no mirror count.resolve_org_scopewith auth off,every_namespace_scope,rollup_member_namespaces,sql_v1_agent_eventsunder a*grant and the analytics export leaving out everyacl-name.- The insights agents and cost reads leaving
acl-names out. audit.listafter a connectedadd, by an admin who is not the owner, showing no item id other than its keyed hash.erasure.exportand auser_iderasure’s census naming noacl-row.- No row carrying a reader’s
user_idunder anacl-name after an admitted reader’sget,searchandask. - An erased namespace’s name registered again with a new mirror.
- N reporting a below-schema heartbeat while a mirror holds rows.
- The
- The list and who may read:
- The access check on fresh and stale listings,
unread, group, stale group, domain, link-only, anyone and unknown permissions, and for a delegated token, a bot and an end-user token. - An email-shaped subject with no verified directory email, and an inheriting
service account, seeing no
emailsdocument. - A group matched only through the owning source’s members row.
- A connected source’s unreadable-list entry admitting the owner through the list’s
ownerentry, never an email or the connection’s store email. - A crawl
fileitem of a link-only file admitting no crawled person. - A verified email at a domain without the identity provider’s domain claim refused
a
domainsdocument. - A crawled person who opened a link-only file only through its link not admitted.
- A deactivated team member admitted by no
linearlist. - Against an
emailsdocument: a person’s own session token and areads_as_creatoragent token admitted, a Settings-made service-account token (withcreator), a delegated child of areads_as_creatortoken other than the assistant’s answer token, and a request on behalf of another person not. - A permission the store marks deleted, whose address now belongs to another verified user, admitting nobody.
- An answer token delegated from the listed person’s session token admitted, an
answer token delegated from a
reads_as_creatortoken admitted as its creator, and a token delegated from an answer token refused. - An assistant answer token delegated from a Settings-made service-account token
refused an
emailsdocument. - A token with consumer
loom-assistantand anassistant-answer-id but no reads-as claim reading as nobody. - A caller of
assistant.ask,loom_assistant_askor the chat route that does not declare the flag getting no mirrored text; anassistant.askand a chat-route answer on a namespace with an active mirrored source and no admitted item answeringmirrored: true, the chat route in its stream’s first event and, unstreamed, in its response’s first field; a plugin session flagged once itsloom_assistant_askanswer carries the flag; a markedassistant.askand chat-route row folded and masked like a marked tool-call row. - A
knowledge.search,context.packandmemory.askon a namespace with an active mirrored source and no admitted item answeringmirrored: trueas the first field; the same reads after the last mirrored source is removed, and while the only one listed is inactive (its owner gone, its cleanup pending), answering no flag and running no mirror leg. - A marked
memory.askwhose candidates are all hidden making the same model and embedding calls as one with no candidates. - A caller admitted to nothing seeing the same
cost.budget_statuschange over 10 and over 10,000 hidden matching documents. - A file another mirrored source holds, added by a second source, written as that source’s own copy with its own list, so the adder sees what they would with no other source.
- A hidden mirrored chunk holding the query term leaving an admitted hit’s
lexical_scoreand rank unchanged. - A namespace whose documents are all mirrored ranking by both legs.
- The access check on fresh and stale listings,
- The check fails closed:
- A leg that cannot read the listing row or the owning source’s registry row showing
no mirrored document, marked
degraded: mirror_unread, and one that cannot read the lists, the members rows or the heads dropping mirrored results without the marker. - The same
degradedmarker for a query that matches a hidden document and one that matches nothing while a mirror read fails. - A failed list, members or heads read with one hidden candidate answering exactly as a query with no candidate: no marker.
- A leg on a lagging replica that sees a fresh listing row and an older list admitting nobody.
- A document missing from a complete listing admitting nobody once its listing row lands.
- A stopped source’s documents admitting nobody before their retirement.
- A leg that cannot read the listing row or the owning source’s registry row showing
no mirrored document, marked
- Every complete listing reads the lists again:
- A turn that finds a narrowed list rewriting the row without rebuilding a chunk, and one that finds the same list writing nothing.
- A listing row landing only after its changed lists.
- A fetch the pass bound deferred writing no list older than the listing row that lands after it.
- After a person’s erasure, no source’s next turn writing a list or members row that names them.
- A namespace source’s per-pass progress unchanged by a mirror’s size.
- A large removal leaving another mirrored source’s content fetches unchanged.
- A mirror’s originals surviving a blob sweep and its chunks re-embedded after a model change.
- The rule and its exceptions:
- Two readers of one namespace, one admitted to a mirrored document, through the
repository-docs listing cache and
memory.ask’s precedent fast path: the other gets none of it. - A request a mirror leg answered keeping no argument or result-side field the leg
shaped in its
agent_eventsrow. - A marked request’s trace and exposure rows unchanged by a hidden mirrored hit.
- A follow-up query a mirror leg shaped leaving the embedding cache untouched.
- A generation over a mirrored chunk capturing no text.
- The sync’s audit rows, status, pass notes and log lines naming no mirrored file id or address.
- No reader but a platform admin reading the work figures on a marked row.
- A viewer’s cost or insights read showing no per-request token spend of another person’s marked request.
- A viewer’s
cost.summarygrouped byuser_idandoperationshowing no single marked request’s tokens. cost.budget_statusandcost.summaryby a viewer reading the same before and after a mirrored sync pass.- A namespace-scoped SQL user reading
sql_v1_agent_eventsandsql_v1_usage_daily, and the analytics export’susagerows, after another person’s marked request, finding its spend and work only as the fold and the mask allow. - A mirrored source’s document counts shown to nobody but a platform admin.
knowledge.readsafter an admitted reader’sknowledge.getof a mirrored address not naming it, nor a namespace document’s address read in the same marked request.- The dashboard’s
knowledge.getand the assistant’smemory.askof an admitted mirrored document answered withmirrored: true. - A later turn of a session that saw
mirrored: truenot captured, and a client that does not declare the flag answered nothing by a mirror leg. - A capture hook recording no turn whose answer carries
mirrored: true. - A plugin session over a collection with an active mirrored source and no admitted hit flagged all the same, its next turn not captured.
loomCLI output holding an admitted mirrored item, and a subagent’s mirror read, not captured into the parent session.- A
loomCLI run through a shell and a prompt-recall injection carryingmirrored: truestopping every hook of the session. - The CLI’s
ingest-transcriptrefusing a transcript that carries the flag. - A dashboard view of an answer carrying
mirrored: true, with or without an admitted item, sending noui.vitals. - A forked or resumed session not captured after its parent saw the flag.
- One test over every op the registry classifies as session content, each refusing a
client that does not declare the flag from the first time
LOOM_KNOWLEDGE_MIRRORED_SOURCES_ENABLEDis set until it is off again and the mirror-erase command has run, in every org, with the switch on and off; and a test that collects every op the shipped hooks, their shared library and the CLI send without the agent choosing the content and asserts each is classed as session content. - A session with clients holding tokens for two orgs, one reading a mirror, capturing nothing into the other.
- The session-content refusal holding once the switch was set with no mirror yet, when the deployment row cannot be read and on a replica whose view lags it, and lifting only once the switch is off and the mirror-erase command has deleted the row.
- A mirror-erase run with the switch still on refused, a restore that loses the
deployment row, and a mirrored
addafter either, with the refusal still holding. - A viewer’s
insights.queryoveranalysis_agent_eventsafter another person’s marked request finding its tokens, cost,modeland outcome fields only as the mask allows, the caller’s own rows included. - A caller admitted to nothing seeing no work figure on its own marked row over a mirror of 10 and of 10,000 hidden documents.
- A checked leg’s every answer field unchanged by a hidden mirrored document.
- The mirror-erase command cut short after deleting mirrors but before the latch row, with the refusal still holding.
- An assistant memory citation from a marked
memory.askreturnedunchecked. - A viewer’s
cost.budget_statusread before and after another person’s marked request differing only by that request’s folded spend, as exception 2 accepts. - The folded per-namespace-day row’s
errorscount unchanged by a marked request that fails in the model call. - The session-content refusal of a hook-sent
context.packfrom a client that does not declare the flag. - An admitted mirrored item in
context.packwriting no recall credit or exposure row. - A later
ingest-transcriptof a session flagged only by inheritance refused from an unflagged shell. - A session started from a flagged session’s process (
claude -pthrough its shell) flagged through the inherited marker, and a hook’scode.searchandcontext.packsending nothing after the flag. - The owner’s
reads_as_creatortoken admitted to the owner-only list of a connected source; an on-behalf request for the owner and a token minted for the owner’s subject withoutsidrefused. - A
claude -pchild that read a mirror flagging its parent session, whose hooks then send nothing. - A namespace copy winning a dedupe carrying only its own chunks and scores; a
reader only the namespace copy admits keeping it; a reader whose query matches
only the mirror copy’s revision getting that hit with
mirrored: true. - A restore that drops a
loom.knowledge_mirrorsrow whileacl-rows stay in the documents, chunks, heads and bucket, thenloom knowledge mirror erase --alldeleting every one of them before the latch. - A non-owner admin’s
sync_nowon a mirrored source with a refused item answering as on a clean source.
- Two readers of one namespace, one admitted to a mirrored document, through the
repository-docs listing cache and
- Items, links and the sources’ ops:
- Integration:
tests/integration/test_document_source_leakage.py, besidetest_cross_principal_leakage.py: for every leg with the check and every id-addressed op, a reader the list leaves out sees no mirrored document, its title, its address, its count or its existence, through the legs,knowledge.reads,audit.list, the org-wide rollups or a source’s status (an admin who is not the owner included), under both storage authorities; a reader the list names sees it; a reader removed from the store’s list stops seeing it after the next listing of every source that holds a copy; one mirrored source’s retirement leaving another source’s copy of the same address readable; asharedmirrored source’s status, read by a viewer no list names, showing no per-file code. - Integration: a
connectedsource is mirrored wherever it writes; a SCIM deactivation and a disconnect stop the person’s sources and retire their mirrored documents;namespace.deleteandcollection.deleteretire the namespace’s sources and clear their credentials.