LiveCEO · internal research · 2026-07-22

Before you turn the traffic on

Can the product survive ~30 new organizations self-serving their own brains — and what has to change inside an org so users stop guessing? A multi-agent audit of the real code and the live production system.

37 agents · 1,278 tool reads every severe claim adversarially verified in code live read-only checks of prod server + DB 20 severe findings confirmed · 0 refuted

Not yet. You are roughly two focused weeks away — and the work is concrete, not architectural.

Nothing about the core product is broken: the onboarding funnel is transactionally correct, tenant isolation is real, the failure-recovery UX is genuinely good. What breaks is everything that is shared: one Anthropic subscription powers every org's chat and every wiki build; one serial worker indexes every org one at a time; one 2-vCPU / 3.7 GB box runs the entire company; and nothing gates who signs up, meters what they consume, or tells a human when it falls over.

With today's system, a 30-org wave plays out like this: most orgs wait hours for a brain they were promised in "5–15 minutes"; every org past ~#5 in the queue gets permanently stuck with an error message that falsely promises an automatic retry; as the cohort goes live, 30 always-on agent containers eat essentially all the RAM the box has; and the free signups' Opus curation shares one subscription with your paying customer's chat — a subscription that already had two multi-hour outages in 8 days at today's load. Meanwhile no one is paged, no user gets an email, and the funnel leaks silently — as it already does today (2 of your 5 current orgs are stuck at zero, unnoticed for 13 and 4 days).

Every mechanism in this report is confirmed in the code with file-and-line evidence; the worst ones come from incidents your own code comments already recorded.

The math that decides everything

2 vCPU · 3.7 GB
the entire production stack — app, ingester, Caddy, every org's agent container
live: nproc / free -h
5 orgs · 2 stuck
current tenants; 40% never connected a source and nobody noticed
live DB, idle 13 d and 4 d
86 min
one observed full reindex (promised in UI: "5–15 minutes")
live journalctl · IndexingProgress.tsx:79
~20.5 h
serialized queue time for 30 first builds at the measured ~41 min worst case
refresh-slack.mjs measured comments
185 min
claim-window TTL — orgs deeper than ~#5 in queue expire and die permanently
db.ts:864–878 · 1082–1084
0.8–1.7 M
Opus tokens one free signup's first wiki build can consume
wikiCurate.ts pass A+B math
2 outages
multi-hour 429/529 incidents in 8 days on the shared subscription — at 1× load
anthropicRetry.ts header, 07-09 & 07-16
0 · 0 · 0
paging channels · outbound emails the product can send · offsite backups
repo-wide dependency + script sweep

What breaks with 30 new brains — 7 root causes

48 findings were raised across eight audit dimensions; all 20 blocker/high claims survived adversarial verification. Deduplicated, they collapse into seven root causes. Fix these seven and the finding list mostly evaporates.

Blocker quotas · chat · costeffort: days

1 · One Anthropic subscription is the platform's single heartbeat

Every org's chat agent container mounts the same host credential; every wiki-curation pass (Opus, up to ~192k tokens per call, 4–9 calls per build) resolves the same credential; so do artifact generation and media description. There is no global concurrency cap — only a per-org cap of 10 turns, so 30 orgs could legally put ~300 concurrent turns plus a queue of Opus builds onto one subscription. The chat path has no retry at all (the carefully built anthropicRetry.ts protects only indexing); a 429/529 surfaces to the user as a raw SDK error over SSE. This is not hypothetical: the retry module's own header documents two multi-hour capacity outages in 8 days caused by a single customer at current load. Free signups and your paying anchor customer share one blast radius.

Evidence (verified file:line)

deploy/agent-lifecycle/start-agent.sh:97–100 — every agent container mounts the same host secret; deploy/agent-runner/runner-server.mts:54 reads it for every tenant. src/lib/anthropicClient.ts — "shared Anthropic credential resolution for every app-process LLM call site." src/lib/wikiCurate.ts:746–872 — Opus by default, 32k max output per call, ~160k-token corpus re-sent once per entity type. src/lib/rateLimit.ts:12 — the only throttle anywhere is per-org. src/lib/anthropicRetry.ts header — documents the 2026-07-09 and 2026-07-16 outages; grep confirms it is used only by the indexing path.

Fix
  • Global cross-org admission cap on in-flight turns, sized below the observed 429 threshold.
  • Give the chat path the same classified retry/backoff the indexing path has, with a friendly "high demand — retrying" SSE state instead of a raw error string.
  • Move wiki curation to a metered API key, separate from the subscription that serves live chat — then a signup burst structurally cannot degrade Vortex's chat.
  • Read the actual subscription tier/limits from the Anthropic console before trusting any of the above sizing (minutes of work, still unverified — see caveats).
Blocker indexing queueeffort: ~1 day

2 · The queue permanently kills any first-time org it can't reach within 3 hours

Indexing is one serial worker draining a strict FIFO. Each queued run's claim nonce expires 185 minutes after minting, and the worker only claims unexpired runs. At the measured ~41 min per first build, org #6 in the queue has already expired before the worker gets to it. The reaper then marks it failed with UI copy that promises "Retrying on the next scheduled refresh" — but the auto-retry query requires last_indexed_at IS NOT NULL, which structurally excludes every never-indexed org. The org is stuck forever unless a human finds the Retry button; retry re-mints a nonce at the back of the queue. Queue starvation already produced two real incidents at n=2 orgs (2026-07-14, 2026-07-20 — both documented in code comments).

Evidence (verified file:line)

src/lib/db.ts:864–878 (185-min TTL), 961–1019 (FIFO claim, unexpired only), 1082–1084 (the NULL exclusion), 1223–1225 (the false-promise error string, rendered verbatim by IndexingFailed.tsx:123); src/lib/indexingLiveness.ts:41–47; deploy/ingester/ingester.mjs:1171–1260 (single-worker loop). Incident notes: indexingLiveness.ts and scripts/trigger-reindex.mjs:55–63.

Fix
  • Auto-re-enqueue a QUEUED run whose nonce lapsed from queue depth (nothing died — mint a fresh nonce) instead of terminal-failing it.
  • Let never-indexed failed orgs into listRefreshCandidates so the promised auto-retry is actually true.
  • Scale nonce TTL with queue depth; later, add a second ingester worker — claimPendingIndexing already uses FOR UPDATE SKIP LOCKED and its comment says it was built for exactly that.
Blocker signup · abuse · billingeffort: 1–2 days

3 · Nothing gates signup, nothing meters usage, and the schema can't even say who pays

The onboarding page says it in plain text: "Anyone can create a workspace." Any free-email address gets an org, a Composio project, a subdomain, an Opus-powered wiki build and unlimited chat — no approval, no CAPTCHA, no per-IP rate limit, no cooldown on re-triggering full rebuilds (any member can hammer the most expensive operation in the system back-to-back), no daily message budget, and — confirmed across the entire 1,351-line schema — no billing, plan, tier, or quota column anywhere. There is no way to even express "Vortex is the paying customer, protect it." The two drive-by orgs already sitting in prod show this door is already being walked through.

Evidence (verified file:line)

src/app/onboarding/page.tsx:19–21; src/lib/publicEmailDomains.ts (individual track); scripts/init-db.sql — zero quota/billing/tier columns (confirmed by full read + information_schema query); src/app/api/indexing/start/route.ts — refuses only when no sources or already running, no cooldown; src/lib/rateLimit.ts — concurrency only, no volume caps; no middleware.ts, no signup rate limit.

Fix
  • Daily org-creation cap with alerting; past N per day, signups land on a waitlist instead of provisioning.
  • Per-org reindex cooldown and a daily chat budget with a clear user-facing message, enforced server-side before the shared credential is touched.
  • A minimal organizations.tier column (anchor | trial | standard) that the rate limiter and queue ordering key off — the smallest change that lets you protect the paying customer specifically.
  • Reserved-slug blocklist (www, api, admin, status, billing…) — today any org can claim them as subdomains.
Blocker observability · backups · deployeffort: hours–days

4 · When it breaks, no human finds out — and there are no backups

There is no paging or alerting channel anywhere: Chief is an automated issue-fixer that runs on the same box, so a dead host reports nothing. The product cannot send a single email — no mail dependency exists in package.json at all, so no "your brain is ready," no failure notices, and teammate "invites" are silent DB rows nobody ever learns about. There is no offsite backup: no pg_dump for Neon in the repo, and the on-disk wikis' only "backup" is a .bak sibling directory on the same disk, which the org-wipe script deletes together with the primary. And every deploy is a hard cutover — docker rm -f, no health gate (raw 502s, all live chat streams dropped) — that also unconditionally restarts the ingester, discarding up to 160 minutes of whatever build was mid-flight.

Evidence (verified file:line)

Repo-wide grep: no pagerduty/twilio/sendgrid/resend/nodemailer/SES/webhook-alert dependency or call. docs/chief-logging.md:20–23 — Chief's own scope limits. deploy/redeploy-app.sh — rm-then-run, no readiness probe; Caddyfile.template reverse_proxy has no health checks; the same script restarts liveceo-ingester unconditionally. Wiki snapshot: refresh-slack.mjs:152–171; scripts/wipe-server.sh:20–22 deletes primary + .bak together.

Fix
  • Hours, do first: an external uptime probe from any other host hitting liveceo.ai every minute, wired to Slack/SMS. This one piece turns "crashes loudly" into "someone actually finds out."
  • A minimal transactional-email lane with exactly two events to start: build-succeeded, build-failed.
  • pg_dump cron + rsync of /srv/liveceo/wikis to another location.
  • Deploy: build and health-check the new container before removing the old one; skip the ingester restart while a run is CLAIMED.
High capacity · memoryeffort: 1–2 days

5 · The box cannot hold 30 ready orgs

Every org that reaches "ready" gets a permanent, always-on agent container (2 GB cap, no CPU limit) that idles at ~115–120 MB RAM and 7–8% CPU. Thirty more is ~3.45 GB of idle residency on a 3.7 GB machine — before the app, Caddy, the ingester, and any actual chat load. The likely failure order: the ingester (the thing every new org is waiting on) gets OOM-killed or starved exactly during the onboarding wave. Your own runbook already called this: "escalate isolation at ≈20 tenants," idle-reaper "when it matters." It matters now. Also: the disk is 82% full, and 22.7 GB of it is never-pruned Docker build cache.

Evidence (verified + live)

deploy/agent-lifecycle/start-agent.sh:88–115 (2 GB, no --cpus, restart unless-stopped); live docker stats on prod; deploy/agent-isolation.md:186–195 (the ≈20-tenant note and missing idle-reaper); live df -h / docker system df -v (38 GB disk, 6.6 GB free, 22.67 GB build cache); no prune step in any deploy script. Also confirmed: the in-container runner accepts unlimited concurrent /run requests (runner-server.mts has no semaphore) inside a 2 GB / 128-pid box.

Fix
  • Ship the idle-reaper: stop agent containers after N idle minutes, lazy-start on the next chat turn. Single highest-leverage capacity fix.
  • Add --cpus caps; add a small in-container admission limit (3–4 concurrent turns) in the runner.
  • docker builder prune in the deploy script + a disk-usage alert at 85%.
  • Honestly: also just resize the Hetzner box before the campaign. It is the cheapest line item in this entire report.
High activation · truth-in-UIeffort: ~1 day

6 · Time-to-first-answer: promised in minutes, delivered in hours

The wait screen tells every new org "This usually takes 5–15 minutes" — a hardcoded string. Reality on the live server: 86 minutes for one routine reindex, ~41 minutes measured worst-case first build, 160-minute hard ceiling, all through one serial queue that a cohort shares. The status API cannot even distinguish "queued behind 20 orgs" from "actively building" — no queue position, no ETA, and (because no email exists) no way to tell the user when it's done even though the UI invites them to leave. This is the activation moment of the entire product, and it's the least honest screen in it.

Evidence (verified file:line)

src/components/readiness/IndexingProgress.tsx:79 (copy re-verified in the worktree today); duplicated in IntegrationsView.tsx:141; src/app/api/org/status/route.ts returns no QUEUED/CLAIMED distinction; queue depth is knowable server-side from the claim table. Live 86-min build: journalctl timestamps 06:00:28→07:26:35.

Fix
  • Expose queue position + rolling-average ETA in /api/org/status; replace the fixed copy with a state-aware one ("You're #3 in line · builds are averaging ~40 min today").
  • "We'll email you when it's ready" + the actual email (root cause 4's lane).
High refresh schedulingeffort: hours

7 · Every org refreshes at 06:00 UTC into the same serial queue

Every org is created with the identical cron default 0 6 * * * and nothing ever staggers it. Today's 3 orgs already serialize ~90–100 minutes of daily refresh; curation reprocesses each org's full accumulated corpus every run, so per-org cost grows over time. Past ~30 orgs, the daily herd can outrun the day — and because an unfinished queue just carries into the next 06:00 tick, staleness compounds silently with no metric or alert watching it.

Evidence (verified file:line)

scripts/init-db.sql:30 (shared default; grep confirms no other writer); deploy/refresh-scheduler/liveceo-refresh.timer (per-minute tick); src/app/api/internal/build-wiki/route.ts:815–823 (full-corpus curation each run); db.ts:1071–1094 (mid-build orgs skipped, so backlog defers rather than displaces).

Fix
  • Stagger refresh_schedule at org creation (hash of org id → minute of day).
  • Emit queue-depth at each scheduled tick as a metric with an alert threshold.
Sharp edgessmaller but real

Four more confirmed issues that don't need their own card: (a) the Composio project-creation call in signup has no retry/timeout and its failure leaves the user on a permanently stuck "Creating…" spinner (composio.ts:89–121, OnboardingForm.tsx:114 — hours to fix); (b) the pg pools have no connectionTimeoutMillis, so pool exhaustion hangs requests silently instead of failing loudly (db.ts:50–62 — hours); (c) chat turns have no wall-clock timeout, so a runaway turn holds its concurrency slot indefinitely; (d) per-subdomain Let's Encrypt issuance happens invisibly inside Caddy — a rate-limit rejection would surface as a bare TLS error to the customer with nothing logged anywhere (Caddyfile.template, tls-check — a day to instrument).

The do-before-traffic checklist

Ordered by leverage. Tier 0 is the honest minimum before pointing any campaign at the signup page; Tier 1 lands while the first cohort is arriving; Tier 2 follows fast.

Tier 0 Before any traffic — ~one focused week

  1. hoursExternal uptime probe → Slack/SMS paging, from a host that isn't the prod box. Everything else assumes someone finds out when things break.
  2. daysGlobal Anthropic admission cap + retry/backoff in the chat path (mirror anthropicRetry.ts). Root cause 1 — protects every tenant at once.
  3. daySeparate metered API key for wiki curation. Free-signup build cost can no longer touch paid chat availability.
  4. dayFix the queue-death path: re-enqueue expired QUEUED first builds; admit never-indexed orgs to auto-retry. Root cause 2 — stops permanent stranding.
  5. daySignup cap + waitlist past N per day. Converts an uncontrolled burst into a controlled ramp.
  6. 1–2 dAgent-container idle-reaper + --cpus caps; resize the Hetzner box. Root cause 5 — RAM headroom for the wave.
  7. hoursdocker builder prune in deploy + disk alert at 85%. 22.7 GB reclaimable today on an 82%-full disk.
  8. hoursStagger per-org refresh crons. Root cause 7.
  9. hoursconnectionTimeoutMillis on both pg pools. Exhaustion fails loudly instead of hanging.
  10. dayQueue position + honest ETA in the wait screen (kill "5–15 minutes"). Root cause 6 — the activation moment.

Tier 1 While the first cohort lands — week two

  1. daysTransactional email lane: build-ready, build-failed, teammate invites, "still not set up?" nudge. Closes the biggest activation leak (already visible at n=5).
  2. daysPer-org budgets: reindex cooldown + daily chat cap with clear messaging.
  3. dayorganizations.tier column; rate limiter + queue ordering respect it (anchor first).
  4. dayOffsite backups: pg_dump cron + wiki rsync to a second location.
  5. daysDeploy health-gate (warm swap) + never restart the ingester mid-build.
  6. hoursComposio retry/timeout in signup + fix the stuck "Creating…" spinner.
  7. daysSecond ingester worker (locking already supports it) — halves queue math.
  8. dayRunner in-container admission cap + per-turn wall-clock timeout.
  9. dayLog/alert ACME issuance outcomes; pre-warm certs for known signups.

Tier 2 Fast follow

  1. The in-org UX pack below — one coherent pass.
  2. Reserved-slug blocklist; "your teammate already created this workspace" redirect on signup conflict.
  3. Restrict /api/internal/* to loopback at Caddy (defense-in-depth over the single shared secret).
  4. Activation analytics: scheduled report of orgs idle N days with zero sources (found by hand this time).
  5. Automate full org deletion (DB + wiki dir + container + Composio) to actually meet the privacy policy's 30-day promise.
  6. Telegram-personal ingestion: counterparty-consent screen + per-dialog picker + gate behind access management.
  7. Isolation escalation per your own runbook (Kata at ~20 tenants); confirm Recall/WorkOS/Composio plan limits with vendors.

Inside the org: stop making users guess

The audit's blunt summary: the product has eight top-level areas and zero guidance — no tour, no tooltips, no contextual help anywhere in src/components. Two entire features are invocable only by typing magic phrases into chat. Discovery currently depends on curiosity-clicking rail icons. The concrete pass, ordered by first-session impact:

src/components/EmptyState.tsx · the first screen every ready org sees

Teach the invisible features in the suggested questions

All four suggested chips are generic Q&A. Dashboards and Alerts are conversational-only features ("show me X as a chart", "alert me if Y") — a user who doesn't guess the phrasing never finds them. Add one dashboard-shaped and one alert-shaped chip, and a conditional dataset-question chip when the org has registered datasets. Add a small "what's here" strip naming the eight areas — none exists today.

src/components/readiness/IndexingProgress.tsx · the activation wait

Make the wait honest and self-updating

Replace "This usually takes 5–15 minutes" with live queue position and a rolling ETA; say "we'll email you when it's ready" — and mean it. Dashboards/Alerts empty states currently tell a not-yet-indexed user to "ask in chat," which is impossible; make them indexing-aware with a link back to the progress screen.

src/components/Sidebar.tsx:66–78 · chat/[chatId] route

Close the gate the sidebar walks around

"New chat" bypasses the readiness guard entirely: a never-indexed user lands on a fully working composer, types their question, sends it — and only then gets a 409 error bubble. Disable the button pre-readiness, or run the guard on the chat page itself (the gate's own comment says the UI check is supposed to exist).

src/components/settings/SettingsView.tsx · 2,508 lines, one scroll

Split Settings; rename the three "Telegrams"

Nine unrelated cards in one anchor-less scroll, including three different features all labeled Telegram — one of which silently ingests a member's private 1:1 chats. Split into tabs (Workspace / Access / Integrations & Bots / Domains) and rename the privacy-sensitive one so it's unmistakable: "Telegram — ingest my private chats." The "Members" card is actually an email allowlist, not a roster — label it as what it does.

src/lib/sessionUser.ts:99–120 · SettingsView.tsx:114

Tell founders that everyone is currently an admin

With access management off (the default for every new org), every invited teammate can edit the email-domain allowlist, disconnect sources, and register domains — and the UI never mentions role separation exists (the section simply disappears when the flag is off; only an ops script can enable it). Render a locked "Restrict who can manage this workspace" card, and ship a self-serve toggle — 30 concurrent new orgs cannot run through an ops-script activation path.

src/components/integrations/IntegrationsView.tsx

Curate the catalog toward "feeds your brain"

The connect step drops users into a 1000+ toolkit catalog where connecting many of them does nothing for the wiki — and the user only learns that afterward. Pin a "Recommended for your knowledge base" section (the curated && ingestible flags already exist server-side), badge ingestible sources pre-connect, and add a persistent connect → build → ask step indicator across the first-run screens.

layout.tsx, EmptyState, Composer, ChatView, Sidebar, NeverIndexedGate, global-error

Finish the rename, send the invites, nudge the stalled

The internal name "MainBrain" still fronts most of the UI while login and the privacy policy say LiveCEO — one find-and-replace pass. Teammate invites are silent DB rows: send an actual email with the workspace link. And orgs that sit never-indexed for days (2 of 5 today) should get one nudge back to /integrations. Also: the per-person access console (/settings/access) and the Google Meet admin wizard are fully built but reachable only by typing the URL — link them or hide them deliberately.

Honest take

What's genuinely good — and verified, not flattery: the onboarding transaction is properly hardened (anti-hijack domain derivation, orphan-safe external provisioning); the readiness state machine is honest; the failure/retry UX is better than most mature products; isolation is real (gVisor, egress proxy, RLS, a properly gated TLS endpoint); and your discipline of writing measured numbers and post-mortems into code comments is why this audit could be this precise. The no-fallbacks doctrine did its job: failures are loud and debuggable.

But you don't have a scaling problem — you have a sharing problem. The subscription that pays for free users' builds is the same subscription that keeps your paying customer's chat alive. Your cost model and your availability model are one object. That's the thing to separate before marketing spends a single click, and it's days of work, not a rewrite.

"Will it break?" is the wrong question — it will break politely. Loud crashes for users, silence for you: no pager, no email, no backup, no funnel metric. The most honest datapoint in this whole audit is that 40% of your current signups are already stuck at zero and nobody noticed for two weeks. Traffic doesn't create that problem; it multiplies it by ten.

Your own repo predicted this. The runbook says escalate isolation at ~20 tenants and add an idle-reaper "when it matters." The retry module documents the subscription falling over twice at 1× load. The incident notes document queue starvation at two orgs. The codebase has been telling you its limits; 30 orgs is past most of them.

Recommendation on the campaign itself: don't make the first cohort fully self-serve. Cap provisioning at ~5 orgs/day with a waitlist ("we onboard a handful of companies a day") — it reads as exclusivity, costs one day of code, converts the burst into a ramp, and buys the queue fixes time to land. And before strangers arrive, take the privacy items seriously: Telegram-personal ingests other people's private messages into a company-readable wiki with no consent step, and the privacy policy promises a 30-day deletion that currently requires a human running a four-system manual runbook. Friendly beta users forgive that; strangers' lawyers don't.

Caveats — what this research is not