Async MCP server with job queue: design-system extractor for Cursor
A naive MCP server holds the connection while it works, so long operations hit timeouts. An async MCP server drops them into a job queue and returns a job ID immediately: browser automation, ML inference, and page crawling scale horizontally while the client polls by ID.
Problem
A naive MCP server does its work synchronously: the agent calls a tool and waits for the response. Fine for fast operations. For long ones — browser-automation site analysis, ML inference, page crawling — it blocks: client-side timeouts, no horizontal scaling, a single heavy request holding the connection open. Analyzing a site with Playwright takes tens of seconds; an MCP client won't wait that long.
Methodology
Separate submit from result: an async job queue with poll-by-ID.
-
1. Submit returns a job ID immediately
(
202), without blocking. Status is polled with a separateGET /job/:idcall. -
2. Playwright in a container does the actual analysis
(design tokens: colors, typography, spacing, components) — isolated, headless, with
--no-sandboxunder the container user; the host'snode_modulesis explicitly excluded from the image. - 3. BullMQ on top of Redis — job persistence, retries, and concurrency limits. Not a hand-rolled queue: retries, backoff, and retention come out of the box.
- 4. MCP server on 3001, separate from REST on 3000: multi-protocol (HTTP / HTTPS / stdio) for different deployment scenarios and Cursor integration.
The queue isn't "Redis as storage" — it's a config with deliberate defaults:
// config/queue.ts — defaultJobOptions
{
attempts: 3, // 3 attempts per job, not one-shot
backoff: { type: 'exponential', delay: 2000 }, // 2s → 4s → 8s
removeOnComplete: { age: 3600, count: 1000 }, // succeeded: 1h or 1000 jobs
removeOnFail: { age: 86400 }, // failed: 24h (for triage)
}
export const maxConcurrentJobs =
parseInt(process.env.MAX_CONCURRENT_ANALYSES || '5', 10);
5 isn't magic: one headless
Playwright-Chromium costs ~150–250 MB RSS plus CPU for rendering; five parallel browsers is
the ceiling a modest single-core VM holds without swap and OOM-kill. One env var moves it to
match the hardware. A retrying job holds its slot the
whole time — with attempts: 3
and backoff, budget for that in capacity.
The worker is thin; all logic lives in the job processor. Graceful shutdown is mandatory, otherwise SIGTERM kills the browser mid-analysis:
// src/worker.ts
const worker = new Worker('website-analysis',
async (job) => { await processWebsiteAnalysis(job); },
{ ...queueConfig, concurrency: maxConcurrentJobs });
worker.on('failed', (job, err) =>
logger.error({ jobId: job?.id, error: { message: err?.message, code: (err as any)?.code } },
'Analysis job failed'));
process.on('SIGTERM', async () => { await worker.close(); process.exit(0); });
Inside the analysis there's a time budget, not a single timeout.
networkidle is capped at 30
seconds; on the remaining budget it falls back to the softer
load:
const networkIdleTimeout = Math.min(ANALYSIS_TIMEOUT, 30000); // ANALYSIS_TIMEOUT=60000
try {
await page.goto(url, { waitUntil: 'networkidle', timeout: networkIdleTimeout });
} catch {
logger.warn(logContext, 'networkidle timeout, falling back to load');
await page.goto(url, { waitUntil: 'load', timeout: ANALYSIS_TIMEOUT - networkIdleTimeout });
}
networkidle never fires on
sites with analytics/trackers — a blind timeout would be a false failure. The fallback to
load rescues those sites,
losing some late-loading content. Errors are split by meaning rather than dumped into
"failed": Website load timeout
(page didn't fit the budget),
Website inaccessible
(net::ERR_*), and bot-protection
(cloudflare/captcha) — on the last one we warn and continue:
some tokens can still be extracted from the placeholder page. Extraction is bounded by
MAX_ELEMENTS=10000 so a heavy
DOM doesn't eat memory and time.
Job lifecycle and what 404 actually means
POST /analyze → 202 {"job_id": "abc123"}
GET /job/abc123 → 200 {"status": "in-progress"}
GET /job/abc123 → 200 {"status": "completed", "result_location": "..."}
GET /job/abc123 → 404 # ~5s after completion
The detail that's easy to get wrong:
404 is
not returned on read. The status endpoint on
GET deletes nothing — it reads
the job:status:<id> key
from Redis. The job processor deletes the key on a timer,
5 seconds after the terminal state:
await updateJobStatus(jobId, 'completed', { result_location: designSystemPath });
setTimeout(() => deleteJobStatus(jobId), 5000); // window to fetch the result
So within the 5s window you can poll as many times as you like and reliably get
200 completed.
404 = "the job finished more
than ~5s ago or never existed." This isn't "grab it on the
first try" — it's "grab it within the window."
Artifact
github.com/dobryakov/design-system-mcp (TypeScript, Docker/Compose, BullMQ, Playwright E2E, Jest — unit / integration / e2e kept separate). ★8 / 4 forks — the only repo in the series with independent community validation.
Production service layout:
# docker-compose.yml
services:
redis:
image: redis:7-alpine
volumes: [ redis-data:/data ] # queue survives restart
healthcheck: { test: ["CMD", "redis-cli", "ping"] }
api:
build: .
ports: [ "${PORT:-3000}:3000", "${MCP_PORT:-3001}:3001" ] # REST + MCP separate
environment:
- MAX_CONCURRENT_ANALYSES=${MAX_CONCURRENT_ANALYSES:-5}
- REDIS_URL=${REDIS_URL:-redis://redis:6379}
depends_on: { redis: { condition: service_healthy } }
volumes: { redis-data: {} }
Where it breaks
-
The status retention window is a timer, not read-triggered.
The 5 seconds are hardcoded. A client that polls at intervals >5s (slow network,
throttling) will miss the terminal status entirely and
see only
404, unable to tell "done" from "never existed." This is a design smell: deletion should depend on delivery, not on wall-clock time.
Mitigation (already in the code): the authoritative record doesn't live in that key. BullMQ holds the job —removeOnComplete {age: 3600}(succeeded — one hour),removeOnFail {age: 86400}(failed — a day). The ephemeral status key is a convenience cache over the queue; a robust client reads job state from BullMQ, where it outlives the window. In production I'd either widen the window to the order of the client's poll interval, or serve the terminal status from BullMQ and keep the 5-second key only as a hot-path optimization. - Stateless contract vs. consumer convenience. The server deliberately doesn't accumulate result state in the status key — simplicity and no memory leak on abandoned jobs. The price is the "grab it within the window" contract. The choice is deliberate: for an agent consumer (Cursor polls tightly) the window is enough; for an integration that polls rarely — read from the queue. Not a "bug," but an explicit boundary the client is designed around.
-
Max 5 concurrent + retries.
OOM protection, but under a spike jobs queue up, and each retrying one
(
attempts: 3, exp backoff) holds its slot until it succeeds or exhausts its attempts. You need a backpressure contract on the client side and queue-depth monitoring.
What to monitor to catch degradation by signal, not by complaint: (1)waitingdepth in BullMQ → a spike means you've hit concurrency; (2) the rate of workerfailedevents and the number of records in theremoveOnFailtail (that's why they live 24h); (3) the share ofnetworkidle timeout, falling back to loadwarnings → target sites are getting heavier; (4) exhaustingmaxRetriesPerRequest: 3on the Redis connection → Redis health. - Bot-protection doesn't block, it degrades silently. On cloudflare/captcha it warns and continues, returning partial tokens. The risk: a "successful" result built from a placeholder. Mitigation: a flag in the result for a triggered bot indicator, so the consumer can tell a full extraction from a partial one.
For whom and why
The pattern "async submit + poll + a deliberate stateless trade-off" is architectural, not a tutorial. The conversation here isn't "how to build a queue" — it's about where the boundaries run: status retention, capacity under retries, degradation instead of refusal. The ★8 give independent confirmation that the topic resonates with a technical audience — the strongest point in the series by external validation.
Keep reading
About the author
Grigoriy Dobryakov — fractional / interim eng + AI leadership
Building MCP-based tooling for your engineering team?
Async patterns, stateless design trade-offs, and production-grade MCP server architecture — with a working artifact, not a tutorial.
Discuss a fractional audit Email meOther breakdowns
An engineering breakdown series: real task → methodology → working artifact → honest breakdown of where it fails.
Back to series →