Async MCP Server with Job Queue: Why Polling, Not Blocking | Grigoriy Dobryakov

Grigoriy Dobryakov

Howto · breakdown

Breakdown 04 GitHub · ★8 / 4 forks

Async MCP server with job queue: design-system extractor for Cursor

A naive MCP server holds the connection while it works, so long operations hit timeouts. An async MCP server drops them into a job queue and returns a job ID immediately: browser automation, ML inference, and page crawling scale horizontally while the client polls by ID.

CTO Head of AI Architect Tech Lead

Problem

A naive MCP server does its work synchronously: the agent calls a tool and waits for the response. Fine for fast operations. For long ones — browser-automation site analysis, ML inference, page crawling — it blocks: client-side timeouts, no horizontal scaling, a single heavy request holding the connection open. Analyzing a site with Playwright takes tens of seconds; an MCP client won't wait that long.

Methodology

Separate submit from result: an async job queue with poll-by-ID.

  1. 1. Submit returns a job ID immediately (202), without blocking. Status is polled with a separate GET /job/:id call.
  2. 2. Playwright in a container does the actual analysis (design tokens: colors, typography, spacing, components) — isolated, headless, with --no-sandbox under the container user; the host's node_modules is explicitly excluded from the image.
  3. 3. BullMQ on top of Redis — job persistence, retries, and concurrency limits. Not a hand-rolled queue: retries, backoff, and retention come out of the box.
  4. 4. MCP server on 3001, separate from REST on 3000: multi-protocol (HTTP / HTTPS / stdio) for different deployment scenarios and Cursor integration.

The queue isn't "Redis as storage" — it's a config with deliberate defaults:

// config/queue.ts — defaultJobOptions
{
  attempts: 3,                              // 3 attempts per job, not one-shot
  backoff: { type: 'exponential', delay: 2000 },  // 2s → 4s → 8s
  removeOnComplete: { age: 3600, count: 1000 },    // succeeded: 1h or 1000 jobs
  removeOnFail:     { age: 86400 },                // failed: 24h (for triage)
}
export const maxConcurrentJobs =
  parseInt(process.env.MAX_CONCURRENT_ANALYSES || '5', 10);

5 isn't magic: one headless Playwright-Chromium costs ~150–250 MB RSS plus CPU for rendering; five parallel browsers is the ceiling a modest single-core VM holds without swap and OOM-kill. One env var moves it to match the hardware. A retrying job holds its slot the whole time — with attempts: 3 and backoff, budget for that in capacity.

The worker is thin; all logic lives in the job processor. Graceful shutdown is mandatory, otherwise SIGTERM kills the browser mid-analysis:

// src/worker.ts
const worker = new Worker('website-analysis',
  async (job) => { await processWebsiteAnalysis(job); },
  { ...queueConfig, concurrency: maxConcurrentJobs });

worker.on('failed', (job, err) =>
  logger.error({ jobId: job?.id, error: { message: err?.message, code: (err as any)?.code } },
    'Analysis job failed'));

process.on('SIGTERM', async () => { await worker.close(); process.exit(0); });

Inside the analysis there's a time budget, not a single timeout. networkidle is capped at 30 seconds; on the remaining budget it falls back to the softer load:

const networkIdleTimeout = Math.min(ANALYSIS_TIMEOUT, 30000); // ANALYSIS_TIMEOUT=60000
try {
  await page.goto(url, { waitUntil: 'networkidle', timeout: networkIdleTimeout });
} catch {
  logger.warn(logContext, 'networkidle timeout, falling back to load');
  await page.goto(url, { waitUntil: 'load', timeout: ANALYSIS_TIMEOUT - networkIdleTimeout });
}

networkidle never fires on sites with analytics/trackers — a blind timeout would be a false failure. The fallback to load rescues those sites, losing some late-loading content. Errors are split by meaning rather than dumped into "failed": Website load timeout (page didn't fit the budget), Website inaccessible (net::ERR_*), and bot-protection (cloudflare/captcha) — on the last one we warn and continue: some tokens can still be extracted from the placeholder page. Extraction is bounded by MAX_ELEMENTS=10000 so a heavy DOM doesn't eat memory and time.

Job lifecycle and what 404 actually means

POST /analyze     → 202 {"job_id": "abc123"}
GET  /job/abc123  → 200 {"status": "in-progress"}
GET  /job/abc123  → 200 {"status": "completed", "result_location": "..."}
GET  /job/abc123  → 404   # ~5s after completion

The detail that's easy to get wrong: 404 is not returned on read. The status endpoint on GET deletes nothing — it reads the job:status:<id> key from Redis. The job processor deletes the key on a timer, 5 seconds after the terminal state:

await updateJobStatus(jobId, 'completed', { result_location: designSystemPath });
setTimeout(() => deleteJobStatus(jobId), 5000);  // window to fetch the result

So within the 5s window you can poll as many times as you like and reliably get 200 completed. 404 = "the job finished more than ~5s ago or never existed." This isn't "grab it on the first try" — it's "grab it within the window."

Artifact

github.com/dobryakov/design-system-mcp (TypeScript, Docker/Compose, BullMQ, Playwright E2E, Jest — unit / integration / e2e kept separate). ★8 / 4 forks — the only repo in the series with independent community validation.

Production service layout:

# docker-compose.yml
services:
  redis:
    image: redis:7-alpine
    volumes: [ redis-data:/data ]          # queue survives restart
    healthcheck: { test: ["CMD", "redis-cli", "ping"] }
  api:
    build: .
    ports: [ "${PORT:-3000}:3000", "${MCP_PORT:-3001}:3001" ]  # REST + MCP separate
    environment:
      - MAX_CONCURRENT_ANALYSES=${MAX_CONCURRENT_ANALYSES:-5}
      - REDIS_URL=${REDIS_URL:-redis://redis:6379}
    depends_on: { redis: { condition: service_healthy } }
volumes: { redis-data: {} }

Where it breaks

For whom and why

The pattern "async submit + poll + a deliberate stateless trade-off" is architectural, not a tutorial. The conversation here isn't "how to build a queue" — it's about where the boundaries run: status retention, capacity under retries, degradation instead of refusal. The ★8 give independent confirmation that the topic resonates with a technical audience — the strongest point in the series by external validation.

About the author

Grigoriy Dobryakov — fractional / interim eng + AI leadership

Building MCP-based tooling for your engineering team?

Async patterns, stateless design trade-offs, and production-grade MCP server architecture — with a working artifact, not a tutorial.

Discuss a fractional audit Email me

Other breakdowns

An engineering breakdown series: real task → methodology → working artifact → honest breakdown of where it fails.

Back to series →