Automating Code Reviews with AI and Distributed Systems

By Ahmed Sulaimon • 2025-10-03

I built the Auto PR Review Assistant to automate the first-pass review of GitHub pull requests. The project grew from a single-process prototype into a multi-tenant microservice stack with deployment, packaging, and production-hardening workstreams. Below I describe the core problem I hit, the design I implemented, operational choices, and what I learned.

Summary

• I moved from a single shared Redis queue to namespaced keys by installation_id (pr-review-queue:<id>, pr-review-history:<id>) to eliminate data leakage and make per-tenant retention/pruning simple.
• I implemented a small “wake” keep-alive pattern to help keep worker instances alive on hosts that aggressively spin down idle services.
• The system is split into a FastAPI webhook listener, a review worker, and a CLI. The CLI is packaged for PyPI and publishing is automated via GitHub Actions + OIDC.

The problem I started with

At first I used global Redis keys:

• pr-review-queue
• pr-review-history

That worked locally but failed as the app gained multiple installations:

1. Cross-tenant data leakage — jobs and histories from different GitHub App installations mixed together, exposing other users’ reviews.
2. Operational scale & hygiene — a single global list made pruning, debugging, and per-tenant policies hard.

Also, hosting a continuously running worker on free managed platforms proved unreliable: instances would spin down when idle and new jobs would sit unprocessed.

My solution: namespaced Redis + microservices

The fix was straightforward: namespace Redis keys by GitHub App installation_id.

Key patterns

• Queue: pr-review-queue:<installation_id>
• History: pr-review-history:<installation_id>

Request flow

1. GitHub → FastAPI webhook listener (validates signature).
2. Listener extracts installation.id and LPUSHes a job into pr-review-queue:<installation_id>.
3. Review worker BRPOP(s) the installation-specific queue, fetches PR metadata/diffs, calls the LLM, posts comments, and appends history to pr-review-history:<installation_id>.

Benefits

• Strong tenant isolation -- no mixed history.
• Easier pruning and per-installation retention.
• Clearer logs and easier troubleshooting (keys map directly to an installation).

Hosting & the worker keep-alive pattern

To run a worker on free tiers (that don’t offer persistent background workers) I used a pragmatic approach:

• Deploy the review engine as a FastAPI web service (binds a port so it’s allowed by hosts).
• Launch the worker loop as an async background task on FastAPI startup.
• Add a small wake/keep-alive endpoint. The webhook listener calls or pings this endpoint after enqueueing a job so the service receives traffic and stays alive long enough to process jobs.

This is a cost-conscious trade-off -- it works well enough for demos and low usage, but for production I recommend a host that supports long-running workers or a paid worker plan.

GitHub App auth: JWTs and installation tokens

GitHub App auth required care:

• The App private key must be PEM-formatted. When stored in env vars I convert backslash n ( ) sequences back into real newlines before using PyJWT.
• I generate a short-lived JWT (≤10m) for the App, then exchange it for an installation access token per installation.
• Installation tokens expire, so I refresh tokens and retry on 401 responses.

Add logging and retries around token creation and API calls -- that made the workflow much more robust.

LLM integration: prompting and parsing

AI is the interesting part, but it’s also noisy. My approach:

• Prompt the model to return JSON-only in a concrete schema (file, line, comment).
• Use a resilient parser that accepts a few JSON shapes and safely no-ops on malformed output.
• Split large PR diffs into smaller chunks and rate-limit requests to control quota usage.

This makes downstream processing predictable and reduces failure modes from unexpected model output.

CLI & packaging

The CLI offers these commands:

• pr-review list-prs — list recent analyzed PRs (per installation).
• pr-review show-pr <id> — show details and AI comments.
• pr-review recheck-pr <id> — requeue a PR for re-review.

I store API_URL and installation_id in ~/.pr-review/config.json for a frictionless UX. The CLI is packaged and the release workflow uses GitHub Actions + OIDC to publish to PyPI securely (no long-lived PyPI tokens in the repo).

Testing, CI/CD & observability

• Unit tests (pytest) with mocks for Redis, GitHub, and the LLM keep tests fast and deterministic.
• CI uses GitHub Actions to run tests and packaging steps.
• Observability: services expose /health and emit structured logs; adding metrics or Sentry would be a straightforward next step.

Small but important engineering bits

• Fixed PEM encoding issues for private keys stored in envs.
• Recheck logic reuses the installation_id from history so users don’t need to supply it again.
• Interactive CLI setup lets users save installation_id to the config file during first run.
• Namespacing is a light-weight multi-tenant model that can be upgraded to stronger isolation (separate Redis DBs or instances) if needed.

What I’d improve next

• Protect the query API with authentication so only installation owners can access their history.
• Add a compact web dashboard for browsing reviews (better UX than CLI alone).
• Explore Redis Streams or consumer groups for robust horizontal scaling.
• Harden posting-idempotency and backoff to avoid duplicate comments.

Closing thoughts

Small design choices can have big outcomes. Namespacing Redis keys by installation_id fixed security and scaling issues early. Operational realities (free-host spin-down) forced pragmatic trade-offs -- the wake endpoint is one such pattern that kept things usable at low cost. Finally, integrating APIs and LLMs taught me the value of strict output contracts, defensive parsing, and strong observability.

Installation

If you want to try the GitHub App, install it here:
https://github.com/apps/auto-pr-review-assistant/installations/new

CLI tool Installation and usage Guide: https://github.com/Ahmedsulaimon/Auto-PR-Review-Assistant/blob/main/cli/README.md