Building a self-hosted OTA pipeline for 480,000 users at $12/month

OTA (over-the-air) updates let you ship JavaScript bundle changes to a React Native app instantly, without going through an app store review cycle. For a nonprofit app with half a million active users and a tight budget, getting this right mattered. This post documents how we designed, built, and stabilised the system currently running in production for UniApplyForMe.

480,000 monthly active users. $12/month infrastructure. ~5 minutes to ship a fix.

The problem with the previous setup

The app was running a custom OTA pipeline built on react-native-ota-hot-update, AWS S3, CloudFront, and a custom Hono API endpoint. It had real problems that had accumulated over time.

Windows path separators. Node uses backslashes on Windows. The pipeline had no handling for this, causing silent failures during publishing.

Hermes toolchain requirement. iOS bundles had to be compiled on macOS. Publishing from Windows or Linux produced bundles that crashed on apply silently, with no useful error.

Three-channel routing. Google Play, Samsung Galaxy Store, and Huawei AppGallery each needed separate routing that was fragile and undocumented.

No code signing. A compromised CDN could theoretically serve a malicious bundle to users. The pipeline had no cryptographic verification.

The maintenance cost was disproportionate to what it provided.

Why xprem

xprem is an open-source, self-hosted implementation of Expo's official OTA update protocol the same protocol that powers EAS Update. Using the official protocol meant standard expo-updates on the app side, no forks, and code signing built in with per-app RSA key pairs.

The Go binary runs as a single Docker container alongside Postgres. Asset storage goes to any S3-compatible bucket. The whole system is self-contained and auditable. The core is MIT licensed.

Architecture

The system has three layers: the OTA server, asset delivery, and the CI publishing pipeline.

The manifest check a few kilobytes of signed JSON goes through xprem. The actual bundle download goes directly from the device to Cloudflare's edge network. The Droplet never handles asset traffic at scale.

Publishing flow

1. Developer pushes a code change
2. Manual trigger on GitHub Actions
3. macOS runner exports the iOS bundle, Ubuntu runner exports Android
4. Both publish to xprem via eoas publish
5. xprem stores bundles in Cloudflare R2

Device update flow

1. App launches, expo-updates calls GET /manifest on xprem
2. xprem returns a signed manifest pointing to asset URLs on ota-assets.uniapplyforme.app
3. Device verifies the RSA signature against the embedded certificate
4. Assets download directly from Cloudflare CDN
5. Update applies on next launch

Infrastructure decisions

Cloudflare R2 over AWS S3

The previous setup used S3 behind CloudFront specifically to avoid S3's egress fees (up to $0.09/GB). R2 eliminates the problem entirely egress is free at any volume. OTA assets use content-addressed, immutable filenames, so Cloudflare cache hit rates are effectively 100% after the first device downloads each update.

Cost comparison

GitHub Actions for publishing

Publishing iOS OTA bundles requires the Hermes bytecode compiler, which produces architecture-specific bytecode. Bundles compiled on Linux were crashing on apply with ErrorRecovery.crash() in expo-updates. The iOS job runs on macos-latest. The Android job runs on ubuntu-latest.

The workflow uses workflow_dispatch manual trigger only to avoid publishing on every commit shared with the web app codebase.

Non-obvious issues

These are the problems that cost the most debugging time.

Wrong xprem environment variable names. xprem uses DB_URL (not DATABASE_URL), STORAGE_MODE (not STORAGE_TYPE), AWS_BASE_ENDPOINT (not S3_ENDPOINT), and S3_BUCKET_NAME (not S3_BUCKET). Using the wrong names causes xprem to boot in stateless mode with no error indicating why.

expo-channel-name missing from asset requests. expo-updates sends the expo-channel-name header with manifest requests but not asset requests. xprem's /assets endpoint requires this header to resolve the channel-branch mapping. A Caddy directive injects it from the branch query parameter that xprem includes in every asset URL it generates.

Certificate PEM format corruption. The certificate downloaded from the xprem dashboard had spaces where newlines should be. This caused iOS ASN1Error error 0 at runtime which looks identical to a signing key mismatch. Verify the certificate with new crypto.X509Certificate(cert) in Node before building. If it throws, the PEM is malformed.

Results

What to do differently

Start with GitHub Actions. The local publishing path on Windows is not viable for iOS bundles. Set up the CI workflow before any testing.

Validate the certificate before building. Run new crypto.X509Certificate(cert) in Node. A certificate that fails this check will produce ASN1Error error 0 on iOS, which is indistinguishable from other signing failures until you know what to look for.

Pass all env vars to CI. Any EXPO_PUBLIC_* variable referenced at module load time will crash the bundle if missing during export. Validate that all required env vars are present before publishing.

Open source

xprem is available at github.com/mercuretechnologies/xprem. The MIT core covers everything needed for a production deployment.