Processing large jobs with Edge Functions, Cron, and Queues

When you're building applications that process large amounts of data, you quickly run into a fundamental problem: trying to do everything at once leads to timeouts, crashes, and frustrated users. The solution isn't to buy bigger servers. It's to break big jobs into small, manageable pieces.

Supabase gives you three tools that work beautifully together for this: Edge Functions for serverless compute, Cron for scheduling, and database queues for reliable job processing.

Here's how to use them to build a system that can handle serious scale.

The three-layer pattern#

The architecture is simple but powerful. Think of it like an assembly line:

Collection: Cron jobs run Edge Functions that discover work and add tasks to queues

Distribution: Other cron jobs route tasks from main queues to specialized processing queues

Processing: Specialized workers handle specific types of tasks from their assigned queues

This breaks apart the complexity. Instead of one giant function that scrapes websites, processes content with AI, and stores everything, you have focused functions that each do one thing well.

Real example: Building an NFL news aggregator#

Let's say you want to build a dashboard that tracks NFL (American football) news from multiple sources including NFL-related websites and NFL-related videos on YouTube, automatically tags articles by topic, and lets users search by player or team. When they see an article they’re interested in, they can click on it and visit the website that hosts the article. It’s like a dedicated Twitter feed for the NFL without any of the toxicity.

This sounds straightforward, but at scale this becomes complex fast. You need to monitor dozens of news sites, process hundreds of articles daily, make API calls to OpenAI for content analysis, generate vector embeddings for search, and store everything efficiently. Do this wrong and a single broken webpage crashes your entire pipeline.

We need to build a more resilient approach. With Supabase Edge Functions, Cron, and Queues, we have the building blocks for a robust content extraction and categorization pipeline.

Setting up the foundation#

Everything starts with the database, and Supabase is Postgres at its core. We know what we’re getting: scalable, dependable, and standard.

The database design for the application follows a clean pattern. You have content tables for storing articles and videos, queue tables for managing work, entity tables for NFL players and teams, and relationship tables linking everything together. For example:

Collection: Finding new content#

The collection layer seeks out new NFL-related content and runs on a schedule to discover new articles and videos. We create a collector for every site we want to search. A cron job triggers every 30 minutes to begin collection:

The Edge Function does the actual scraping. The trick is being selective about what you collect:

This simple filter prevents collecting promotional content or videos. You only want actual news articles.

When parsing HTML, you need to handle relative URLs properly:

And always deduplicate within a single scraping session:

For database insertion, let the database handle duplicates rather than checking in your application:

This approach is more reliable than complex application-level deduplication logic.

Distribution: Smart routing#

The distribution layer identifies articles that need processing and routes them to appropriate queues. The key insight is using separate queue tables for different content sources. NFL.com articles need different parsing than ESPN articles, so they get routed to specialized processors. It runs more frequently than collection: every 5 minutes:

The Edge Function finds unprocessed articles using a simple SQL query:

Then it routes based on the source site:

This separation is crucial because each site has different HTML structures and parsing requirements.

Processing: The heavy lifting#

Each content source gets its own processor that runs on its own schedule. NFL.com gets processed every 15 seconds because it's high-priority:

The processor handles one article at a time to stay within Edge Function timeout limits:

Content extraction requires site-specific CSS selectors:

Date parsing often needs custom logic for each site's format:

After scraping, the article gets analyzed with AI to extract entities:

Finally, create the relationships and generate embeddings:

The critical pattern is the finally block. We use it to always mark queue items as processed, preventing infinite loops when articles fail to process:

Monitoring with Sentry#

While the finally block prevents infinite loops, you still need visibility into what's actually failing. Sentry integration gives you detailed error tracking for your Edge Functions.

First, set up Sentry in your Edge Function:

Then wrap your processing logic with proper error capture:

This gives you real-time alerts when processors fail and detailed context for debugging production issues.

Processing user interactions through the pipeline#

The same pipeline pattern works for user-generated events. When someone clicks, shares, or saves an article, you don't want to block their response while updating trending scores for every player and team mentioned in that article.

Instead, treat interactions like any other job to be processed:

Then let a separate cron job process the trending updates in batches:

The processor can handle multiple interactions efficiently:

This keeps your user interface snappy while ensuring trending scores get updated reliably. If the trending processor goes down, interactions are safely queued and will be processed when it recovers.

AI-powered content scoring#

To surface the most important content automatically, use AI to analyze article context and assign importance scores.

Define scores for different news types:

Prompt OpenAI with structured output:

Process articles in batches to manage API costs:

Background tasks for expensive operations#

Some operations are too expensive to run synchronously, even in your cron-triggered processors. Vector embedding generation and bulk AI analysis benefit from background task patterns.

Edge Functions support background tasks that continue processing after the main response completes:

typescript

For operations that might take longer than Edge Function limits, break them into smaller background chunks:

typescript

This pattern keeps your main processing pipeline fast while ensuring expensive operations complete reliably.

Why this works#

This pattern succeeds because it embraces the constraints of serverless computing rather than fighting them. Edge Functions have time limits, so you process one item at a time. External APIs have rate limits, so you control timing with cron schedules. Failures happen, so you isolate them to individual tasks.

The result is a system that scales horizontally by adding more cron jobs and queues. Each component can fail independently without bringing down the whole pipeline. Users get fresh content as it becomes available rather than waiting for batch jobs to complete.

Most importantly, it's built entirely with Supabase primitives — no external queue systems or job schedulers required. You get enterprise-grade reliability with startup simplicity.