Background Tasks & Queues¶

Nori provides two ways to handle background operations: 1. Starlette BackgroundTasks (Volatile, in-process) 2. Persistent Job Queues (Database-backed, survives restarts)

1. Volatile Background Tasks (`background`)¶

Uses Starlette's BackgroundTask. Ideal for quick, non-critical tasks like sending a notification or indexing a search document where losing the task on a server restart is acceptable.

from core.tasks import background

# Create a task
task = background(send_welcome_email, user.email, user.name)

# Pass it to a response
return JSONResponse({'ok': True}, background=task)

2. Persistent Queues (`push`)¶

Nori features a robust, multi-driver persistent queue system. Jobs are stored in a database (or Redis) and processed by a background worker. Use this for critical tasks like bulk emails, PDF generation, or heavy processing.

Key Robustness Features¶

Atomic Locking: Only one worker can process a single job at a time (race-condition free).
Exponential Backoff: If a job fails, the next attempt is delayed by (attempts⁴) × 15 seconds: ~15s → ~4m → ~20m → ~1h → ~3h. This gives external services time to recover.
Dead Letters: After 5 failed attempts, the job is marked with failed_at and stopped for manual inspection.
Graceful Shutdown: The worker finishes the current job before exiting on SIGINT/SIGTERM.

Configuration (.env)¶

Variable	Values	Description
`QUEUE_DRIVER`	`memory`, `database`, `redis`	`database` is recommended for production.
`REDIS_URL`	Redis URL	Required if using the `redis` driver.

Sending a Job to the Queue¶

from core import push

async def store(self, request):
    # Syntax: await push('module.path:function_name', *args, delay=0, **kwargs)

    # Simple push
    await push('modules.mail:send_welcome', email=user.email)

    # Delayed push (send in 1 hour)
    await push('modules.reminder:notify', user.id, delay=3600)

    return JSONResponse({'status': 'Job queued'})

Running the Worker¶

To process the queued jobs, run the Nori worker in a separate process (ideal for a sidecar container or systemd service):

python3 nori.py queue:work

You can specify a custom queue name (default is default):

python3 nori.py queue:work --name high_priority

Error Handling¶

In Volatile Tasks (`background`)¶

The error is logged via nori.tasks logger.
The exception is not re-raised.

In Persistent Queues (`push`)¶

The error is logged via nori.queue logger.
The attempts counter is incremented.
The next retry is scheduled with exponential backoff.
After 5 failures, it is marked as failed (Dead Letter).

When to use which?¶

Feature	`background()`	`push()`
Persistence	No (Lost on restart)	Yes (Stored in DB/Redis)
Retries	No	Yes (Exponential backoff)
Worker process	Not needed	Required (`queue:work`)
Atomic	No	Yes (One worker per job)
Best for	Logs, fast notifications	Emails, PDFs, Heavy Syncing