@@ -37,31 +38,37 @@ export function Features() {
title: "Functional",
description: "Functional programming approach with dependency injection with no macros.",
Component: FunctionalApproach,
+ href: "/docs/guides/tasks/introduction"
},
{
title: "Web-based UI",
description: "Intuitive web interface for managing and monitoring background tasks.",
Component: WebBasedUI,
+ href: "/docs/products/web-board"
},
{
title: "Compatible with popular backends",
description: "Easily connect your projects to databases and services.",
Component: CompatibleBackends,
+ href: "/docs/introduction/architecture#the-backend"
},
{
title: "Flexible Middleware",
description: "Provides middleware support build on top of tower",
Component: Middleware,
+ href: "/docs/guides/workers/middleware"
},
{
- title: "Workflow Orchestration",
+ title: "Workflow Orchestration",
description: "Coordinate complex background tasks with ease.",
Component: Workflows,
+ href: "/docs/guides/workflows"
},
{
title: "Observability",
description: "Gain insights into your background tasks with built-in instrumentation.",
Component: Observability,
+ href: "/docs/integrations/tracing"
},
]
diff --git a/content/docs/100-introduction/100-overview.mdx b/content/docs/100-introduction/100-overview.mdx
index b406976..5eb2cb5 100644
--- a/content/docs/100-introduction/100-overview.mdx
+++ b/content/docs/100-introduction/100-overview.mdx
@@ -1,7 +1,7 @@
---
title: Overview
navTitle: Overview
-excerpt: Apalis is a simple yet powerful Rust library for building reliable, scalable, and multithreaded background task and message processing systems.
+excerpt: Apalis is a simple yet powerful Rust library for building durable, distributed and multithreaded background jobs, tasks and messages processing systems.
bottomNavigation: pagination
---
@@ -9,7 +9,7 @@ apalis — Scalable background task processing in Rust, made simple and extensib
## What is apalis?
-Apalis is a simple yet powerful Rust library for building reliable, scalable, and multithreaded background task and message processing systems.
+Apalis is a simple yet powerful Rust library for building durable, distributed and multithreaded background jobs, tasks and messages processing systems.
## Core Features
@@ -17,16 +17,16 @@ Some of the main apalis features include:
| **Feature** | **Description** |
|---------------------|----------------------------------------------------------------------------------------------------------------------|
-| **Clean API** | A clean, macro-free API that promotes a predictable and functional approach to task handling. |
-| **Tower Integration** | Tasks are built on `tower::Service`, giving access to the rich ecosystem of Tower middleware and utilities. |
+| **Clean API** | A clean, macro-free API that promotes a predictable and [functional approach to task handling](docs/guides/tasks/introduction#async-function). |
+| **Tower Integration** | Tasks are built on [`tower::Service`](https://docs.rs/tower-service/latest/tower_service/trait.Service.html), giving access to the rich ecosystem of [tower middleware](https://docs.rs/tower/latest/tower/#the-tower-ecosystem) and utilities. |
| **Stream-Based Sources** | Any type that implements `Stream` can act as a task source — queues, channels, or custom pipelines. |
| **Runtime Agnostic** | Works with both `tokio` and `async-std`, offering flexibility across async runtimes. |
-| **High Concurrency** | Supports configurable workers, task queues, and thread pools for optimal performance. |
+| **High Concurrency** | Supports configurable [workers](/docs/guides/workers/introduction), [queues](/docs/guides/backend/introduction), and multi-threading for optimal performance. |
| **Extensibility** | Modular and flexible, allowing easy addition of new features or connectors tailored to your needs. |
-| **Scalability** | Efficiently handles growing workloads with a multithreaded processing model. |
-| **Reliability** | Built-in retry strategies and robust error handling ensure consistent task execution. |
-| **Integration Ready** | Out-of-the-box support for Redis, PostgreSQL, and other common systems. |
-| **Observability** | Detailed logging, metrics, and tracing help you monitor and debug task workflows. |
+| **Scalability** | Efficiently handles growing workloads with a distributed processing model. |
+| **Reliability** | Built-in [retry strategies](/docs/guides/tasks/error-handling#retries) and [robust error handling](/docs/guides/tasks/error-handling#overview) ensure consistent task execution. |
+| **Integration Ready** | Out-of-the-box support for [Redis, PostgreSQL, and other common backends](/docs/guides/backend/introduction#implementations). |
+| **Observability** | Detailed logging, metrics, and [tracing](/docs/integrations/tracing) help you monitor and debug task workflows. |
## How to Use These Docs
diff --git a/content/docs/100-introduction/200-quickstart.mdx b/content/docs/100-introduction/200-quickstart.mdx
index afece6e..95ad0ff 100644
--- a/content/docs/100-introduction/200-quickstart.mdx
+++ b/content/docs/100-introduction/200-quickstart.mdx
@@ -70,14 +70,14 @@ async fn send_email(email: Email) {
println!("Sending email to {:?}", email);
}
```
-This defines an async function `send_email` — the task handler — which receives an `Email` message and performs the work associated with it.
+This defines an async function `send_email` — [the task handler](/docs/guides/tasks/introduction#async-function) — which receives an `Email` message and performs the work associated with it.
The function returns a `Result<(), E>` where `E: StdError`.
### Setting Up a Worker
-With the task handler in place, we can now configure the worker that will fetch and execute these tasks.
+With the task handler in place, we can now configure the [worker](/docs/guides/workers/introduction) that will fetch and execute these tasks.
-A worker is responsible for polling the backend for tasks and executing them using the handler function.
+A worker is responsible for polling the [backend](/docs/guides/backend/introduction) for tasks and executing them using the handler function.
```rust name="worker" mode="inline"
WorkerBuilder::new("email_worker")
@@ -96,7 +96,7 @@ The worker continuously polls the tasks, executing them as they become available
### Enqueue a Task
-With the worker and backend in place, you can now enqueue a message to be processed.
+With the worker and backend in place, you can now [enqueue a task](/docs/guides/backend/pushing-tasks) to be processed.
In this example, we’ll push a task to the MemoryStorage-backed queue inside the same binary, right before the worker runs.
diff --git a/content/docs/100-introduction/300-architecture.mdx b/content/docs/100-introduction/300-architecture.mdx
index 646a580..79d0aaa 100644
--- a/content/docs/100-introduction/300-architecture.mdx
+++ b/content/docs/100-introduction/300-architecture.mdx
@@ -4,7 +4,7 @@ excerpt: Apalis is built around a small set of composable abstractions — a bac
bottomNavigation: pagination
---
-Apalis is designed around a small number of composable abstractions that fit together cleanly. Once you understand the three core layers — **backends**, **workers**, and **middleware** — every other concept in the documentation is a natural extension of them.
+Apalis is designed around a small number of composable abstractions that fit together cleanly. Once you understand the three core layers — [**backends**](/docs/guides/backend/introduction), [**workers**](/docs/guides/workers/introduction), and [**middleware**](/docs/guides/workers/middleware) — every other concept in the documentation is a natural extension of them.
This page gives you a structural map of the whole system before you dive into the details.
@@ -23,8 +23,8 @@ flowchart LR
Three things happen in every Apalis deployment:
-1. **Tasks are pushed** into a backend via `TaskSink` — encoded, wrapped in a `Task` envelope, and written to storage
-2. **A worker polls** the backend for tasks, drives them through a middleware stack, and hands them to your handler function
+1. [**Tasks are pushed**](/docs/guides/backend/pushing-tasks) into a backend via `TaskSink` — encoded, wrapped in a `Task` envelope, and written to storage
+2. **A worker polls** the backend for tasks, drives them through a [middleware stack](/docs/guides/workers/middleware), and hands them to your handler function
3. **The backend receives heartbeats** from the worker so it can detect stalled workers and requeue their tasks
Everything else — workflows, shared connections, observability, piping — is built on top of these three interactions.
@@ -35,7 +35,7 @@ Everything else — workflows, shared connections, observability, piping — is
### The Task Envelope
-Every job in Apalis is wrapped in a `Task
` before it touches any backend. The envelope carries three things:
+Every job in Apalis is wrapped in a [`Task`](https://crates.io/crates/apalis-core/1.0.0-rc.7#tasks) before it touches any backend. The envelope carries three things:
- **`Args`** — your job data (an email address, a user ID, a struct)
- **`Context`** — per-task metadata: attempt count, scheduled time, priority, tracing context
@@ -73,7 +73,7 @@ Currently supported backends:
| `apalis-mysql` | MySQL / MariaDB | ✅ Persistent |
| `apalis-sqlite` | SQLite | ✅ Persistent |
| `apalis-redis` | Redis | ✅ Persistent |
-| `apalis-cron` | In-memory schedule | ⚡ Ephemeral — use `pipe_to` for durability |
+| `apalis-cron` | In-memory schedule | ⚡ Ephemeral — use [`pipe_to`](/docs/advanced-concepts/piping-streams) for durability |
| `apalis-amqp` | AMQP broker | ✅ Persistent |
| `apalis-nats` | NATS | ✅ Persistent |
@@ -81,7 +81,7 @@ Currently supported backends:
### The Worker
-A worker drives a backend's task stream through a Tower service stack and into your handler. It is built with `WorkerBuilder`:
+A worker drives a backend's task stream through a Tower service stack and into your handler. It is built with [`WorkerBuilder`](https://docs.rs/apalis-core/1.0.0-rc.7/apalis_core/worker/builder/index.html):
```
WorkerBuilder::new("name")
.backend(B) → sets the task source
@@ -111,7 +111,7 @@ flowchart LR
T1[TraceLayer] --> TO1[TimeoutLayer] --> R1[RetryLayer] --> H[Handler] --> R2[RetryLayer] --> TO2[TimeoutLayer] --> T2[TraceLayer]
```
-Because middleware is composable Tower services, anything in the Tower ecosystem — rate limiting, circuit breaking, load shedding, custom instrumentation — works without modification. See [Middleware Order](/docs/middleware-order) for guidance on stacking layers correctly.
+Because middleware is composable Tower services, anything in the Tower ecosystem — [rate limiting](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.rate_limit), [circuit breaking](/docs/guides/workers/middleware#circuit-breaker), [load shedding](https://docs.rs/tower/latest/tower/load_shed/index.html), custom instrumentation — works without modification. See [Middleware Order](/docs/guides/workers/order-of-middleware) for guidance on stacking layers correctly.
---
@@ -128,7 +128,7 @@ Vec → stored in backend
Email → handler receives this
```
-Apalis ships `JsonCodec` (default), `MsgPackCodec`, and `BincodeCodec`. The codec is fixed per backend via `BackendExt::Codec`. See [Codecs](/docs/guides/backend/codecs).
+Apalis ships [`JsonCodec`](https://docs.rs/apalis-codec/latest/apalis_codec/json/struct.JsonCodec.html) (default), `MsgPackCodec`, and `BincodeCodec`. The codec is fixed per backend via `BackendExt::Codec`. See [Codecs](/docs/guides/backend/codecs).
---
@@ -146,14 +146,14 @@ When multiple workers share the same data store, `MakeShared` lets them derive i
### Piping — `PipeExt`
-Any `Stream- >` can be routed into a backend via `.pipe_to()`. The resulting `Pipe` implements `Backend`, so the worker sees a unified task stream without knowing the upstream source was a cron schedule, a CDC feed, or a channel. See [Piping Streams](/docs/advanced-concepts/piping-streams).
+Any `Stream
- >` can be routed into a backend via [`.pipe_to()`](/docs/advanced-concepts/piping-streams). The resulting `Pipe` implements `Backend`, so the worker sees a unified task stream without knowing the upstream source was a cron schedule, a CDC feed, or a channel. See [Piping Streams](/docs/advanced-concepts/piping-streams).
### Workflows
For multi-step jobs, `apalis_workflow` provides two higher-level backend types that both implement `Backend` and slot into a standard `WorkerBuilder`:
-- **`Workflow`** — a linear pipeline of typed async steps connected with combinators (`and_then`, `filter_map`, `fold`, etc.)
-- **`DagFlow`** — a directed acyclic graph where independent steps run in parallel and dependent steps wait only for their specific inputs
+- [**`Workflow`**](/docs/guides/workflows/sequential-workflow) — a linear pipeline of typed async steps connected with combinators (`and_then`, `filter_map`, `fold`, etc.)
+- [**`DagFlow`**](/docs/guides/workflows/dag-workflow) — a directed acyclic graph where independent steps run in parallel and dependent steps wait only for their specific inputs
See [Sequential Workflows](/docs/guides/workflows/sequential-workflow) and [DAG Workflows](/docs/guides/workflows/dag-workflow).
diff --git a/content/docs/100-introduction/400-use-cases.mdx b/content/docs/100-introduction/400-use-cases.mdx
index b83bfef..f34b73e 100644
--- a/content/docs/100-introduction/400-use-cases.mdx
+++ b/content/docs/100-introduction/400-use-cases.mdx
@@ -53,6 +53,8 @@ Apalis can be used to build robust data pipelines that process large volumes of
Workers can process jobs concurrently, enabling efficient handling of high-throughput workloads.
+Learn more about [Sequential Workflows](/docs/guides/workflows/sequential-workflow) which provides durable and distributed pipelined processings
+
---
### Webhook Handling and Event Processing
@@ -78,6 +80,8 @@ Some operations are computationally expensive and should not block the main appl
By offloading these tasks to workers, your application remains responsive and scalable.
+Learn more about [Parallelizing tasks](/docs/guides/workers/middleware#parallelize) and [Long Running Tasks](/docs/guides/workers/middleware#long-running-jobs).
+
---
### Task Orchestration and Workflows
@@ -90,6 +94,8 @@ Apalis can coordinate multi-step workflows where tasks depend on each other. Thi
You can model workflows as a series of jobs, where each step enqueues the next, enabling flexible and composable pipelines.
+Learn more about [Sequential Workflows](/docs/guides/workflows/sequential-workflow) and [DAG Workflows](/docs/guides/workflows/dag-workflow).
+
---
### Retry and Failure Handling
@@ -103,6 +109,8 @@ Failures are inevitable in distributed systems. Apalis provides mechanisms to ha
This ensures that your system remains robust even in the face of transient failures.
+See [Retries](/docs/guides/tasks/error-handling#retries) and [Catching Panics](/docs/guides/tasks/error-handling#catching-panics) for more.
+
---
### Rate-Limited and External API Calls
diff --git a/content/docs/200-guides/100-tasks/100-introduction.mdx b/content/docs/200-guides/100-tasks/100-introduction.mdx
index 2ac8c06..1e99f05 100644
--- a/content/docs/200-guides/100-tasks/100-introduction.mdx
+++ b/content/docs/200-guides/100-tasks/100-introduction.mdx
@@ -6,12 +6,12 @@ bottomNavigation: pagination
---
Apalis provides flexible ways to define task handlers that process your background tasks.
-You can pass in any type that implements [`IntoWorkerService`] such as:
+You can pass in any type that implements [`IntoWorkerService`](https://docs.rs/apalis-core/1.0.0-rc.7/apalis_core/worker/builder/trait.IntoWorkerService.html) such as:
1. Async functions with dependency injection
-2. A type that implements `tower::Service` trait implementation.
-3. Workflow: Sequential workflows
-4. DagFlow: DAG workflows
+2. A type that implements [`tower::Service`]((https://docs.rs/tower-service/latest/tower_service/trait.Service.html)) trait implementation.
+3. Workflow: [Sequential workflows](/docs/guides/workflows/sequential-workflow)
+4. DagFlow: [DAG workflows](/docs/guides/workflows/dag-workflow)
All workflows accept are a representation of multiple services which can accept async functions or tower services.
We will discuss these with the knowledge that the same knowledge holds for workflows.
@@ -66,11 +66,11 @@ async fn complex_handler(
## Service trait
-For more advanced scenarios requiring fine-grained control over request processing, you can implement the `tower::Service` trait directly. This approach accepts `Request` and provides maximum flexibility.
+For more advanced scenarios requiring fine-grained control over request processing, you can implement the [`tower::Service`](https://docs.rs/tower-service/latest/tower_service/trait.Service.html) trait directly. This approach accepts `Request` and provides maximum flexibility.
### When to Use Tower Services
-Consider using `tower::Service` when you need:
+Consider using [`tower::Service`](https://docs.rs/tower-service/latest/tower_service/trait.Service.html) when you need:
- Custom task preprocessing or validation
- Complex error handling strategies
diff --git a/content/docs/200-guides/100-tasks/200-state-management.mdx b/content/docs/200-guides/100-tasks/200-state-management.mdx
index 672e436..e6731ff 100644
--- a/content/docs/200-guides/100-tasks/200-state-management.mdx
+++ b/content/docs/200-guides/100-tasks/200-state-management.mdx
@@ -4,7 +4,12 @@ excerpt: Consider Context, Data and extending
bottomNavigation: pagination
---
-Effective state management is crucial for building robust task processing systems. Apalis provides two primary mechanisms for managing state in your task services: **Request Context** for backend-specific metadata and `Data` for application-wide shared state. Understanding when and how to use each approach will help you build more maintainable and efficient task processing systems.
+Effective state management is crucial for building robust task processing systems. Apalis provides two primary mechanisms for managing state in your task services:
+
+1. **Task Context** for backend-specific metadata
+2. `Data` for application-wide shared state.
+
+Understanding when and how to use each approach will help you build more maintainable and efficient task processing systems.
## Execution Context
@@ -34,8 +39,8 @@ You can access backend context provided the relevant backend you are using eg:
The following types can be accessed as part of the task state:
-- [`Attempt`] : The current attempt
-- [`TaskId`] : The current tasks id
+- [`Attempt`](https://docs.rs/apalis-core/1.0.0-rc.7/apalis_core/task/attempt/index.html) : The current attempt
+- [`TaskId`](https://docs.rs/apalis-core/1.0.0-rc.7/apalis_core/task/task_id/index.html) : The current tasks id
```rust name="backend_context" mode="inline"
async fn process(
@@ -55,7 +60,7 @@ async fn process(
}
```
-## State with `Data`
+## State with [`Data`](https://docs.rs/apalis-core/1.0.0-rc.7/apalis_core/task/data/index.html)
`Data` provides a way to share application-wide state across all task handlers. This is ideal for configuration, database connections, external service clients, and other resources that should be shared across your application.
diff --git a/content/docs/200-guides/200-workers/250-middleware.mdx b/content/docs/200-guides/200-workers/250-middleware.mdx
index 8b964f2..730b44d 100644
--- a/content/docs/200-guides/200-workers/250-middleware.mdx
+++ b/content/docs/200-guides/200-workers/250-middleware.mdx
@@ -1,6 +1,6 @@
---
title: Middleware
-excerpt: Apalis offers in-built middleware to help manage different use cases
+excerpt: Apalis workers are powered by composable middleware layers. This allows you to extend, control, and observe job execution without modifying your core business logic.
bottomNavigation: pagination
---
@@ -22,7 +22,31 @@ They are applied as layers around your job handlers, similar to HTTP middleware
---
-## Built-in Middleware
+## General Middleware
+
+| Middleware | Description |
+|------------|-------------|
+| [`option_layer`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.option_layer) | Conditionally applies a middleware layer if `Some`, otherwise skips it. Useful for feature flags or runtime configuration. |
+| [`layer_fn`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.layer_fn) | Wraps a function into a Tower `Layer`, allowing quick custom middleware without defining a full type. |
+| [`concurrency`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.concurrency) | Limits the maximum number of in-flight requests being processed concurrently. |
+| [`rate_limit`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.rate_limit) | Restricts the number of requests allowed within a specified time duration. |
+| [`retry`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.retry) | Automatically retries failed requests using a provided retry policy. |
+| [`timeout`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.timeout) | Fails requests that take longer than the specified duration. |
+| [`filter`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.filter) | Rejects requests synchronously based on a predicate function. |
+| [`filter_async`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.filter_async) | Rejects requests asynchronously using an async predicate. |
+| [`map_request`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.map_request) | Transforms incoming request types before they reach the service. |
+| [`map_response`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.map_response) | Transforms responses returned by the service. |
+| [`map_err`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.map_err) | Maps errors produced by the service into another type. |
+| [`map_future`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.map_future) | Transforms the future returned by the service before it is awaited. |
+| [`then`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.then) | Executes an async function after the service completes, regardless of success or failure. |
+| [`and_then`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.and_then) | Chains another async computation that only runs if the service succeeds. |
+| [`map_result`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.map_result) | Maps the full result (`Result`) into another value, regardless of outcome. |
+| [`catch_panic`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.catch_panic) | Catches panics during execution and converts them into errors. |
+| [`enable_tracing`](https://docs.rs/apalis/1.0.0-rc.7/apalis/layers/trait.WorkerBuilderExt.html#tymethod.enable_tracing) | Adds tracing instrumentation using the `tracing` crate for observability. |
+
+---
+
+## Built-in Worker Middleware
Apalis provides several built-in middleware to handle common production concerns.