Skip to content
Cloudflare Docs

Getting Started

Cloudflare Pipelines allows you to ingest and load high volumes of real time streaming data into R2 Object Storage, without managing any infrastructure.

By following this guide, you will:

  1. Setup an R2 bucket
  2. Create a pipeline, with HTTP as a source, and an R2 bucket as a sink
  3. Send data to your pipeline's HTTP ingestion endpoint
  4. Verify the output delivered to R2

Prerequisites

To use Pipelines, you will need:

  1. Sign up for a Cloudflare account ↗.
  2. Install Node.js ↗.

Node.js version manager

Use a Node version manager like Volta ↗ or nvm ↗ to avoid permission issues and change Node.js versions. Wrangler, discussed later in this guide, requires a Node version of 16.17.0 or later.

1. Set up an R2 bucket

Create a bucket by following the get started guide for R2, or by running the command below:

Terminal window
npx wrangler r2 bucket create clickstream-bucket

Save the bucket name for the next step.

2. Create a Pipeline

To create a pipeline using Wrangler, run the following command in a terminal, and specify:

  • The name of your pipeline
  • The name of the R2 bucket you created in step 1
Terminal window
npx wrangler pipelines create clickstream-pipeline --r2-bucket clickstream-bucket --batch-max-seconds 5 --compression none

After running this command, you'll be prompted to authorize Cloudflare Workers Pipelines to create an R2 API token on your behalf. These tokens used by your pipeline when loading data into your bucket. You can approve the request through the browser link which will open automatically.

If you prefer not to authenticate this way, you may pass your R2 API Token to Wrangler:

Terminal window
npx wrangler pipelines create clickstream-pipeline --r2-bucket clickstream-bucket --r2-access-key-id [ACCESS-KEY-ID] --r2-secret-access-key [SECRET-ACCESS-KEY] --batch-max-seconds 5 --compression none

When choosing a name for your pipeline:

  1. Ensure it is descriptive and relevant to the type of events you intend to ingest. You cannot change the name of the pipeline after creating it.
  2. Pipeline names must be between 1 and 63 characters long.
  3. The name cannot contain special characters outside dashes (-).
  4. The name must start and end with a letter or a number.

You'll notice that we have set two optional flags while creating the pipeline: --batch-max-seconds and --compression. We've added these flags to make it faster for you to see the output of your first pipeline. For production use cases, we recommend keeping the default settings.

Once you create your pipeline, you will receive a HTTP endpoint which you can post data to. You should see output as shown below:

Terminal window
🌀 Authorizing R2 bucket "clickstream-bucket"
🌀 Creating pipeline named "clickstream-pipeline"
✅ Successfully created pipeline clickstream-pipeline with ID 91f312b8ca484e5db404bd3e3ef256fn


You can now send data to your pipeline with:
curl "https://91f312b8ca484e5db404bd3e3ef256fn.pipelines.cloudflare.com/" -d '[{ "foo":"bar }]'

3. Post data to your pipeline

Use a curl command in your terminal to post an array of JSON objects to the endpoint you received in step 1.

Terminal window
curl -H "Content-Type:application/json" \
    -d '[{"event":"viewedCart", "timestamp": "2025-04-03T15:42:30Z"},{"event":"cartAbandoned", "timestamp": "2025-04-03T15:42:37Z"}]' \
    <HTTP-endpoint>

Once the pipeline successfully accepts the data, you will receive a success message.

Pipelines handle batching the data, so you can continue posting data to the Pipeline. Once a batch is filled up, the data will be partitioned by date, and written to your R2 bucket.

4. Verify in R2

Open the R2 dashboard ↗, and navigate to the R2 bucket you created in step 1. You will see a directory, labeled with today's date (such as event_date=2025-03-05). Click on the directory, and you'll see a sub-directory with the current hour (such as hr=04). You should see a JSON file, containing the data you posted in step 3. Download the file, and open it in a text editor of your choice, to verify that the data posted in step 2 is present.

Next steps

If you have any feature requests or notice any bugs, share your feedback directly with the Cloudflare team by joining the Cloudflare Developers community on Discord ↗.

Cloudflare DashboardDiscordCommunityLearning CenterSupport Portal