Skip to content
Cloudflare Docs

Configure HTTP Endpoint

Pipelines support data ingestion over HTTP. When you create a new pipeline, you'll receive a globally scalable ingestion endpoint. To ingest data, make HTTP POST requests to the endpoint.

Terminal window
$ npx wrangler@latest pipelines create my-clickstream-pipeline --r2-bucket my-bucket


🌀 Authorizing R2 bucket "my-bucket"
🌀 Creating pipeline named "my-clickstream-pipeline"
✅ Successfully created pipeline my-clickstream-pipeline


Id:    0e00c5ff09b34d018152af98d06f5a1xvc
Name:  my-clickstream-pipeline
Sources:
  HTTP:
    Endpoint:        https://0e00c5ff09b34d018152af98d06f5a1xvc.pipelines.cloudflare.com/
    Authentication:  off
    Format:          JSON
  Worker:
    Format:  JSON
Destination:
  Type:         R2
  Bucket:       my-bucket
  Format:       newline-delimited JSON
  Compression:  GZIP
Batch hints:
  Max bytes:     100 MB
  Max duration:  300 seconds
  Max records:   100,000


🎉 You can now send data to your Pipeline!


Send data to your Pipeline's HTTP endpoint:
curl "https://0e00c5ff09b34d018152af98d06f5a1xvc.pipelines.cloudflare.com/" -d '[{ ...JSON_DATA... }]'

Accepted data formats

Pipelines accept arrays of valid JSON objects. You can send multiple objects in a single request, provided the total data volume is within the documented limits. Sending data in a different format will result in an error.

For example, you can send data to your pipeline using a curl command like this:

Terminal window
curl -X POST https://<PIPELINE-ID>.pipelines.cloudflare.com \
  -H "Content-Type: application/json" \
  -d '[{"foo":"bar"}, {"foo":"bar"}, {"foo":"bar"}]'


{"success":true,"result":{"committed":3}}

Turning HTTP ingestion off

By default, ingestion via HTTP is turned on. You can turn it off by excluding it from the list of sources, by using --sources when creating or updating a pipeline.

Terminal window
$ npx wrangler pipelines create [PIPELINE-NAME] --r2-bucket [R2-BUCKET-NAME] --sources worker

Ingestion URLs are tied to your pipeline ID. Turning HTTP off, and then turning it back on, will not change the URL.

Authentication

You can secure your HTTP ingestion endpoint using Cloudflare API tokens. By default, authentication is turned off. To configure authentication, use the --require-http-auth flag while creating or updating a pipeline.

Terminal window
$ npx wrangler pipelines create [PIPELINE-NAME] --r2-bucket [R2-BUCKET-NAME] --require-http-auth true

Once authentication is turned on, you will need to include a Cloudflare API token in your request headers.

Get API token

  1. Log in to the Cloudflare dashboard ↗ and select your account.
  2. Navigate to your API Keys ↗
  3. Select Create Token
  4. Choose the template for Workers Pipelines. Click on continue to summary, and finally on create token. Make sure to copy the API token, and save it securely.

Making authenticated requests

Include the API token you created in the previous step in the headers for your request:

Terminal window
curl https://<PIPELINE-ID>.pipelines.cloudflare.com
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${API_TOKEN}" \
  -d '[{"foo":"bar"}, {"foo":"bar"}, {"foo":"bar"}]'

Specifying CORS Settings

If you want to use your pipeline to ingest client side data, such as website clicks, you'll need to configure your Cross-Origin Resource Sharing (CORS) settings ↗.

Without setting your CORS settings, browsers will restrict requests made to your pipeline endpoint. For example, if your website domain is https://my-website.com, and you want to post client side data to your pipeline at https://<PIPELINE-ID>.pipelines.cloudflare.com, without CORS settings, the request will fail.

To fix this, you need to configure your pipeline to accept requests from https://my-website.com. You can do so while creating or updating a pipeline, using the flag --cors-origins. You can specify multiple domains separated by a space.

Terminal window
$ npx wrangler pipelines update [PIPELINE-NAME] --cors-origins https://mydomain.com http://localhost:8787

You can specify that all cross origin requests are accepted. We recommend only using this option in development, and not for production use cases.

Terminal window
$ npx wrangler pipelines update [PIPELINE-NAME] --cors-origins "*"

After your the --cors-origins have been set on your pipeline, your pipeline will respond to preflight requests and POST requests with the appropriate Access-Control-Allow-Origin headers set.

Cloudflare DashboardDiscordCommunityLearning CenterSupport Portal