HTTP Protocol for Barnard59 Linked Data Pipelines

2.1.0 · active · verified Wed Apr 22

barnard59-http provides a set of HTTP protocol operations (such as GET, POST, PUT, DELETE) that function as 'steps' within Barnard59 Linked Data pipelines. Barnard59 is an Extract, Transform, Load (ETL) toolkit primarily designed for working with RDF and Linked Data, where pipelines are declaratively defined using RDF graphs (e.g., in Turtle or JSON-LD format) and executed by the `barnard59` CLI or programmatic API. This package extends the core Barnard59 functionality, enabling pipelines to interact with external HTTP resources. The current stable version is 2.1.0, and it is part of an actively maintained monorepo, with frequent patch and minor releases across its constituent packages. Key differentiators include its deep integration with RDF/Linked Data principles, its streaming architecture for handling large datasets, and the flexibility to embed custom JavaScript logic directly into RDF-defined pipelines.

Common errors

```
ERR_UNKNOWN_FILE_EXTENSION: Unknown file extension ".ttl" for .../pipeline.ttl
```
cause The Node.js runtime is trying to interpret the .ttl (Turtle) pipeline definition file as a JavaScript module, which it cannot do by default in an ESM context.

fix Ensure you are running the pipeline using the `barnard59` CLI (e.g., `barnard59 run pipeline.ttl`) or programmatically loading the RDF graph, not directly trying to `import` or `require` the `.ttl` file as a JS module.
```
Error: HttpError: 404 Not Found (or similar 4xx/5xx status code)
```
cause The HTTP request made by a barnard59-http step (e.g., `http:get`) received an error status code from the target server, indicating a client-side (4xx) or server-side (5xx) issue.

fix Verify the `http:url` in your pipeline definition is correct and accessible. Check the target API's documentation for expected endpoints and parameters. Implement `pipeline:catch` steps to handle these errors gracefully and potentially retry failed requests.
```
TypeError: Invalid RDF/JS object provided to operation
```
cause A custom JavaScript step or another pipeline operation received an RDF/JS object (e.g., Quad, Term, Dataset) that does not conform to the expected RDF/JS specification or the specific version used by barnard59.

fix Check the `rdf-js` version used by your `barnard59` installation and ensure any custom code or external RDF/JS libraries are compatible. Inspect the input to the problematic step to ensure it's a valid RDF/JS object.

Warnings

breaking Older versions of barnard59 and its related packages might have strict peer dependency requirements for rdf-js. Upgrading individual barnard59-* packages without ensuring compatibility across the entire ecosystem can lead to runtime errors due to differing rdf-js interface expectations.
Fix: Always update barnard59 packages within the same major version range when possible. Refer to the main barnard59 monorepo changelog for cross-package compatibility notes. Ensure your project's rdf-js dependencies align with the barnard59 ecosystem's expectations.
gotcha Barnard59 is designed for streaming data. While barnard59-http operations inherently support streams, certain pipeline designs or custom steps that collect entire datasets into memory can negate streaming benefits and lead to out-of-memory errors for large inputs.
Fix: Design custom pipeline steps to be stream-aware, processing data chunk-by-chunk. Avoid operations like `toArray()` on large streams within custom handlers unless strictly necessary and memory usage is controlled.
gotcha Since barnard59 and its modular packages like barnard59-http have transitioned to an ESM-first approach, using CommonJS `require()` syntax can lead to `ERR_UNKNOWN_FILE_EXTENSION` or `Cannot find module` errors, particularly in Node.js environments configured for CJS.
Fix: Ensure your project uses ES Modules (e.g., `"type": "module"` in package.json or `.mjs` file extensions) and uses `import` statements. If a pure CommonJS environment is required, consider using dynamic `import()` or transpilation.
gotcha HTTP requests made via barnard59-http steps are subject to network failures, incorrect URLs, or API authentication issues, leading to pipeline halts or unexpected data. Error handling within the pipeline definition is crucial.
Fix: Implement `pipeline:catch` steps in your RDF pipeline definitions to gracefully handle HTTP errors. Utilize `http:retry` for transient network issues or `http:timeout` for long-running requests where applicable.

Install

npm install barnard59-http npm
yarn add barnard59-http yarn
pnpm add barnard59-http pnpm

Imports

get
wrong:
```
const { get } = require('barnard59-http')
```
correct:
```
import { get } from 'barnard59-http'
```
barnard59-http functions are primarily used as pipeline steps in RDF definitions, but can be imported programmatically. The library is ESM-first.
post
wrong:
```
const { post } = require('barnard59-http')
```
correct:
```
import { post } from 'barnard59-http'
```
Used for HTTP POST requests within a pipeline. ESM import is standard; CommonJS `require` might fail due to the package's module type.
request
wrong:
```
import request from 'barnard59-http'
```
correct:
```
import { request } from 'barnard59-http'
```
Provides a generic HTTP request step, allowing full control over method, headers, and body. It is a named export, not a default export.

Quickstart

Demonstrates defining an RDF pipeline in Turtle (.ttl) to fetch JSON data via an `http:get` step and process it with a custom TypeScript step. It showcases integration within the Barnard59 ecosystem and execution via the `barnard59` CLI.

/* pipeline.ttl */
@prefix pipeline: <https://pipeline.described.at/> .
@prefix p: <http://barnard59.com/pipeline/> .
@prefix http: <http://barnard59.com/http/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://example.org/pipeline/fetchData> a p:Pipeline ;
  rdfs:label "Fetch JSON data from an API" ;
  pipeline:runs (
    [ a http:get ;
      http:url "https://jsonplaceholder.typicode.com/todos/1"^^xsd:anyURI ;
      rdfs:label "Fetch Todo 1"
    ]
    [ a p:map ;
      rdfs:label "Parse JSON and extract title" ;
      pipeline:handle <node:./transform.js#parseJsonAndExtractTitle>
    ]
    [ a p:catch ;
      rdfs:label "Handle Errors"
      pipeline:handle <node:./transform.js#logError>
    ]
  ) .

/* transform.ts */
import { Context, Stream } from '@barnard59/core';

export function parseJsonAndExtractTitle(this: Context, chunk: Stream) {
  const content = chunk.toString(); // Assuming the HTTP response is a string
  try {
    const json = JSON.parse(content);
    return this.env.rdf.literal(json.title || 'No Title');
  } catch (error) {
    console.error('Failed to parse JSON or extract title:', error);
    return this.env.rdf.literal('Error processing data');
  }
}

export function logError(this: Context, error: Error) {
  console.error('Pipeline error caught:', error.message);
  return this.env.rdf.literal(`Error: ${error.message}`);
}

/* Usage with barnard59 CLI: */
// npm install -g barnard59
// npm install barnard59-http @barnard59/core @zazuko/env
// barnard59 run pipeline.ttl --pipeline http://example.org/pipeline/fetchData

view raw JSON →