PipelineWise Singer Python Library

2.0.1 · maintenance · verified Sat Apr 11

This library is a fork of Singer's singer-python, specifically tailored for PipelineWise compatibility. It provides utilities for implementing the Singer.io data replication specification, enabling taps (data extractors) and targets (data loaders) to communicate using a standard JSON-based message format over stdout. The current version is 2.0.1, with releases occurring infrequently, typically driven by critical bug fixes or significant feature enhancements like performance improvements.

Warnings

breaking Version 2.0.0 replaced the standard `json` library with `orjson` for improved performance. While generally a drop-in replacement, applications with custom JSON handling or those relying on specific `json` library behaviors not supported by `orjson` (e.g., certain `json.dumps` parameters) may experience unexpected issues.
Fix: Review custom JSON serialization/deserialization logic in your application. Test thoroughly after upgrading to ensure compatibility with `orjson`. Consult `orjson` documentation for any behavioral differences.
gotcha The library does not provide a default logging configuration. Users are responsible for setting up their own logging using standard Python `logging` module practices. If no logging is configured, log messages from the library might not appear or might go to stderr without proper formatting. An environment variable `LOGGING_CONF_FILE` can be used to point to a logging configuration file.
Fix: Implement a logging configuration in your application that utilizes `pipelinewise-singer-python`. Refer to Python's `logging` module documentation or set the `LOGGING_CONF_FILE` environment variable to a valid logging configuration file path.
gotcha The `BATCH` message type, introduced in `v1.2.0` and enhanced in `v1.3.0` (with `time_extracted`), allows for more efficient data transfer. However, older taps or targets in a Singer.io pipeline might not fully support this message type, leading to compatibility issues or data processing failures if not all components are updated.
Fix: Ensure all taps and targets in your Singer.io pipeline are updated to versions that explicitly support the `BATCH` message type, especially if you intend to leverage its performance benefits. Check component documentation for `BATCH` support.

Install

pip install pipelinewise-singer-python Install latest version

Imports

singer
```
import singer
```
The entire library's functionality is typically accessed via the top-level 'singer' module after import.

Quickstart

This quickstart demonstrates basic usage of the `pipelinewise-singer-python` library to emit Singer.io compliant messages (schema, record, state) to standard output. These messages can then be consumed by a Singer.io target. The example defines a simple schema and writes three records, followed by a state message.

import singer
import sys
import json

# Define a simple schema for demonstration
schema = {
    'properties': {
        'id': {'type': 'integer', 'key': True},
        'name': {'type': 'string'},
        'value': {'type': 'number'}
    }
}

# Write the schema message
singer.write_schema('my_stream', schema, ['id'])

# Write some record messages
records = [
    {'id': 1, 'name': 'Item A', 'value': 100.5},
    {'id': 2, 'name': 'Item B', 'value': 200.0},
    {'id': 3, 'name': 'Item C', 'value': 150.75}
]

for record in records:
    singer.write_record('my_stream', record)

# Write a state message (optional, but good practice for incremental processing)
singer.write_state({'last_processed_id': records[-1]['id']})

print("\n--- Output captured (simulated stdout) ---")
# For demonstration, manually capture output to show what 'singer' writes
# In a real Singer pipeline, this output goes to stdout.

view raw JSON →