DuckDB HTTPFS Extension

1.5.1 verified Fri May 01 auth: no python

Pip-installable DuckDB httpfs extension for reading/writing remote files via HTTP(S) and S3. Version 1.5.1 is the latest, compatible with DuckDB 1.5.1. Releases track DuckDB versions. Development is active.

pip install duckdb-extension-httpfs==1.5.1

Common errors

error ImportError: cannot import name 'httpfs' from 'duckdb_extension_httpfs' ↓

cause Trying to import the extension as a Python module instead of loading it with duckdb.load_extension().

fix

Replace 'from duckdb_extension_httpfs import httpfs' with 'import duckdb; duckdb.load_extension('httpfs')'.

error IO Error: Failed to download extension "httpfs" at URL "https://extensions.duckdb.org/..." ↓

cause DuckDB version does not match the extension version, or the extension is not installed.

fix

Install the matching duckdb-extension-httpfs version (e.g., pip install duckdb-extension-httpfs==1.5.1) and ensure DuckDB is same version.

error Invalid Input Error: Extension "httpfs" is not available on this platform ↓

cause Extension not built for the current platform (e.g., running DuckDB from source or unsupported OS).

fix

Use the official DuckDB Python wheel and install the corresponding extension wheel; check platform support on https://github.com/santosh-d3vpl3x/duckdb_extensions.

error HTTP Error: 403 Forbidden (s3) ↓

cause Missing or incorrect S3 credentials or permissions.

fix

Set correct s3_access_key_id, s3_secret_access_key, and s3_region before running queries, or configure IAM role if using AWS.

Warnings

breaking Python 3.8 is no longer supported starting from version 1.4.0. Requires Python >=3.9. ↓

fix Upgrade to Python 3.9+ or use duckdb-extension-httpfs==1.3.2 for Python 3.8 support.

gotcha The extension version must exactly match the DuckDB major.minor version (e.g., both 1.5.x). Mixing versions (e.g., httpfs 1.5.0 with DuckDB 1.4.4) will cause load failures. ↓

fix Use the same DuckDB and duckdb-extension-httpfs version (e.g., both 1.5.1).

deprecated Direct Python module import (from duckdb_extension_httpfs import ...) was never officially supported. The only supported way is duckdb.load_extension('httpfs'). ↓

fix Use duckdb.load_extension('httpfs') after importing duckdb.

gotcha S3 credentials are set with session-level SET statements, not using environment variables by default. Forgetting to set s3_access_key_id/s3_secret_access_key causes permission errors. ↓

fix Set S3 configuration via SET s3_access_key_id = '...'; SET s3_secret_access_key = '...'; SET s3_region = '...'; in the session.

Install

pip install duckdb duckdb-extension-httpfs

Imports

httpfs
wrong
```
from duckdb_extension_httpfs import httpfs
```
correct
```
import duckdb; duckdb.load_extension('httpfs')
```
The extension is not a Python module; use duckdb.load_extension() to register it.

Quickstart

Load the httpfs extension and query S3 data. Replace ACCESS_KEY/SECRET_KEY with your credentials.

import duckdb

duckdb.load_extension('httpfs')

db = duckdb.connect()
db.execute("""
SET s3_access_key_id = 'ACCESS_KEY';
SET s3_secret_access_key = 'SECRET_KEY';
SET s3_region = 'us-east-1';
CREATE TABLE test AS SELECT * FROM read_parquet('s3://bucket/file.parquet')
""")
print(db.execute('SELECT COUNT(*) FROM test').fetchone())