Flumedb: Modular Stream-Based Database

2.1.8 · active · verified Wed Apr 22

Flumedb is a modular database system built around an append-only log and streaming views. It acts as an orchestrator, connecting a single `flumelog-*` module (the durable primary data store) with zero or more `flumeview-*` modules (which provide flexible, rebuildable indexes and query interfaces). The core innovation is its star-shaped pipeline architecture, where views stream data from the central log and build their own optimized models. Unlike older designs like `level-sublevel`, `flumedb` views can be easily updated or added; they simply rebuild from the main log if their version changes. This current stable version is 2.1.8. The project does not explicitly state a release cadence, but its components are actively maintained within the `flumedb` GitHub organization. Key differentiators include its robust view rebuilding mechanism, asynchronous views that automatically synchronize upon read, and the ability for views to optimize for queries rather than durability, as they can always regenerate from the canonical log.

Common errors

```
Error: Cannot find module 'flumelog-memory'
```
cause A required log implementation module (like `flumelog-memory` or `flumelog-offset`) was not installed.

fix Ensure you have installed the specific log module: `npm install flumelog-memory` (or `flumelog-offset`, etc.)
```
TypeError: db.myViewName.get is not a function
```
cause The `flumeview-*` module was either not correctly passed to `db.use()`, or its public methods (like `get`) are not correctly exposed or named by the view module itself.

fix Verify that `db.use('myViewName', MyViewConstructor(version, ...))` is called correctly during initialization and that `MyViewConstructor` returns an object with the expected API methods. Also, ensure the view module is installed: `npm install flumeview-myviewname`.

Warnings

gotcha Views in Flumedb are asynchronous. While the main log's `append` callback indicates data has been written, views might not be immediately up-to-date. A read from an unsynced view will block until the view building is complete. Applications interacting with large datasets or complex views should consider displaying progress bars during initial view indexing or reindexing.
Fix: Implement UI feedback (e.g., loading indicators) for operations that rely on potentially unsynced views, especially on application startup or after view definition changes. Use `flumedb.rebuild()` for explicit view reconstruction, and handle its callback.
gotcha View rebuilding on startup can be a time-consuming process for large databases. If a view's version number changes, or a new view is added, it will rebuild from the entire log. This process, while robust, can take several minutes depending on data size and view complexity, impacting initial application responsiveness.
Fix: Optimize view logic for efficiency. For critical startup paths, consider strategies to defer less critical view synchronization or pre-build views in deployment. Monitor and benchmark view rebuild times to set user expectations.
gotcha Flumedb itself is an orchestrator; it requires separate `flumelog-*` and `flumeview-*` modules to be installed and passed to its constructor and `.use()` method respectively. Forgetting to install these companion modules will lead to runtime errors.
Fix: Always `npm install` the necessary `flumelog-*` (e.g., `flumelog-memory`, `flumelog-offset`) and `flumeview-*` (e.g., `flumeview-reduce`, `flumeview-level`) packages that your application uses.

Install

npm install flumedb npm
yarn add flumedb yarn
pnpm add flumedb pnpm

Imports

Flume
wrong:
```
import Flume from 'flumedb'
```
correct:
```
const Flume = require('flumedb')
```
Flumedb v2 is primarily a CommonJS module. Direct ES Module imports are not supported without a transpiler or dynamic import.

Quickstart

This example demonstrates how to initialize FlumeDB with an in-memory log, attach a reducing view, and then append data and query the view.

var MemLog = require('flumelog-memory') // In-memory log for simple testing
var Reduce = require('flumeview-reduce') // A view that aggregates data
var Flume = require('flumedb')

// Initialize FlumeDB with an in-memory log
var db = Flume(MemLog())
  // Attach a 'sum' view using flumeview-reduce
  .use(
    'sum',
    Reduce(1, function (acc, item) {
      // This reduce function sums the 'foo' property of appended items
      return (acc || 0) + item.foo
    })
  )

// Append some data to the log
db.append({ foo: 1 }, function (err, seq) {
  if (err) throw err
  console.log('Appended item with sequence:', seq)
  
  // Get the current sum from the 'sum' view
  db.sum.get(function (err, value) {
    if (err) throw err
    console.log('Current sum from view:', value) // Expected output: 1
  })
})

db.append({ foo: 5 }, function (err, seq) {
  if (err) throw err
  console.log('Appended another item with sequence:', seq)

  db.sum.get(function (err, value) {
    if (err) throw err
    console.log('Current sum after second append:', value) // Expected output: 6
  })
})

view raw JSON →