{"library":"stream-concat","title":"Node.js Stream Concatenation","type":"library","description":"The `stream-concat` library provides a simple and efficient mechanism for concatenating multiple Node.js readable streams into a single output readable stream. Currently at version `2.0.0`, it maintains an active development status, though a specific release cadence isn't explicitly defined. A core differentiator is its ability to handle streams in two ways: either by accepting an array of pre-existing streams or, more efficiently, by using a user-supplied function that returns the next stream on demand. This 'on-the-fly' stream creation significantly reduces memory consumption and improves performance, particularly when dealing with large datasets or numerous files, by preventing all streams from being buffered simultaneously. It builds upon Node.js's `Transform` stream, leveraging its underlying options like `highWaterMark` and `objectMode` for fine-grained control over stream behavior.","language":"javascript","status":"active","last_verified":"Sat Apr 25","install":{"commands":["npm install stream-concat"],"cli":null},"imports":["const StreamConcat = require('stream-concat');"],"auth":{"required":false,"env_vars":[]},"links":{"homepage":null,"github":"https://github.com/sedenardi/node-stream-concat","docs":null,"changelog":null,"pypi":null,"npm":"https://www.npmjs.com/package/stream-concat","openapi_spec":null,"status_page":null,"smithery":null},"quickstart":{"code":"const fs = require('fs');\nconst StreamConcat = require('stream-concat');\nconst path = require('path');\n\n// Create some dummy files for demonstration\nfs.writeFileSync('file1.csv', 'header1,data1a\\nheader2,data1b\\n');\nfs.writeFileSync('file2.csv', 'header3,data2a\\nheader4,data2b\\n');\nfs.writeFileSync('file3.csv', 'header5,data3a\\nheader6,data3b\\n');\n\nconst fileNames = ['file1.csv', 'file2.csv', 'file3.csv'];\nlet fileIndex = 0;\n\n// Use a function to defer stream creation, reducing memory footprint\nconst nextStream = () => {\n  if (fileIndex === fileNames.length) {\n    return null;\n  }\n  console.log(`Reading: ${fileNames[fileIndex]}`);\n  return fs.createReadStream(fileNames[fileIndex++]);\n};\n\nconst output = fs.createWriteStream('combined.csv');\n\nconst combinedStream = new StreamConcat(nextStream);\n\ncombinedStream.pipe(output)\n  .on('finish', () => {\n    console.log('All files combined into combined.csv');\n    // Clean up dummy files\n    fileNames.forEach(file => fs.unlinkSync(file));\n    fs.unlinkSync('combined.csv');\n  })\n  .on('error', (err) => {\n    console.error('Error combining streams:', err);\n  });\n","lang":"javascript","description":"This example demonstrates how to concatenate multiple files efficiently using a `nextStream` function. This approach defers opening new streams until they are needed, optimizing memory usage compared to providing an array of all streams upfront.","tag":null,"tag_description":null,"last_tested":null,"results":[]},"compatibility":null}