Node.js

Buffers & Streams

1. Introduction 🌊

Buffers and streams are how Node handles raw binary data and large volumes of data efficiently, without loading everything into memory at once. Whether you're processing uploads, reading video files, or piping compressed data across a network, these two building blocks are essential.

2. What are Buffers? 🧱

A Buffer is a fixed-size chunk of raw binary memory, outside V8's usual string/object heap. Buffers represent data the way it exists at the byte level — essential for handling files, network protocols, and binary formats.

Fixed length once allocated — cannot be resized.
Available globally without requiring an import.
Backed by V8's ArrayBuffer under the hood.

3. Creating Buffers 🏗️

creating-buffers.js

Buffer.alloc(10);              // 10 zero-filled bytes
Buffer.allocUnsafe(10);        // 10 uninitialized bytes (faster, riskier)
Buffer.from('hello', 'utf-8'); // from a string
Buffer.from([72, 101, 108, 108, 111]); // from an array of bytes

Caution

Buffer.allocUnsafe() is faster because it doesn't zero out memory — the returned buffer may contain old, sensitive data until you overwrite it.

4. Reading Buffers 🔍

reading-buffers.js

const buf = Buffer.from('hello');

console.log(buf[0]);           // 104 (byte value of 'h')
console.log(buf.toString());   // 'hello'
console.log(buf.toString('hex')); // '68656c6c6f'
console.log(buf.length);       // 5

5. Writing Buffers ✍️

writing-buffers.js

const buf = Buffer.alloc(5);
buf.write('abc');
console.log(buf); // <Buffer 61 62 63 00 00>

6. Buffer Encoding 🔤

Buffers can be converted to and from strings using various encodings, each interpreting the underlying bytes differently.

Encoding	Use Case
utf-8	Default text encoding
base64	Encoding binary data as ASCII text
hex	Human-readable byte-level debugging
ascii	Legacy 7-bit text

encoding-example.js

const buf = Buffer.from('Node.js');
console.log(buf.toString('base64')); // 'Tm9kZS5qcw=='

7. Buffer Methods 🧰

buffer-methods.js

const a = Buffer.from('abc');
const b = Buffer.from('abd');

Buffer.concat([a, b]);       // joins multiple buffers
a.equals(b);                 // false
a.compare(b);                // -1 (a < b)
a.slice(0, 2);                // <Buffer 61 62> (view, not a copy)
a.copy(Buffer.alloc(3));     // copies bytes into target

Important

buf.slice() returns a view onto the same underlying memory — modifying the slice modifies the original buffer too.

8. Binary Data 💾

Buffers are the right tool whenever data isn't meant to be interpreted as text: images, audio, compressed archives, or custom binary protocols.

binary-data.js

const fs = require('node:fs/promises');

const imageBuffer = await fs.readFile('./logo.png');
console.log(imageBuffer.readUInt8(0)); // first byte of the PNG header

9. Streams Overview 🚿

A stream processes data incrementally, in chunks, rather than requiring the whole dataset to be available at once. All streams in Node are instances of EventEmitter.

Stream Types

Readable — data flows out

Writable — data flows in

Duplex — both directions

Transform — duplex that modifies data in transit

10. Readable Streams 📥

readable-stream.js

const fs = require('node:fs');

const readable = fs.createReadStream('./large-file.txt', { encoding: 'utf-8' });

readable.on('data', (chunk) => {
  console.log(`Received ${chunk.length} characters`);
});

readable.on('end', () => console.log('No more data'));

11. Writable Streams 📤

writable-stream.js

const fs = require('node:fs');

const writable = fs.createWriteStream('./output.txt');

writable.write('First line\n');
writable.write('Second line\n');
writable.end('Final line\n');

writable.on('finish', () => console.log('All writes flushed'));

12. Duplex Streams ↔️

A Duplex stream implements both Readable and Writable interfaces independently — data going in isn't necessarily related to data coming out. A net.Socket is a classic example.

duplex-stream.js

const { Duplex } = require('node:stream');

const duplex = new Duplex({
  read(size) {
    this.push('some data');
    this.push(null);
  },
  write(chunk, encoding, callback) {
    console.log('Received:', chunk.toString());
    callback();
  },
});

13. Transform Streams 🔄

A Transform stream is a special Duplex where the output is derived from the input — think compression, encryption, or text case conversion.

transform-stream.js

const { Transform } = require('node:stream');

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

process.stdin.pipe(upperCaseTransform).pipe(process.stdout);

14. Stream Events 📡

Event	Fires When
'data'	A chunk is available to read
'end'	No more data will be provided (readable)
'finish'	All data has been flushed (writable)
'error'	An error occurs anywhere in the stream
'close'	The underlying resource has been closed

15. Piping Streams 🔗

.pipe() connects a Readable stream's output directly to a Writable stream's input, automatically managing data flow and backpressure.

piping.js

const fs = require('node:fs');

fs.createReadStream('./input.txt')
  .pipe(fs.createWriteStream('./output.txt'));

16. Backpressure 🚦

Backpressure occurs when a Writable stream can't consume data as fast as a Readable stream produces it. .pipe() handles this automatically by pausing the readable side until the writable side signals it's ready via the 'drain' event.

backpressure-manual.js

function writeData(writable, data) {
  const canContinue = writable.write(data);
  if (!canContinue) {
    writable.once('drain', () => console.log('Buffer drained, safe to write more'));
  }
}

Warning

Ignoring backpressure by calling .write() in a tight loop without checking its return value can cause unbounded memory growth.

17. Stream Chaining ⛓️

stream-chaining.js

const fs = require('node:fs');
const zlib = require('node:zlib');

fs.createReadStream('./access.log')
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream('./access.log.gz'));

18. File Streams 📄

Covered in depth in the File System tutorial — fs.createReadStream() and fs.createWriteStream() are the primary entry points for streaming files.

19. Network Streams 🌐

HTTP request and response objects, as well as net.Socket connections, are all streams — enabling data to be processed as it arrives over the network rather than waiting for the full payload.

network-stream.js

const http = require('node:http');

const server = http.createServer((req, res) => {
  req.pipe(res); // echo the request body back as the response
});

server.listen(3000);

20. Compression Streams 🗜️

compression.js

const zlib = require('node:zlib');
const fs = require('node:fs');

fs.createReadStream('./data.json')
  .pipe(zlib.createBrotliCompress())
  .pipe(fs.createWriteStream('./data.json.br'));

21. Custom Streams 🛠️

custom-readable.js

const { Readable } = require('node:stream');

class CounterStream extends Readable {
  constructor(max) {
    super();
    this.current = 0;
    this.max = max;
  }

  _read() {
    if (this.current < this.max) {
      this.push(String(this.current++));
    } else {
      this.push(null); // signal end of stream
    }
  }
}

new CounterStream(5).pipe(process.stdout);

22. Object Mode 📦

By default, streams work with Buffer/string chunks. Object mode ({ objectMode: true }) allows a stream to emit any JavaScript value, useful for pipelines of structured data rather than raw bytes.

object-mode.js

const { Transform } = require('node:stream');

const toUpperName = new Transform({
  objectMode: true,
  transform(record, encoding, callback) {
    this.push({ ...record, name: record.name.toUpperCase() });
    callback();
  },
});

23. Stream Pipeline 🧵

pipeline() (from node:stream/promises) is the recommended way to connect streams: it properly propagates errors and cleans up resources, unlike chained .pipe() calls alone.

pipeline-example.js

const { pipeline } = require('node:stream/promises');
const fs = require('node:fs');
const zlib = require('node:zlib');

async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    zlib.createGzip(),
    fs.createWriteStream(output)
  );
  console.log('Compression complete');
}

Best Practice

Prefer pipeline() over manually chained .pipe() calls — it automatically destroys all streams if any of them errors.

24. Performance Optimization ⚡

Use Buffer.allocUnsafe() only when you'll immediately overwrite every byte.
Stream large files instead of buffering them fully in memory.
Tune the highWaterMark option to balance memory usage against throughput.
Avoid unnecessary encoding conversions between Buffer and string in hot paths.

25. Best Practices ✅

Always use pipeline() instead of raw .pipe() chains in production code.
Handle the 'error' event on every stream you create manually.
Respect the boolean return value of .write() to avoid backpressure-related memory bloat.
Prefer Buffer.from() over the deprecated new Buffer() constructor.

26. Common Mistakes ⚠️

Using the deprecated new Buffer(size) constructor instead of Buffer.alloc()/Buffer.from().
Ignoring the 'error' event on streams, leading to silent failures or crashes.
Chaining .pipe() without handling errors on every stream in the chain.
Assuming buf.slice() creates an independent copy rather than a shared view.
Writing to a stream in a loop without checking for backpressure.

27. Frequently Asked Questions ❓

Question

What's the difference between Buffer.alloc and Buffer.allocUnsafe?

Answer

Buffer.alloc() zero-fills the memory (safer, slightly slower); Buffer.allocUnsafe() skips that step for performance, but may expose old memory contents until overwritten.

Question

Why use pipeline() instead of .pipe()?

Answer

pipeline() automatically forwards errors and destroys all streams in the chain if one fails, preventing resource leaks that plain .pipe() chains are prone to.

Question

What is object mode used for?

Answer

Object mode lets a stream carry structured JavaScript values instead of raw bytes — useful for pipelines that process records, rows, or parsed objects rather than binary data.

28. Summary 📝

Summary

Buffers give Node direct access to raw binary data, while streams let that data — or any structured data in object mode — flow through a program in manageable chunks. Understanding Readable, Writable, Duplex, and Transform streams, along with backpressure and the pipeline() helper, is essential for building memory-efficient, production-grade Node applications.

1. Introduction 🌊

2. What are Buffers? 🧱

3. Creating Buffers 🏗️

creating-buffers.js

🟡Caution

4. Reading Buffers 🔍

reading-buffers.js

5. Writing Buffers ✍️

writing-buffers.js

6. Buffer Encoding 🔤

encoding-example.js

7. Buffer Methods 🧰

buffer-methods.js

❗Important

8. Binary Data 💾

binary-data.js

9. Streams Overview 🚿

10. Readable Streams 📥

readable-stream.js

11. Writable Streams 📤

writable-stream.js

12. Duplex Streams ↔️

duplex-stream.js

13. Transform Streams 🔄

transform-stream.js

14. Stream Events 📡

15. Piping Streams 🔗

piping.js

16. Backpressure 🚦

backpressure-manual.js

⚠️Warning

17. Stream Chaining ⛓️

stream-chaining.js

18. File Streams 📄

19. Network Streams 🌐

network-stream.js

20. Compression Streams 🗜️

compression.js

21. Custom Streams 🛠️

custom-readable.js

22. Object Mode 📦

object-mode.js

23. Stream Pipeline 🧵

pipeline-example.js

⭐Best Practice

24. Performance Optimization ⚡

25. Best Practices ✅

26. Common Mistakes ⚠️

27. Frequently Asked Questions ❓

❓Question

💬Answer

❓Question

💬Answer

❓Question

💬Answer

28. Summary 📝

📋Summary

Caution

Important

Warning

Best Practice

Question

Answer

Question

Answer

Question

Answer

Summary