buffer

Module Export

[value] |> [value]
[value] buffer by number |> [[value]]

Buffers an iterator into a static list, or splits it into fixed-size chunks.

The buffer function comes in two forms:

Without arguments — Loads the entire iterator into memory as a list, so it can be enumerated multiple times without changing, or to trade memory for speed when dealing with complex iterator transforms.

With by <count> — Splits the iterator into chunks of <count> items each, returning an iterator of lists. The final chunk may contain fewer items if the source length is not a multiple of <count>. Memory usage is bounded by the chunk size, so this is the streaming-friendly way to process a large source in groups.

Examples

Buffering a stream

import 'iterators'

from myStream buffer

Chunking into fixed-size groups

import 'iterators'

from [1, 2, 3, 4, 5] buffer by 2

Result:

[[1, 2], [3, 4], [5]]

Processing a stream in batches

Combining buffer by with select lets you process a large stream in fixed-size groups while keeping constant memory:

import 'iterators'

from
  myStream
  buffer by 100
  select chunk => chunk count