JavaScript Iteration Protocols

Luciano Mammino (@loige)

META_SLIDE!

const array = ['foo', 'bar', 'baz']

for (const item of array) {
  console.log(item)
}

 

Prints all the items in the array!

Does it need to be an array? 🤔

Output:
foo
bar
baz
const str = 'foo'

for (const item of str) {
  console.log(item)
}
Output:
f
o
o
const set = new Set(['foo', 'bar', 'baz'])

for (const item of set) {
  console.log(item)
}
Output:
foo
bar
baz
const map = new Map([
  ['foo', 'bar'], ['baz', 'qux']
])

for (const item of map) {
  console.log(item)
}
Output:
[ 'foo', 'bar' ]
[ 'baz', 'qux' ]
const obj = {
  foo: 'bar',
  baz: 'qux'
}

for (const item of obj) {
  console.log(item)
}
Output:
⛔️ Uncaught TypeError: obj is not iterable

OMG `for ... of`

does not work with plain objects! 😱

const obj = {
  foo: 'bar',
  baz: 'qux'
}

for (const item of Object.entries(obj)) {
  console.log(item)
}
Output:
[ 'foo', 'bar' ]
[ 'baz', 'qux' ]
const array = ['foo', 'bar', 'baz']

console.log(...array)
Output:
foo bar baz

spread syntax!

  • Iterators
  • Iterables
  • Async Iterators
  • Async Iterables
  • Generator functions
  • Async Generators

📸 The bigger picture

Knowing iteration protocols allows us:

  • Understand JavaScript better
  • Write more modern, interoperable and idiomatic code
  • Be able to write our own custom iterables (even async)

😥 Why should we care?

Who is this guy !?

👋 I'm Luciano (🇮🇹🍕🍝🤌)

👨‍💻 Senior Architect @ fourTheorem (Dublin 🇮🇪)

📔 Co-Author of Node.js Design Patterns  👉

Let's connect!

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

Always re-imagining

We are a pioneering technology consultancy focused on AWS and serverless

 

Accelerated Serverless | AI as a Service | Platform Modernisation

✉️ Reach out to us at  hello@fourTheorem.com

😇 We are always looking for talent: fth.link/careers

We host a weekly podcast about AWS

📒 AGENDA

  • Generators
  • Iterator protocol
  • Iterable protocol
  • Async Iterator protocol
  • Async Iterable protcol
  • Real-life™️ examples

Let's clone 🐑🐏

git clone https://github.com/lmammino/iteration-protocols-workshop.git
cd iteration-protocols-workshop
npm i

➡️ Generators

Generator fn & obj

function * myGenerator () {
  // generator body
  yield 'someValue'
  // ... do more stuff
}
const genObj = myGenerator()
genObj.next() // -> { done: false, value: 'someValue' }
function * fruitGen () {
  yield '🍑'
  yield '🍉'
  yield '🍋'
  yield '🥭'
}

const fruitGenObj = fruitGen()
console.log(fruitGenObj.next()) // { value: '🍑', done: false }
console.log(fruitGenObj.next()) // { value: '🍉', done: false }
console.log(fruitGenObj.next()) // { value: '🍋', done: false }
console.log(fruitGenObj.next()) // { value: '🥭', done: false }
console.log(fruitGenObj.next()) // { value: undefined, done: true }
function * fruitGen () {
  yield '🍑'
  yield '🍉'
  yield '🍋'
  yield '🥭'
}

const fruitGenObj = fruitGen()
// generator objects are iterable!
for (const fruit of fruitGenObj) {
  console.log(fruit)
}

// 🍑
// 🍉
// 🍋
// 🥭
function * range (start, end) {
  for (let i = start; i < end; i++) {
    yield i
  }
}

// generators are lazy!
for (const i of range(0, Number.MAX_VALUE)) {
  console.log(i)
}

const zeroToTen = [...range(0, 11)]
// generators can be "endless"
function * cycle (values) {
  let current = 0
  while (true) {
    yield values[current]
    current = (current + 1) % values.length
  }
}

for (const value of cycle(['even', 'odd'])) {
  console.log(value)
}

// even
// odd
// even
// ...

📝 Mini-summary

  • A generator function returns a generator object which is both an iterator and an iterable.
  • A generator function uses `yield` to yield a value and pause its execution. The generator object is used to make progress on an instance of the generator (by calling `next()`).
  • Generator functions are a great way to create custom iterable objects.
  • Generator objects are lazy and they can be endless.

📝 Exercise(s)

02-generators/exercises/zip.js
function * take (n, iterable) {
  // take at most n items from iterable and 
  // yield them one by one (lazily)
}
02-generators/exercises/zip.js
function * zip (iterable1, iterable2) {
  // consume the two iterables until any of the 2 completes
  // yield a pair taking 1 item from the first and one from
  // the second at every step
}

➡️ Iterators & Iterables

Iterator Obj

An object that acts like a cursor to iterate over blocks of data sequentially

Iterable Obj

An object that contains data that can be iterated over sequentially

The iterator protocol

An object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

function createCountdown (from) {
  let nextVal = from
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }

      return { 
        done: false,
        value: nextVal--
      }
    }
  }
}

A factory function that creates an iterator

returns an object

... which has a next() method

... which returns an object with

done & value

const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true }

🔥
Generator Objects
 are iterators (and iterables)!

function * createCountdown (from) {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}
const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }

console.log(countdown.next())
// { done: false, value: 2 }

console.log(countdown.next())
// { done: false, value: 1 }

console.log(countdown.next())
// { done: false, value: 0 }

console.log(countdown.next())
// { done: true, value: undefined }

The iterable protocol

An object is iterable if it implements the Symbol.iterator method, a zero-argument function that returns an iterator.

function createCountdown (from) {
  let nextVal = from
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }

        return { done: false, value: nextVal-- }
      }
    })
  }
}
function createCountdown (from) {
  return {
    [Symbol.iterator]: function * () {
      for (let i = from; i >= 0; i--) {
        yield i
      }
    }
  }
}
function * createCountdown () {
  for (let i = from; i >= 0; i--) {
    yield i
  }
}

🔥 or just use generators!

const countdown = createCountdown(3)

for (const value of countdown) {
  console.log(value)
}

// 3
// 2
// 1
// 0

An object can be an iterable and an iterator at the same time!

const iterableIterator = {
  next () {
    return { done: false, value: 'hello' }
  },
  [Symbol.iterator] () {
    return this
  }
}

📝 Mini-summary 1/2

  • An iterator is an object that allows us to traverse a collection

  • The iterator protocol specifies that an object is an iterator if it has a `next()` method that returns an object with the shape `{done, value}`.

    • `done` (a boolean) tells us if the iteration is completed

    • `value` represents the value from the current iteration.

  • You can write an iterator as an anonymous object (e.g. returned by a factory function), using classes or using generators.

📝 Mini-summary 2/2

  • The iterable protocol defines what's expected for a JavaScript object to be considered iterable. That is an object that holds a collection of data on which you can iterate on sequentially.

  • An object is iterable if it implements a special method called `Symbol.iterator` which returns an iterator. (An object is iterable if you can get an iterator from it!)

  • Generator functions produce objects that are iterable.

  • We saw that generators produce objects that are also iterators.

  • It is possible to have objects that are both iterator and iterable. The trick is to create the object as an iterator and to implement a `Symbol.iterator` that returns the object itself (`this`).

📝 Exercise(S)

04-iterable-protocol/exercises/binarytree.js
class BinaryTree {
  // <implementation provided>
  // ...
  // make this iterable!
}
04-iterable-protocol/exercises/entries.js
function entriesIterable (obj) {
  // Return an iterable that produce key/value pairs
  // from obj
}

OK, very cool!
But, so far this is all synchronous iteration.
What about async? 🙄

➡️ ASYNC Iterators & Iterables

The async iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

import { setTimeout } from 'node:timers/promises'

function createAsyncCountdown (from, delay = 1000) {
  let nextVal = from
  return {
    async next () {
      await setTimeout(delay)
      if (nextVal < 0) {
        return { done: true }
      }

      return { done: false, value: nextVal-- }
    }
  }
}
const countdown = createAsyncCountdown(3)
console.log(await countdown.next())
// { done: false, value: 3 }

console.log(await countdown.next())
// { done: false, value: 2 }

console.log(await countdown.next())
// { done: false, value: 1 }

console.log(await countdown.next())
// { done: false, value: 0 }

console.log(await countdown.next())
// { done: true }
import { setTimeout } from 'node:timers/promises'

// async generators "produce" async iterators!

async function * createAsyncCountdown (from, delay = 1000) {
  for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}

The async iterable protocol

An object is an async iterable if it implements the `Symbol.asyncIterator` method, a zero-argument function that returns an async iterator.

import { setTimeout } from 'node:timers/promises'

function createAsyncCountdown (from, delay = 1000) {
  return {
    [Symbol.asyncIterator]: async function * () {
      for (let i = from; i >= 0; i--) {
        await setTimeout(delay)
        yield i
      }
    }
  }
}

HOT TIP 🔥

With async generators we can create objects that are both async iterators and async iterables!

(We don't need to specify Symbol.asyncIterator explicitly!)
import { setTimeout } from 'node:timers/promises'

// async generators "produce" async iterators
// (and iterables!)

async function * createAsyncCountdown (from, delay = 1000) {
  for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}
const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value)
}

📝 Mini-summary 1/2

  • Async iterators are the asynchronous counterpart of iterators.

  • They are useful to iterate over data that becomes available asynchronously (e.g. coming from a database or a REST API).

  • A good example is a paginated API, we could build an async iterator that gives a new page for every iteration.

  • An object is an async iterator if it has a `next()` method which returns a `Promise` that resolves to an object with the shape: `{done, value}`.

  • The main difference with the iterator protocol is that this time `next()` returns a promise.

  • When we call next we need to make sure we `await` the returned promise.

📝 Mini-summary 2/2

  • The async iterable protocol defines what it means for an object to be an async iterable.

  • Once you have an async iterable you can use the `for await ... of` syntax on it.

  • An object is an async iterable if it has a special method called `Symbol.asyncIterator` that returns an async iterator.

  • Async iterables are a great way to abstract paginated data that is available asynchronously or similar operations like pulling jobs from a remote queue.

  • A small spoiler, async iterables can also be used with Node.js streams...

📝 Exercise

06-async-iterable-protocol/exercises/rickmorty.js
function createCharactersPaginator () {
  // return an iterator that produces pages 
  // with name of Rick and Morty characters
  // taken from the API
  // https://rickandmortyapi.com/api/character
}

// This is what we want to support 👇
const paginator = createCharactersPaginator()
for await (const page of paginator) {
  console.log(page)
}

➡️ Tips &

ASYNC Iterators in Node.js

Is this object an iterable?

function isIterable (obj) {
  return typeof obj[Symbol.iterator] === 'function'
}

const array = [1, 2, 3]
console.log(array, isIterable(array)) // true

const genericObj = { foo: 'bar' }
console.log(isIterable(genericObj)) // false
console.log(isIterable(Object.entries(genericObj))) // true

const fakeIterable = {
  [Symbol.iterator] () { return 'notAnIterator' }
}
console.log(fakeIterable, isIterable(fakeIterable)) // true 😡

Is this object an Async iterable?

function isAsyncIterable (obj) {
  return typeof obj[Symbol.asyncIterator] === 'function'
}

Are there async iterable objects in Node.js core?

Readable Streams!

console.log(
  typeof process.stdin[Symbol.asyncIterator] === 'function'
) // true!

let bytes = 0

for await (const chunk of process.stdin) {
  bytes += chunk.length
}

console.log(`${bytes} bytes read from stdin`)

⚠️ Backpressure warning

import { createWriteStream } from 'node:fs'

const dest = createWriteStream('data.bin')

let bytes = 0
for await (const chunk of process.stdin) {
  // what if we are writing too much too fast?!! 😱
  dest.write(chunk)
  bytes += chunk.length
}
dest.end()

console.log(`${bytes} written into data.bin`)
import { createWriteStream } from 'node:fs'
import { once } from 'node:events'

const dest = createWriteStream('data.bin')

let bytes = 0
for await (const chunk of process.stdin) {
  const canContinue = dest.write(chunk)
  bytes += chunk.length
  if (!canContinue) {
    // backpressure, now we stop and we need to wait for drain
    await once(dest, 'drain')
    // ok now it's safe to resume writing
  }
}
dest.end()

console.log(`${bytes} written into data.bin`)

handling backpressure like a pro!

... or you can use pipeline()

import { pipeline } from 'node:stream/promises'
import {createReadStream, createWriteStream} from 'node:fs'


await pipeline(
  createReadStream('lowercase.txt'),
  async function* (source) {
    for await (const chunk of source) {
      yield await processChunk(chunk)
    }
  },
  createWriteStream('uppercase.txt')
)

console.log('Pipeline succeeded.')

pipeline() supports async iterables! 😱

import { on } from 'node:events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}

creates an async iterable that will yield every time the `match` event happens

import { on } from 'node:events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}

// ⚠️ WE WILL NEVER GET HERE 👇
console.log('Done')

This loop doesn't know when to stop!

You could pass an

AbortController here

for more fine-grained

control

📝 Final summary

  • You can check if an object is an iterable by checking `typeof obj[Symbol.iterator] === 'function'`

  • You can check if an object is an async iterable with `typeof obj[Symbol.asyncIterator] === 'function'`

  • In both cases, there's no guarantee that the iterable protocol is implemented correctly (the function might not return an iterator 😥)

  • Node.js Readable streams are also async iterable objects, so you could use `for await ... of` to consume the data in chunks

  • If you do that and you end up writing data somewhere else, you'll need to handle backpressure yourself. It might be better to use `pipeline()` instead.

  • You can convert Node.js event emitters to async iterable objects by using the `on` function from the module `events`.

Cover picture by Camille Minouflet on Unsplash

Back cover picture by Antonino Cicero on Unsplash

THANKS! 🙌

JavaScript Iteration Protocols - Workshop NodeConf EU 2022

By Luciano Mammino

JavaScript Iteration Protocols - Workshop NodeConf EU 2022

How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for...in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!

  • 2,135