Reiterating JavaScript & Node.js iterators

Luciano Mammino (@loige)

RomaJS - June 16, 2021

How many ways do you know to "do iteration" in JavaScript & Node.js? 😰

for i / for...in / for...of
while / do while
Array.forEach / Array.map / Array.flatMap / Array.reduce / Array.reduceRight  / Array.filter / Array.find / Array.findIndex / Array.entries / Array.values / Array.every / Array.some
Object.keys / Object.values / Object.entries
Iterators / Generators
 Spread operator

[...iterable]

Events / Streams

Async iterators / Async generators

for await...of

That was 28 different concepts! 😳

📝 AGENDA

  • Iteration protocols... What? Why?
  • Syntax review
  • Iteration protocols
    • Iterator protocol
    • Iterable protocol
    • Generator functions
    • Async iterator protocol
    • Async iterable protocol
  • Tips & tricks

Let me introduce myself...

I'm Luciano (🇮🇹🍕🍝) 👋

👨‍💻  Senior Architect

Co-Author of Node.js Design Patterns  👉

Connect with me:
 

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

We are business focused technologists that deliver.


Accelerated Serverless | AI as a Service | Platform Modernisation

WE ARE HIRING: Do you want to work with us?

Iteration protocols

what? why? 🤔

  • An attempt at standardizing "iteration" behaviors providing a consistent and interoperable API
    • Use for...of, for await...of and spread operator.
  • You can also create lazy iterators
  • You can also deal with async iteration
  • You can create your own custom iterators/iterables

Syntax Review 🧐

const judokas = [
  'Driulis Gonzalez Morales',
  'Ilias Iliadis',
  'Tadahiro Nomura',
  'Anton Geesink',
  'Teddy Riner',
  'Ryoko Tani'
]

for (const judoka of judokas) {
  console.log(judoka)
}

for...of

Driulis Gonzalez Morales
Ilias Iliadis
Tadahiro Nomura
Anton Geesink
Teddy Riner
Ryoko Tani
OUTPUT
const judoka = 'Ryoko Tani'

for (const char of judoka) {
  console.log(char)
}

for...of (with strings)

R
y
o
k
o 

T
a
n
i
OUTPUT
const medals = new Set([
  'gold',
  'silver',
  'bronze'
])

for (const medal of medals) {
  console.log(medal)
}

for...of (with Set)

gold
silver
bronze

 

OUTPUT
const medallists = new Map([
  ['Teddy Riner', 33],
  ['Driulis Gonzalez Morales', 16],
  ['Ryoko Tani', 16],
  ['Ilias Iliadis', 15]
])

for (const [judoka, medals] of medallists) {
  console.log(`${judoka} has won ${medals} medals`)
}

for...of (with Map)

Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals
OUTPUT
const medallists = {
  'Teddy Riner': 33,
  'Driulis Gonzalez Morales': 16,
  'Ryoko Tani': 16,
  'Ilias Iliadis': 15
}

for (const [judoka, medals] of Object.entries(medallists)) {
  console.log(`${judoka} has won ${medals} medals`)
}

for...of (with object & Object.entries)

Teddy Riner has won 33 medals
Driulis Gonzalez Morales has won 16 medals
Ryoko Tani has won 16 medals
Ilias Iliadis has won 15 medals
OUTPUT
const medallists = {
  'Teddy Riner': 33,
  'Driulis Gonzalez Morales': 16,
  'Ryoko Tani': 16,
  'Ilias Iliadis': 15
}

for (const [judoka, medals] of medallists) {
  console.log(`${judoka} has won ${medals} medals`)
}

for...of (with object literals)

for (const [judoka, medals] of medallists) {
                              ^
TypeError: medallists is not iterable
  at Object. (.../05-for-of-object.js:8:32)
ERROR
const countdown = [3, 2, 1, 0]

// spread into array
const from5to0 = [5, 4, ...countdown]
console.log(from5to0) // [ 5, 4, 3, 2, 1, 0 ]

// spread function arguments
console.log('countdown data:', ...countdown)
// countdown data: 3 2 1 0

Spread operator

import {
  DynamoDBClient,
  paginateListTables
} from '@aws-sdk/client-dynamodb'

const client = new DynamoDBClient({});

for await (const page of paginateListTables({ client }, {})) {
  // page.TableNames is an array of table names
  for (const tableName of page.TableNames) {
    console.log(tableName)
  }
}

for await...of (async iterable)

Iteration Protocols 😵‍💫

Iterator protocol

In JavaScript, an object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value.

function createCountdown (start) {
  let nextVal = start
  return {
    next () {
      if (nextVal < 0) {
        return { done: true }
      }
      return {
        done: false,
        value: nextVal--
      }
    }
  }
}

Countdown iterator

const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }

Iterable protocol

An object is iterable if it implements the @@iterator* method, a zero-argument function that returns an iterator.

 

* Symbol.iterator

function createCountdown (start) {
  let nextVal = start
  return {
    [Symbol.iterator]: () => ({
      next () {
        if (nextVal < 0) {
          return { done: true }
        }
        return {
          done: false, 
          value: nextVal--
        }
      }
    })
  }
}

Countdown iterable

const countdown = createCountdown(3)
for (const value of countdown) {
  console.log(value)
}

// 3
// 2
// 1
// 0

Can an object be both an iterator and an iterable?! 🤨

const iterableIterator = {
  next() {
    return { done: false, value: "hello" }
  },
  [Symbol.iterator]() {
    return this
  }
}

Iterator + Iterable

iterableIterator.next()
// { done: false, value: "hello" }

for (const value of iterableIterator) {
 console.log(value)
}

// hello
// hello
// hello
// ...

Generators

A generator function "produces" an object that is both an iterator and an iterable! 🤯

function * createCountdown (start) {
  for (let i = start; i >= 0; i--) {
    yield i
  }
}
// As iterable
const countdown = createCountdown(3)
for (const value of countdown) {
 console.log(value)
}
// 3
// 2
// 1
// 0
// As iterator
const countdown = createCountdown(3)
console.log(countdown.next()) // { done: false, value: 3 }
console.log(countdown.next()) // { done: false, value: 2 }
console.log(countdown.next()) // { done: false, value: 1 }
console.log(countdown.next()) // { done: false, value: 0 }
console.log(countdown.next()) // { done: true }

Well, what about async iteration? 🤌

Async Iterator protocol

An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value.

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (start, delay = 1000) {
  let nextVal = start
  return {
    async next () {
      await setTimeout(delay)
      if (nextVal < 0) {
        return { done: true }
      }
      return { done: false, value: nextVal-- }
    }
  }
}
const countdown = createAsyncCountdown(3)
console.log(await countdown.next()) // { done: false, value: 3 }
console.log(await countdown.next()) // { done: false, value: 2 }
console.log(await countdown.next()) // { done: false, value: 1 }
console.log(await countdown.next()) // { done: false, value: 0 }
console.log(await countdown.next()) // { done: true }

Async Iterable protocol

An object is an async iterable if it implements the @@asyncIterator* method, a zero-argument function that returns an async iterator.


* Symbol.asyncIterator

import { setTimeout } from 'timers/promises'

function createAsyncCountdown (start, delay = 1000) {
  return {
    [Symbol.asyncIterator]: function () {
      let nextVal = start
      return {
        async next () {
          await setTimeout(delay)
          if (nextVal < 0) {
            return { done: true }
          }
          return { done: false, value: nextVal-- }
        }
      }
    }
  }
}
const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value) // 3 ... 2 ... 1 ... 0
}
import { setTimeout } from 'timers/promises'

async function * createAsyncCountdown (start, delay = 1000) {
  for (let i = start; i >= 0; i--) {
    await setTimeout(delay)
    yield i
  }
}
const countdown = createAsyncCountdown(3)

for await (const value of countdown) {
  console.log(value) // 3 ... 2 ... 1 ... 0
}

When to use async iterators

  • Sequential iteration pattern

  • Data arriving in order over time

  • You need to complete processing the current “chunk” before you can request the next one

  • Examples

    • paginated iteration

    • consuming tasks from a remote queue

Tips & pitfalls ☠️

import { createReadStream } from 'fs'

const sourceStream = createReadStream('bigdata.csv')

let bytes = 0
for await (const chunk of sourceStream) {
  bytes += chunk.length
}

console.log(`bigdata.csv: ${bytes} bytes`)

Node.js readable streams are async iterators

What about
backpressure? 😩

import { createReadStream } from 'fs'
import { once } from 'events'

const sourceStream = createReadStream('bigdata.csv')
const destStream = new SlowTransform()

for await (const chunk of sourceStream) {
  const canContinue = destStream.write(chunk)
  if (!canContinue) {
    // backpressure, now we stop and we need to wait for drain
    await once(destStream, 'drain')
    // ok now it's safe to resume writing
  }
}

But if you are dealing with streaming pipelines it's probably easier to use pipeline().

import { pipeline } from 'stream/promises'
import { createReadStream, createWriteStream } from 'fs'
import { createBrotliCompress } from 'zlib'

const sourceStream = createReadStream('bigdata.csv')
const compress = createBrotliCompress()
const destStream = createWriteStream('bigdata.csv.br')

await pipeline(
  sourceStream,
  compress,
  destStream
)

In Node.js we can convert any Event Emitter to an Async Iterator! 😱

import { on } from 'events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}
import { on } from 'events'
import glob from 'glob' // from npm

const matcher = glob('**/*.js')

for await (const [filePath] of on(matcher, 'match')) {
  console.log(filePath)
}

// ⚠️  DANGER, DANGER (high voltage ⚡️): We'll never get here!
console.log('ALL DONE! :)')
import { on } from 'events'
import glob from 'glob'

const matcher = glob('**/*.js')
const ac = new global.AbortController()

matcher.once('end', () => ac.abort())

try {
  for await (const [filePath] of on(matcher, 'match', { signal: ac.signal })) {
    console.log(`./${filePath}`)
  }
} catch (err) {
  if (!ac.signal.aborted) {
    console.error(err)
    process.exit(1)
  }
  // we ignore the AbortError
}

console.log('NOW WE GETTING HERE! :)') // YAY! 😻

NOTE:
If you know ahead of time how many events you need to process you can also use a break in the for...await loop.

LAST TIP:

Can we use async iterators to handle web requests a-la-Deno? 🦕

import { createServer } from 'http'
import { on } from 'events'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  res.end('hello dear friend')
}

EASY PEASY LEMON SQUEEZY! 🍋

 

But... wait, aren't we processing all requests in series, now? 😱

import { createServer } from 'http'
import { on } from 'events'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  // ... AS LONG AS WE DON'T USE await HERE, WE ARE FINE!
}
import { createServer } from 'http'
import { on } from 'events'
import { setTimeout } from 'timers/promises'

const server = createServer()
server.listen(8000)

for await (const [req, res] of on(server, 'request')) {
  await setTimeout(1000)
  res.end('hello dear friend')
}

Let's stick to the basics... 😅

import { createServer } from 'http'
import { setTimeout } from 'timers/promises'

const server = createServer(async function (req, res) {
  await setTimeout(1000)
  res.end('hello dear friend')
})

server.listen(8000)

Conclusion 🤓

  • Iterable protocols are a way to standardize iteration in JavaScript and Node.js
  • Async iterators are ergonomic tools for sequential asynchronous iteration
  • But don't use them for everything!
    • Consuming data from paginated APIs or reading messages from a queue are good examples!
    • Handling web requests or events from an emitter might not be the best use cases!

Want to learn more? 👩‍🏫

If you enjoyed this talk, you might also enjoy nodejsdp.link 😛

Let's connect:

  loige.co (blog)

  @loige (twitter)

  loige (twitch)

  lmammino (github)

Reiterating JavaScript & Node.js iterators

By Luciano Mammino

Reiterating JavaScript & Node.js iterators

In this talk we will revisit the iteration protocols offered by JavaScript (iterable, iterator, async iterable and async iterator). We will explore generators and async generators as well. We will use this opportunity to discuss some interesting applications of these patterns in Node.js and we will compare them with more traditional approaches like Event Emitters and Node.js Streams. Finally, we will discuss some interesting integrations between traditional approaches and iterators and show some patterns and anti-patterns.

  • 3,072