walk.js/README.md

Walk.js (@root/walk)

Walk a directory recursively and handle each entity (files, directories, symlnks, etc).

(a port of Go's filepath.Walk using Node.js v10+'s fs.readdir's withFileTypes and ES 2021)

await Walk.walk(pathname, walkFunc);

async function walkFunc(err, pathname, dirent) {
  // err is failure to lstat a file or directory
  // pathname is relative path, including the file or folder name
  // dirent = { name, isDirectory(), isFile(), isSymbolicLink(), ... }

  if (err) {
    return false;
  }
  console.log(pathname);
}

Table of Contents

• Install
• Usage
  • CommonJS
  • ES Modules
• API
  • Walk.walk
    • walkFunc
    • Example: filter dotfiles
  • Walk.create
    • withFileStats
    • sort (and filter)
• Node walk in <50 Lines of Code
• License (MPL-2.0)

Install

npm install --save @root/walk

Usage

You can use this with Node v12+ using Vanilla JS (CommonJS) or ES2021 (ES Modules).

CommonJS (Vanilla JS / ES5)

var Walk = require("@root/walk");
var path = require("path");

Walk.walk("./", walkFunc).then(function () {
  console.log("Done");
});

// walkFunc must be async, or return a Promise
function walkFunc(err, pathname, dirent) {
  if (err) {
    // throw an error to stop walking
    // (or return to ignore and keep going)
    console.warn("fs stat error for %s: %s", pathname, err.message);
    return Promise.resolve();
  }

  // return false to skip a directory
  // (ex: skipping "dot file" directories)
  if (dirent.isDirectory() && dirent.name.startsWith(".")) {
    return Promise.resolve(false);
  }

  // fs.Dirent is a slimmed-down, faster version of fs.Stats
  console.log("name:", dirent.name, "in", path.dirname(pathname));
  // (only one of these will be true)
  console.log("is file?", dirent.isFile());
  console.log("is link?", dirent.isSymbolicLink());

  return Promise.resolve();
}

ECMAScript 2021 (ES Modules)

@root/walk can be used with async/await or Promises.

import { walk } from "@root/walk";
import path from "path";

const walkFunc = async (err, pathname, dirent) => {
  if (err) {
    throw err;
  }

  if (dirent.isDirectory() && dirent.name.startsWith(".")) {
    return false;
  }

  console.log("name:", dirent.name, "in", path.dirname(pathname));
};

await walk("./", walkFunc);

console.log("Done");

API Documentation

Walk.walk(pathname, walkFunc)

Walk.walk walks pathname (inclusive) and calls walkFunc for each file system entity.

It can be used with Promises:

Walk.walk(pathname, promiseWalker).then(doMore);

Or with async / await:

await Walk.walk(pathname, asyncWalker);

The behavior should exactly match Go's filepath.Walk with a few exceptions:

• uses JavaScript Promises/async/await
• receives dirent rather than lstat (for performance, see withFileStats)
• optional parameters to change stat behavior and sort order

walkFunc

Handles each directory entry

async function walkFunc(err, pathname, dirent) {
  // `err` is a file system stat error
  // `pathname` is the full pathname, including the file name
  // `dirent` is an fs.Dirent with a `name`, `isDirectory`, `isFile`, etc
  return null;
}

Walk.create(options)

Create a custom walker with these options:

• withFileStats: true walkFunc will receive fs.Stats[] from fs.lstat instead of fs.Dirent[]
• sort: (entities) => entities.sort() sort and/or filter entities before walking them

const walk = Walk.create({
  withFileStats: true,
  sort: (entities) => entities.sort()),
});

withFileStats

By default walk will use fs.readdir(pathname, { withFileTypes: true }) which returns fs.Dirent[], which only has name and file type info, but is much faster when you don't need the complete fs.Stats.

Enable withFileStats to use get full fs.Stats. This will use fs.readdir(pathname) (returning String[]) and then call fs.lstat(pathname) - including mtime, birthtime, uid, etc - right after.

const walk = Walk.create({
  withFileStats: true,
});

walk(".", async function (err, pathname, stat) {
  console.log(stat.name, stat.uid, stat.birthtime, stat.isDirectory());
});

sort (and filter)

Sometimes you want to give priority to walking certain directories first.

The sort option allows you to specify a funciton that modifies the fs.Dirent[] entities (default) or String[] filenames (withFileStats: true).

Since you must return the sorted array, you can also filter here if you'd prefer.

const byNameWithoutDotFiles = (entities) => {
  // sort by name
  // filter dot files
  return entities
    .sort((a, b) => {
      if (a.name > b.name) {
        return 1;
      }
      if (a.name < b.name) {
        return -1;
      }
      return 0;
    })
    .filter((ent) => !ent.name.startsWith("."));
};

const walk = Walk.create({ sort: byNameWithoutDotFiles });

walk(".", async function (err, pathname, stat) {
  // each directories contents will be listed alphabetically
  console.log(pathname);
});

Note: this gets the result of fs.readdir(). If withFileStats is true you will get a String[] of filenames - because this hapens BEFORE fs.lstat() is called - otherwise you will get fs.Dirent[].

node walk in 50 lines of code

If you're like me and you hate dependencies, here's the bare minimum node fs walk function:

See snippet.js or https://therootcompany.com/blog/fs-walk-for-node-js/.

License

The main module, as published to NPM, is licensed the MPL-2.0.

The ~50 line snippet is licensed CC0-1.0 (Public Domain).