Broccoli Plugins

The first question you should ask yourself is “what is a Broccoli plugin?”.

Plugins are what a build pipeline developer will interact with most. Plugins are what do the actual work of transforming files at each step of build process. The API of a plugin requires just 2 steps, creating a class that extends the broccoli-plugin base class, and implementing a build() method that performs some work and/or returns a promise.

Let’s have a look at the basic building blocks of a Broccoli plugin.

Broccoli-Plugin base class

const Plugin = require('broccoli-plugin');

class MyPlugin extends Plugin {
    build() {
        // A plugin can receive single or multiple inputs nodes/directories
        // The files are available in this.inputPaths[0]/this.inputPaths[1]...
        // You can do anything with these files that you can do in node
        // The output of your plugin must write to this.outputPath
    }
}

Now you may be asking yourself why we are using the CommonJS module syntax here and not ESM like we do in the Brocfile.js. The reason for this is because ESM syntax is only supported from Broccoli 2.1.0 and is not currently supported in Ember-CLI which uses Broccoli.js for its build system, so the recommendation is to use CommonJS for now.

Constructor

    constructor (inputNode, options) {
        super([inputNode], options);
    }

inputNode: This can be one of two things, a string or another Broccoli plugin. Some plugins will require multiple inputNodes and some only require a single. Think about what your plugin is doing, does it need to operate on multiple input directories or a single one. The majority of plugins only require a single directory. The broccoli-plugin base class requires this argument to be an array however, so always make sure pass an array to super().

If a string is passed, Broccoli expects this to be a source directory within the project and automatically converts this into a broccoli-source plugin.

options: (all optional)

{
    name: '',
    annotation: '',
    persistentOutput: false,
    needsCache: false,
}

options.name: Custom name used when debugging/printing stack traces. Broccoli will use the name of your plugin if this field is not supplied.

options.annotation: In addition to the plugin name, Broccoli uses annotations to provide a descriptive label used during debugging/printing stack traces. This is not often set by the plugin author, but by the consumer of the plugin to tell multiple instances of the same plugin apart.

options.persistentOutput: By default, Broccoli will provide an empty outputPath for the plugin to write files to. This applies when running the serve command, where the Broccoli process is still running and build() is called on a plugin again when some input files have changed and a rebuild is triggered.

If this option is true, the output directory is not automatically emptied between rebuilds. This may be useful if your plugin implements caching to allow the build method to be skipped. Whenever the Broccoli process exits (after build or quitting serve), all directories are deleted.

options.needsCache: Despite the name, needsCache doesn’t provide caching for the plugin. If true, a directory is created for the plugin to store temporary files that may be needed between rebuilds that are not included in the build output. Broccoli sets the path to this directory to this.cachePath.

Build

Build can do anything you can do in node. Additionally, if build() returns a promise, Broccoli will wait until the promise resolves before continuing the rest of the build. Broccoli only builds one plugin at a time in order, from top to bottom of the build graph. The build method is where the grunt (lol, intentional pun) of the work happens.

This function will typically access the following properties:

this.inputPaths: An array of paths on disk corresponding to each node in inputNodes. Your plugin will read files from these paths. These paths are auto-generated by Broccoli and will typically be the output path of that a previous plugin has written to. this.inputPaths remains consistent between rebuilds.

this.outputPath: Broccoli will automatically create a directory for this plugin to write to when it starts up. Your plugin must write files to this path, and Broccoli will use this directory as an inputPath to the next plugin. This directory is emptied by Broccoli before each build, unless the persistentOutput option is true. this.outputPath remains consistent between rebuilds.

this.cachePath: The path on disk to an auxiliary cache directory. Use this to store files that you want preserved between builds but do not end up in your outputPath. This path is only set when the needsCache option is true, and the directory will only be deleted when the Broccoli process exits.

Note: all paths are setup by Broccoli when the process starts, before build() is called for the first time, and stay consistent between rebuilds during serve, you can rely on these paths not changing between plugin.build() being called by Broccoli as changes to source files are detected during serve.

Example plugin

Let’s build a sample concatenation plugin. It’s going to concatenate a set of files matching a glob expression.

const Plugin = require('broccoli-plugin');
const walkSync = require('walk-sync');
const fs = require('fs');

class ConcatPlugin extends Plugin {
    constructor(inputNodes, options) {
        super(inputNodes, options);

        this.fileMatchers = options.globs || ['**/*'];
        this.joinSeparator = options.joinSeparator === undefined || "\n";
        this.outputFile = options.outputFile || 'concat';
    }

    build() {
        const walkOptions = {
            includeBasePath: true,
            directories: false,
            globs: this.fileMatchers,
        };

        const content = this.inputPaths
            .reduce((output, inputPath) => output + this.joinSeparator +
                walkSync(inputPath, walkOptions)
                    .map(file => fs.readFileSync(file, { encoding: 'UTF-8' }))
                    .join(this.joinSeparator),
            '');

        fs.writeFileSync(`${this.outputPath}/${this.outputFile}`, content);
    }
}

module.exports = function concatPlugin(...params) {
    return new ConcatPlugin(...params);
}

module.exports.ConcatPlugin = ConcatPlugin;

Let’s take a look and see what’s happening here.

First, we are importing our dependencies. We are using 3 packages, the broccoli-plugin base class that we are extending, a package called walk-sync which synchronously walks a directory recursively, and the Node fs package.

After this we define the class:

class ConcatPlugin extends Plugin {
    constructor(inputNodes, options) {
        super(inputNodes, options);

        this.fileMatchers = options.globs || ['**/*'];
        this.joinSeparator = options.joinSeparator || "\n";
        this.outputFile = options.outputFile || 'concat';
    }
}

The constructor accepts multiple inputNodes and an options hash. As you can see we are defining 3 options, a default glob expression, the join character and the output file name. These will be used to inform how our build method should work.

Next up we define our build() method. First we setup some options for the walk-sync package then we iterate this.inputPaths using a reduce function. If you’ve not used reduce before, it provides a simple functional programming way of iterating an input and combining the output into an accumulator. In our case we are merely concatenating the output together into one big string.

    build() {
        const walkOptions = {
            includeBasePath: true,
            directories: false,
            globs: this.fileMatchers,
        };

        const content = this.inputPaths
            .reduce((output, inputPath) => output + this.joinSeparator +
                walkSync(inputPath, walkOptions)
                    .map(file => fs.readFileSync(file, { encoding: 'UTF-8' }))
                    .join(this.joinSeparator),
            '');

        fs.writeFileSync(`${this.outputPath}/${this.outputFile}`, content);
    }

Next we iterate each file in the inputPath and read its contents. We do this via the map function, which transforms the array of files into an array of the contents of each file. We then join all the files together with the separator character, which is returned as the result to the reduce method above.

After the above is complete, we now have all of the file contents within the content variable and all that is left is to write that to the outputFile.

That’s it, plugin complete. In our case, the plugin is entirely synchronous and as such just returns at the end. If we needed the plugin to be asynchronous, we could alternatively return a promise and Broccoli would wait until the promise resolved before continuing the build.

Lastly, we export a default function that will be used when this plugin is imported into a build pipeline, and export the class as a named property at the bottom so that other plugin developers can extend it. The name of the class export should match the class, as this is how an ESM export would look so ensures best forward compatibility for the future.

module.exports = function concatPlugin(...params) {
    return new ConcatPlugin(...params);
}

module.exports.ConcatPlugin = ConcatPlugin;

E.g.

// Brocfile.js
const concat = require('concat-plugin');

export default () => concat(['dir1', 'dir2']);

And the class can be imported as follows by another plugin developer:

const { ConcatPlugin } = require('concat-plugin');

class MyPlugin extends ConcatPlugin {

As you can see, there isn’t really any magic happening here, it’s all standard Node code, just wrapped in a build() method that is provided an array of inputPaths and an outputPath to write to.

Caching

Adding caching to a plugin is relatively straightforward. Caching allows a plugin to skip its build() method if it doesn’t need to do anything because its inputs have not changed. We can use a couple of node packages to achieve this. fs-tree-diff is used to calculate a diff between the 2 states of a directory, and walk-sync which synchronously walks all the files in a directory and produces a list, which is passed to fs-tree-diff.

Let’s take a look:

const Plugin = require('broccoli-plugin');
const FSTree = require('fs-tree-diff');
const walkSync = require('walk-sync');

class Cacher extends Plugin {
  constructor (inputNodes, options) {
    super(inputNodes, {
      ...options,
      persistentOutput: true,
    });

    this._previous = [];
  }

  _hasChanged() {
    let changed = false;
    for (let inputPath of this.inputPaths) {
      const current = FSTree.fromEntries(walkSync.entries(inputPath));
      const patch = current.calculatePatch(this._previous[inputPath] || []);
      this._previous[inputPath] = current;

      if (patch.length) {
        changed = true;
      }
    }

    return changed;
  }

  build() {
    if (!this._hasChanged()) {
      return;
    }

    // do work
  }
}

module.exports = function cacher(...params) {
  return new Cacher(...params);
}

module.exports.Cacher = Cacher;

Let’s examine the above. First off we’re importing the packages, nothing exciting there. In the constructor() we’re setting persistentOutput in the options hash to true, this ensures that Broccoli does not delete the output directory before calling build(), we need this so our output is preserved between rebuilds and we can reuse it. Note that when the Broccoli process exits (after build or quitting serve), all directories are deleted, nothing is preserved between builds.

  constructor (inputNodes, options) {
    super(inputNodes, {
      ...options,
      persistentOutput: true,
    });

    this._previous = [];
  }

Next, we’ve added a _hasChanged() method that does our cache state checking.

  _hasChanged() {
    let changed = false;
    for (let inputPath of this.inputPaths) {
      const current = FSTree.fromEntries(walkSync.entries(inputPath));
      const patch = current.calculatePatch(this._previous[inputPath] || []);
      this._previous[inputPath] = current;

      if (patch.length) {
        changed = true;
      }
    }

    return changed;
  }

Here, we’re iterating each of this.inputPaths and using FSTree.fromEntries() and walkSync.entries() together to produce a current state, then calculating a patch which tells us about added/removed/changed files using calculatePatch(), and setting changed if there are any changes, then returning the changed result.

  build() {
    if (!this._hasChanged()) {
      return;
    }

    // do work
  }

Then in the build() method we simply call this._hasChanged() and if nothing has changed, we skip doing any further work. That’s it, our plugin will now skip doing any work if none of the input files have changed.