I have been working on the dev branch of Extwee for the last three weeks. Progress has been slow, and I have been stealing time here and there after work at night to try to work on different parts. The hope, previous to the emotionally disastrous Summer 2020, was to try to get Extwee 2.0 out by the end of 2020. That did not happen. As I type this, it is nearly the end of February 2021, but I am finally much closer to releasing a new update.

Goals for Extwee 2.0

Extwee 1.X fulfilled the primary goals of a Twee compiler and decompiler: it was able to convert Twee 3 notation into Twine 2 HTML and was able to “decompile” Twine 2 HTML into Twee 3 notation. There were some issues with rare edge cases, but it did its job well-enough.

For 2.0, I wanted to add two new features I found myself wanting in a Twee compiler:

• Command-Line Setup
• “Programmable” Classes

Command-Line Setup: npx extwee

As part of my day job, I teach web development and commonly use tools like create-react-app, eslint, and webpack. Something I enjoy in those tools are their ability to be called via npx. Through adding configuration files and using a command combination like npx eslint, a “runner” (package) can be used without explicitly installing it as part of some project. This is extremely useful.

For create-react-app, the runner will create a package.json file and build a basic React template around the file with some default options. For those starting with React, this is a great way to quickly “jump into” writing code through changing the default values and adding the starting template. For Extwee, I want something similar.

Both Extwee and Tweego come packaged with story formats as part of their releases, but these need to be set up to be used with a Twee-based project. Tweego uses the approach of environmental variables and local folders. Extwee, as of 1.X, takes no approach and leaves the setup of story formats up to the user themselves. Other than giving the author some default story formats, it is completely agnostic to how a project might be configured.

One of the things I have been working on lately for Extwee 2.0 is creating an extwee command-line tool for use with the project. Instead of needing to be installed as a package, it could be “called” much more directly using the command combination of npx extwee. From there, different command-line options could be passed to it.

Here is what I am currently working on for 2.0:

• init: Download story formats into a story-formats folder, create an index.twee file (with a generated IFID), and create an extwee.config.js file with default settings.
• compile: Looks for and combine all Twee (.tw, .twee, and .twee3) files in current directory and its subdirectories (unless overridden by extwee.config.js) and create a Twine 2 HTML file. (Output file will be assumed to be output.html unless overwritten by extwee.config.js.)
• decompile: Looks for HTML files and attempt to “decompile” them into Twee files in the current directory only.

Instead of needing to specify the exact command-line options for each use of extwee, as is currently the case for the 1.X branch, the 2.0 release will use an extwee.config.js file with settings for which story format to use (and where to find them), what directories it should use when compiling, and other scripting options to process the found JavaScript and CSS or to access other tool-chains such as using webpack or as part of something like grunt.

Side Note: JS over JSON for ES6 Modules (at least for now)

I strongly prefer to use JSON for settings and other configuration files. Node itself uses it as part of its package.json file and many other services and tools follow the same pattern. However, I also use ES6 module support (a currently experimental feature in Node.js). This has a “known” problem with JSON: ES6 modules cannot import JSON files without enabling support via a command-line flag. In order for other programs to consume the extwee.config.js file safely across both CommonJS and ES6 Modules project, it has to be JS for right now.

In another year or so, this will probably not be a problem, but many services, like Extwee will as well, use JS files instead of the normal JSON for its configuration options as of their current versions in 2021.

“Programmable” Classes

Almost two years ago (as of this writing), I wrote a quick series of posts on Bots Writing Twine. It referenced how to use Extwee and its ability to compile Twee notation into a Twine 2 HTML file to create code for programmatically creating Twine 2 HTML. While I think I have been the only one using such a process for projects since then, I have found when working on 2.0 many of the core classes in Extwee were too tightly coupled together. When creating a Story, for example, it made some assumptions about how the data would be used. This, in turn, created a situation where a Twee or HTML file was always needed to do anything with the project.

Partial Stories

Extwee 2.0 now uses the metaphor of a partial story. Instead of always assuming everything passed to it was a “complete” story, meaning all passages required by the Twee 3 specification were always present, it now assumes it only contains part of any story and more passages could be added at any time. This has opened the door to programmatically create passages and work with instances of Story without needing to touch any files at all. Passages, including required ones such as StoryData, can be added or removed at any time without breaking anything. As part of working with a Story, it constantly updates its own properties based on what it holds and will always be as accurate as it can be based on its partial data.

Static Methods

Extwee contains multiple classes for parsing and writing files. These include classes like TweeParser, for parsing Twee files, and HTMLWriter, for writing HTML files. In all cases, instead of needing to create object instances of these classes, they have been converted into static methods. There was no reason for these classes to have their own stat,e and now they do not. As a direct result of not needing to create object, this also frees them up for use with destructuring assignment and creates smaller overall projects.

Promisifed Testing

Over the last six months, I have been moving away from a testing combination of Mocha + Chai to Jest. Part of this move comes from my teaching of Jest with React-based projects as part of my day job, and another part comes from wanting a single testing solution rather than multiple packages. Earlier this year, as part of working on Simple QBN, I converted it to Jest and have now successfully moved Extwee over as well.

Struggles with Command-Line Testing

Over the last few days, I have been struggling to find a “good” approach to testing command-line options. While Jest is quite good at testing different parts and working through code coverage, as it comes to command-line options, things become harder. This difficultly arises out of the need for tests to run independently and create a separate “process” for each test.

Jest has good support for testing promises, but Node.js does not have a good approach to creating promises based on its exec() and spawn() functions. However, there is a work-around: the Util class in Node.js has a method called promisify() for creating a Promise-based version of the common function pattern.

I have the start of using this approach of a Promisifed exec() for testing the command-line tool version of extwee starting with its “–version” option. It looks like the following:

CLI.test.js

// Import FileReader for reading package.json.
import FileReader from '../src/FileReader.js';
// Import promisify() from 'util'.
import { promisify } from 'util';
// Overwrite exec() locally as a Promise.
const exec = promisify(require('child_process').exec);

// Pull the version of this project from package.json.
const { version } = JSON.parse(FileReader.read('package.json'));

describe('CLI', () => {
  it('Should show correct version', async () => {
    // Wait for the Promise exec() to finish.
    const { stdout } = await exec('node ./bin/extwee.js --version');
    // Compare the package.json version to the console output in stdout
    expect(stdout).toBe(`${version}\n`);
  });
});

Note: Because ES6 Modules cannot easily import JSON files, I use Extwee’s FileReader class — and static method read() — to read the JSON file for testing against the version of package.json as produced by the “–version” usage.

Last night, I started work on tests for compiling Twee via command-line options, but did not include the test in the above code. The next step, within the next few days or so, is to write the code for the test and then move to HTML decompiling tests and command-line code.

Once the command-line interface has full tests, which I hope to finish by the end of February 2021, the next major step is incorporating the planned extwee.config.js file. The current hope is to have Extwee 2.0 out some time around the end of March 2021.