2017’s Generation of Static Site Generators

In 2017 I anticipate we will see a new generation of static site generator (SSG) emerge, with importing and rendering decoupled from one another.

So far the generations of static site generators have been:

1. 1990s — static sites were generated by humans, written by hand, editing HTML directly in tools like frontpage, dreamweaver and notepad
2. 2000s — static sites were generated through desktop applications, where the end-user edited their website via a desktop app, and unbeknown to the user the application would update an internal database, maintain the files, and upload them to the server
3. 2010s — static site were generated through file system compilers, these focused on developer power, with features like file system as the database, and markdown support, early prominent examples include Jekyll for Ruby and DocPad for Node.js

I see a new generation emerging, containing the following interdependent ecosystem parts:

• External Database: content is stored in a modern database system that supports realtime publish/subscribe, such as couchdb and rethinkdb
• Importing: content is imported into the database from any location, such as a computer’s file system, a medium blog, a Wordpress blog
• Rendering: content is exported from the database to any location, such as a static site generator to generate a static website, or to a comment javascript widget
• API Extensions: content is updated in the database using rich API extensions that wrap/extend the database, such as a Zapier or IFTTT extensions that tweet new posts automatically, or SDKs for different languages, or an API extension that allows native comment interfaces to read and add comments to blog posts, or even an extension to provide a Wordpress compatible API hook their desktop interface Calypso up to the database.
• Native Interfaces: content is exposed from the database with interfaces optimised for particular tasks, want a Wordpress/Calypso style interface, want a Medium style interface, want a Tumblr style interface, or a custom interface for your particular client? These interfaces can be built ontop of the API extensions as an alternative to importing content from existing systems.

So what’s the problems with today’s static site generators?

• Todays SSGs use an in-memory database or their own external database system, both of which are designed only for their static site generator, and are not extensible as an open eco-system
• Todays SSGs load, render, and forget on each run, with large static sites taking minutes to generate accurately (change the title of a page that is references in the menu that is used on every page should update every page), realtime changes are not possible with this current system
• Todays SSGs require complex workarounds for anyone but developers to edit then publish the content; web design firm clients may be forced to edit content locally then send to the developer to update, or the developer sets up a workflow on the clients machine, or tells the client to use some special webservice like Prose.io, all of which are inferior compared to what that Wordpress admin interface they are use to — plus when they click save, they must wait minutes to see the final result of each change locally, and even longer to see each change eventually propagate to the live website, all unacceptable to the majority of clients — plus in some systems, clients could break things as they may now edit content or template files that they are not familiar with

The proposed 2017 generation of SSGs will solve this by:

• Importers will be able to allow clients to use the multitude of services they already use, such as Medium, Wordpress, Twitter, Tumblr, Instagram, and for developers the file system, and import that plethora of content into the modern database speedily
• The modern database will readily provide access to the latest direct content and imported content, across the interwebs rather than just a local machine, so that a team can work on the same content, including updating and exporting it
• Exporters will allow for realtime changes to be rendered to all needed destinations, with only the needed changes being rendered and propagated and deployed, rather than the entire static website — reducing minutes to seconds
• API Extensions will allow interactivity with these static sites through pragmatic developer tools or even ideal Native Interfaces for particular clients, allowing dynamic extensible experiences, at little cost

Why hasn’t this occurred yet?

• Today’s SSGs chose in-memory databases due to the complex setup of powerful database systems at the time, however in 2017 powerful database systems are a lot easier to setup, often with one click installers and even as bundled dependencies as part of a applications or SSGs
• Today’s SSGs were intended for the hobbyist programmer, with the value to clients being an afterthought, as such the workflow for clients is incredibly complex, and things like editing and previewing production changes is just too complicated to be worthwhile — now with hindsight we can rearchitect the SSG to meet all needs, with different importers covering file system use cases for developers and external content services for clients
• Today’s SSGs are competitive rather than collaborative because at the time, no one else was doing it, so they could be independent — however now there are over 400 static site generators — and this competition is causing duplicate effort — how many markdown rendering plugins do we really need? — instead importers can be written to import in popular SSG source content to migrate from them, and exporters can be written to export to popular SSG source content to render with them, as interim solutions

Is anyone already doing this?

• Contentful is the main player with their coupled system of Database + API + Interface, and ecosystem of extensions for importers and exporters. However, their current ecosystem is a fraction of what is ultimately possible—I’d like to see SSGs tout Contentful support on their homepages, as well as hundreds of integrations and contributors, not tens. Their solution also requires dependency on connection to the Contentful service, which requires online availability, which means no offline/local development/testing/previews. It also requires fitting within their payment plans, which in turn allow for premium support. Finally, as their solution is a coupled solution, performance and customisation are sacrificed for convenience. None of which are necessarily bad things due to the tradeoffs each thing provides, however still they are limitations worth noting.

What proposals exist for what can be done?

• Existing static site generators can continue to try to meet evergreen needs like realtime content imports and client updates with their existing architectures but ultimately fail as their architectures can only hack such features into them, rather than support them by design
• Contentful can allow local installs of their platform to solve the local development, testing, previews issue, as well as improve their ecosystem and API to offer more power to the developers as a workaround for their proprietary nature.
• The community can organise to develop these ideas further, into an open and collaborative effort between forward-thinking static site generators. A database system will need to be picked, I believe rethinkdb is the best for this, but it should be discussed. Once picked, work can then commence on Importers, Exporters, API Extensions, and Native Interfaces around the chosen database system. SSGs can then update themselves to be exporters for this new system — giving them the benefits of this new united system. An agnostic database system is also an option but its complexity over agility makes it better for a later feature of the system rather than an initial consideration.