Hypermedia with Common Lisp and Datastar
This guide walks from the simplest server push to a full multi-client CQRS application. Each chapter introduces one concept, shows it running live inside this site, and provides the full source of the equivalent standalone program.
Any comments, suggestions, general feedback can be sent to the contact in the footer, or as an issue to any of the involved repositories (if applicable).
About Hypermedia
This guide is not meant as a completely stand-alone introduction to either hypermedia or Datastar: the following are good preliminary readings:
- > Hypermedia Systems - «The revolutionary ideas that empowered the Web», the core concepts from the HTMX author, using HTMX.
- > Datastar: the Hypermedia Framework Guide - Guide to the Datastar specific implementation of hypermedia.
When we speak of hypermedia, we are specifically talking about the way Datastar handles it, and how the Common Lisp SDK implements it. The examples use datastar-cl, the Common Lisp SDK for the Datastar hypermedia framework.
Hypermedia and Common Lisp
While building the SDK and all the other ancillary libraries and improvements, we were inspired by what the BKNR project set out to do years ago:
BKNR is a software launch platform for Lisp satellites. You could replace «launch platform» with framework and «satellites» with «applications», but that would be too many buzzwords (...) BKNR also features a web framework, providing an object oriented web handler architecture.
We believe that hypermedia is a good fit for web development in Common Lisp. We further believe that Common Lisp is an ideal development platform. The usage of BKNR datastore for the prevalence example is thus not a coincidence, but a natural consequence of this inspiration: the work done here is, partially at least, a different take at the BKNR web framework: if BKNR was a launch platform with Sputnik being the first release, the aggregate work done here is Lunokhod-1, moving freely and closer to the stars.
The Common Lisp SDK itself can be used with multiple ways. The specific stack used in the guide (and also in the SDK examples) is the following:
The SDK itself, datastar-cl, includes lc-sse, a generic SSE library that support Hunchentoot, Clack (including Woo), and compression for SSE streams.
Several projects were created in support of datastar-cl: lc-sse, yes, but also cl-brotli (a library that implements brotli compression for Common Lisp), and work on woo to add user channels, thus enabling CQRS in an async web server (something not currently possible otherwise).
The examples and demos will show how it works, and the SDK itself makes use of Common Lisp features - CLOS, macros, etc. - to provide an idiomatic approach to developing reactive web applications. The interactive examples are based on the versions included in the datastar-cl SDK; they can be loaded with e.g.sbcl --load <file>.lisp and usually run on port 8989.
Conceptual approach
Consider the following principles:
The Tao of Datastar:
- Most state lives on the backend. The backend is the source of truth.
- The backend drives the frontend by patching HTML elements and signals.
- Use signals only for user-interaction state and for sending values to the server.
- Trust the morph. Send large regions; idiomorph applies only the diff.
- An SSE event stream is a plain HTTP response with a special content type.
- Separate reads from writes: one long-lived GET stream, short-lived POST commands.
One aspect that can be often confusing for those starting with Datastar is that there is (understandably) a difference between what Datastar itself offers, and several important concepts that are very commonly used with it and even assumed to exist, implemented at the back-end. This guide is also an attempt at showing how they work - CQRS, for example - and why they make sense with Datastar. It is not assumed that readers will be familiar with them, and the examples are simple to a fault, but that's by design.
A note on webservers
Most of the examples use Hunchentoot. Some use Clack with Woo. Either can be used, with Hunchentoot being a good default, especially when using any kind of loop of blocking body in the request handlers: Hunchentoot uses threads and doesn't block, and current operating systems can handle thousands of threads. Clack with Hunchentoot works the same way. Clack with Woo, as mentioned above, is different: using the upstream version will cause any blocking in a request handler to block the entire worker thread, defeating the entire purpose of using an async web server. The additional changes present in the supported branch change this, making Woo a good standard when CQRS is used.
Chapters
- > [01] SSE and patch-elements · the simplest server push
- > [02] Signals · reactive variables fed from the server
- > [03] Scripts · console-log, execute-script, redirect
- > [04] Interaction · data-bind, reading signals on the server
- > [05] Remove and append · patch-elements modes
- > [06] Fat morph · send the whole region, trust idiomorph
- > [07] CQRS · long-lived read stream, short POST commands, the opt-in registry
- > [08] Complete · Snooze routing, data-class, tcp-nodelay
- > [09] Client reactivity · data-computed, data-show, data-attr, data-indicator, loading
- > [10] Event sourcing · append-only log, state as projection, time travel
- > [11] Prevalence · persistent CLOS objects, transactions, indexed reads
(defroute hyper-guide (:get :text/html))