Hotwire: HTML Over The Wire
I’d like to share my experience of encountering Hotwire from the team at Basecamp which was extracted from the tooling used to build their Hey product. I was so pleased to see DHH tweet about it and took some time over my end-of-year vacation to delve into it.
[Thanks to Joachim, Michael & Stefan from INNOQ, and Tudor and the speakerconf cohort for feedback. Also thanks to Oliver for the interest from the Spring Boot team, and special thanks to Bruno for inspiring the Thymeleaf configuration.]
This is the first in a series of #LowJS articles, forming the basis of a talk for YOW! 2021.
The history
Hotwire is an alternative approach to building modern web applications without using much JavaScript by sending HTML instead of JSON over the wire.
This sounds sort of familiar with a “last century” throwback:
By decoupling the data interchange layer from the presentation layer, Ajax allows web pages and, by extension, web applications, to change content dynamically without the need to reload the entire page.
So the Hotwire approach may sound very odd to some, particularly given where we’ve landed after the convoluted history of delivering Web applications:
We started with no data (other than fully finished HTML content) being sent from server to browser. Then we had JavaScript to make the HTML “dynamic” by instructing the browser to change the page. (We don’t even need to bother discussing Java applets.) We also had no HTML at all, sending XML with XSLT for the browser to transform itself. Then a HTML page with JavaScript to Asynchronously request XML and transform it into DOM changes (with thanks to Jesse James Garrett who used the term “AJAX” to neatly encapsulate those ideas). Of course, XML was “too verbose” and “too unreadable” and “too restrictive” and “too flexible”, so (in a spectacular show of a lack of security awareness) we used JSON instead of XML.
We’ve had an ever-increasing complexity and contortionistic Rube Goldberg machinery (sardonically named “Single Page Apps”) to get us to the point we are at now.
The alternative history
I’m not going to rehash the seminal thinking that may have influenced the creation of Hotwire, like Rich Hickey’s why simple is better than easy, or the many works by Stefan Tilkov on why actually using REST and having a resource-oriented client architecture and embracing the browser is a great idea, including (from six years ago when we were at GOTO Amsterdam) using HTML as a data transfer format.
I do strongly recommend you become familiar with those presentations, because they will provide a welcome and stark contrast to the cacophony of muddle-headed thinking that underpins the overwhelming majority of web-based technologies and approaches before us.
The alternative timeline is that after receiving HTML from the server, the browser asynchronously requested fragments of HTML to dynamically alter the parts of the page to change based on user interactions or events occuring on the server. Logic remained on the server. And most importantly, the Frakensteinian nightmare of JavaScript (or something more well-thought through, like its original conception) instead remained strictly limited to a small cadre of presentation-oriented decisions and browser-side coordination of RESTful state transformations. HTTP/2 and HTTP/3 moved faster to adoption and SPAs were never needed (because multi-channel, low-latency, highly concurrent, asynchronous, on-demand, only-what’s-needed, server-pushable, obviated the need for the “snappier” claim for SPAs).
The future
In an attempt to merge these two timelines and remove the major drawbacks of our reality, Hotwire…
makes for fast first-load pages, keeps template rendering on the server, and allows for a simpler, more productive development experience in any programming language, without sacrificing any of the speed or responsiveness associated with a [what is sadly now] traditional single-page application.
Some samples
The Basecamp team uses Ruby on Rails (the first open source project extracted from their work) on the backend, so I decided to try the above promise of “any programming language” using Hotwire with a JVM backend.
I’m well-versed in Java, Clojure, and JRuby, but I decided to use the Kotlin language to implement my samples, and (despite its modern bloat) Spring Boot to drive the application. I’ve also chosen Spring WebFlux to keep on the Reactive side of things.
I greatly prefer logic-less view templates, but I decided to take a first crack at using Thymeleaf because I’d never used it before. Mustache as a template language and vert.x as a reactive app container would also be good choices.
The samples discussed below are on Github.
Turbo Samples
The samples will use:
- Turbo Drive — to show the “snappier” effect that people think you need an SPA for (while navigating to a new page)
- Turbo Frames — to show the simple use of progressive enhancement of UX and updating parts of a page with fragments of other pages (simulating the results of a ping)
- Turbo Streams — to show Server-Sent Events transferring HTML templates to update parts of a page (periodically showing the current average system load)
To include Turbo into our pages, we add a <script> element to the<head> of our HTML pages.
<script crossorigin="anonymous" src="https://unpkg.com/@hotwired/turbo@7.0.0-beta.3/dist/turbo.es5-umd.js"></script>This uses the unpkg service to present the library and allow us to later reference the Turbo object when needed.
Turbo Drive
Turbo Drive is a simple mechanism giving the user of a Hotwire application the same impression of navigating pages as if the application were an SPA.
To make use of Turbo Drive we do nothing in particular. When the user clicks on a link, they’ll navigate. If the response is taking a while (> 500ms) then a progress bar will show at the top of the page. The progress bar can have some of your own style applied.
If you need to opt-out of Turbo for some elements on a page, you can do so (see below “Disabling Turbo”).
It seems like there’s quite a bit extra in Turbo Drive for native mobile navigation via additional adapters but these samples don’t make use of that yet.
Turbo Frames
For me, this is the killer feature of Turbo. It allows you to use a “transclusion” progressive enhancement to request pages. If Turbo isn’t present, not compatible, or unavailable for some reason, the user can still use the app. Importantly, it allows you to test pages both with and without Turbo’s influence.
Instead of navigating the whole page, we can replace a part of the HTML on the current page with the subsequently requested HTML (of even just a part of it). I say a part for the subsequent page because that can be a whole HTML document, accessible as a distinct resource in its own right, and thus available without the Turbo library loaded in the browser.
Turbo Frames allow predefined parts of a page to be updated on request. Any links and forms inside a frame are captured, and the frame contents automatically updated after receiving a response. Regardless of whether the server provides a full document, or just a fragment containing an updated version of the requested frame, only that particular frame will be extracted from the response to replace the existing content.
In the sample index.html, we have the exact same HTML page being linked to both the “Turbo Frame” section and the “Regular” section (see below “Disabling Turbo”). The difference being one is retrieved using Turbo Frames and the other is retrieved as a normal page by the browser.
To use Turbo Frames, the linking element (in our case an HTML anchor <a>) is wrapped in the custom element <turbo-frame>. When Turbo Frames is available and enabled, this triggers the Turbo library to cancel the regular navigation, retrieve the document itself, parse it, and replace the content of the <turbo-frame> element with the matching fragment in the retrieved document.
You can see the matching <turbo-frame id=”greeting_frame”> tags in both documents.
When the page is retrieved, Turbo Frames removes the HTML around the pertinent fragment when inserting into the “turbo frame” part of the requesting page. (Ignore the “th” references if you’re not using ThymeLeaf.)
Apart from it not being a native HTML mechanism supported by browsers (thus the need for the small amount of JavaScript), this is a more elegant and robust alternative to<iframe> (or <frame> of old).
Disabling Turbo
If we don’t want Turbo for some elements, then we can explicitly direct it. The following happens to disable Turbo Drive and Turbo Frames by using the data-turbo="false" attribute any applicable element. We’ve used the containing div but it could be used on individual a elements.
As mentioned earlier, this retrieves the exact same “Greeting” document but instead uses native browser capabilities, not the progressively enhanced capabilities of Turbo Drive or Turbo Frames.
Turbo Streams
We have the in-place content returned from anchors or forms now working in a way that gives the same results as an SPA, but what about forms needing more sophisticated page updates, or server-pushed data from server-sent events or WebSockets? Hotwire can handle that too!
Unfortunately though, here’s where using the Turbo library gets a little confusing for many and, in my opinion, the idioms employed could do with some evolving as time goes by. Nonetheless, the end results are slick.
Turbo Streams - Forms
In our sample we have a form that asks the server to “ping” a network host and show the time it took for the ping to be relayed, or indicate a timeout.
The non-Turbo form is pretty simple. We have had to again add the data-turbo="false" attribute to prevent the Turbo library handling this form:
Our controller that responds to /pinger resource requests is able to produce regular HTML responses, resulting in a new page appearing in the browser (as expected).
The method pinger is annotated as producing text/html or text/vnd.turbo-stream.html. It returns the ping.turbo-stream view name which ThymeLeaf renders and returns as a response to the HTTP request.
In the case of the regular ping form, the request asks for HTML so the response is HTML. As you can see from the above template content, it is a full HTML document that also contains a turbo-stream element. The contents of that element are ignored by the browser (and, currently, turbo.js) when the Content-Type is text/html.
Now what if we wanted the result of the form to be handled more like a Turbo Frame or SPA and included in the requesting document?
We have a regular HTML form (this time without the “disable Turbo” directive) and then anywhere else on the page we could alter an element with the response. Turbo Streams allows for that element to have the following actions on it with respect to the response: append, prepend, replace, update, and remove.
We’d like to simulate a command-line ping to some extent, so we use the append action. This is driven by the server. When this form is submitted, the Accept request header includestext/vnd.turbo-stream.html and the sample app will return a Content-Type response header will also have that value.
When the requesting document receives this, Turbo recognizes the Turbo Stream media type, extracts the HTML from the template within the <turbo-stream>, and applies the action to the target element. In our case, appending the HTML list item to the HTML ordered list that already exists in the requesting document.
Our Turbo Stream document tells Turbo to append the contents of the template to the target element in the requesting document with an id of pings. It provides the contents of the template as an HTML list item with the ping time in milliseconds (or “timeout”).
The affect is that each time the user clicks the Ping button on the page, the ping time is appended to the list.
The integration of ThymeLeaf with SpringBoot isn’t superb at dealing with custom media types, but if we configure a custom view resolver and use slightly different file extensions for the view files (similar to how Bruno Drugowick did in his samples) we get the magic to happen. In our samples, any file with .turbo-stream.html extension will have the Turbo Stream media type applied.
Also note that the produces annotation attribute is for content negotiation only. It allows us to have distinct functionality depending on what media types the user agent accepts as a response to the request. It does not set the content type of the response, because it can contain multiple media types and because the content type is set by whatever constructs the response entity — normally a View Resolver, but it can also be the action method itself depending on the return type.
We happen to have a ping.turbo-stream.html that can be used both as a full HTML document and as a Turbo Stream template. If that weren’t the case, we would have separate controller actions for the various produces Media Types. In the original sample, this was indeed the case and there was a ping.html template file and separate ping.turbo-stream.html template file. I changed the sample due to thoughtful feedback from Sam Stephenson from the Basecamp team and, as a result, it’s even slicker. Thanks Sam!
Turbo Streams - Server-Sent Events
As mentioned, Turbo Streams can handle WebSockets and Server-Sent Events (SSE), too.
I’ve never been a big fan of WebSockets (certainly never over a WAN) and with HTTP/2 and the SSE approach, there’s few compelling reasons to use WebSockets any more. That is, unless you’re not able to use HTTP/2 fully (I mean, it’s only been a decade and we have HTTP/3 now, so that may still be an issue if there are lame intermediaries like some CDNs and load balancers), or unless you have to handle crappy old browser versions (an entirely artificial problem, because any corporate that claims its staff “cannot” use any of the latest, most secure, browser versions is full of it).
Remember we used the unpkg service to be able to access the Turbo value which we now use to connect to an event stream (this is needed until such time as HTML has a means of doing it without JavaScript).
<script type="text/javascript">
if (window["EventSource"] && window["Turbo"]) {
Turbo.connectStreamSource(new EventSource("/load"));
} else {
console.warn("Turbo Streams over SSE not available");
}
</script>In our sample, we have a stream of average system load (and the UTC time that load was inspected) with items being generated every 3 seconds. We’d like items in that stream to be displayed on our web page without any user interaction required, and for that data to update as each item in the stream arrives.
We have a controller, similar to the ping sample that reads that stream and maps it to a Turbo Stream document as an Event Stream (text/event-stream) response.
Not bothering with a response of plain HTML into a new page this time, we’re simply answering with a Turbo Stream document that instructs the browser to replace an existing HTML element (rather than append in the ping sample).
This controller also is using the Thymeleaf integration with WebFlux / Reactor Core to wrap a stream of values (rather than a single value) in a ReactiveDataDriverContextVariable for the model to use during interpolation into the template. This must be accompanied by an element in the template using the Thymeleaf “each” attribute.
This puts Thymeleaf into “SSE” mode, the template is then returned as an Event Stream and each flush of the data-driven buffer (we’ve used a size of 1 for the buffer, but you could use higher values for “pages” of data) results in a response being sent to the browser. Opening up the Network tab on the Inspect tool and examining the load resource will reveal an EventStream tab and you can view the stream of responses coming from the server.
Conclusion
So fare I’m elated with Hotwire. It delivers on the simplicity, progressive enhancement, and light touch of browser-side logic. It has an engaged community. It has a smart team behind it, and they’re using it to build their own products.
I am yet to explore Stimulus, but when I do, it will be another article. On first look, it looks great too and provides just enough client-side state handling without the bloat in other JavaScript libraries.
I’m also yet to explore the way Turbo works in a mobile app context, and the Strada framework (as yet unreleased).
I feel that there is something to yet explore with the detection of Turbo Stream responses and full HTML documents are used such that the same response entity could be used in a traditional (full page request) or progressively enhaned as a Turbo Stream request, using the full power of HTML 5’s Templates and Slots and a little extra contextual smarts in the Turbo library.
That notwithstanding, I’d like to express a huge, heartfelt thanks to the Basecamp team and contributors from the community who have put together this unexpected dream tool. I look forward to using it “in anger” on real projects and introducing it to my team at Secure Code Warrior.
Notes about the samples
For those that are using Kotlin co-routines integrated with Reactor Core (via WebFlux), the samples provide a fair reference for how controller actions can use low boilerplate, follow a Reactive pattern while using a code style familiar to most people.
The suspend functions are mapped to Spring WebFlux’s use of Reactor for us by a Kotlin library. If we were using Java 8+ (or if we wanted to use Reactor Core directly from Kotlin), we could use a Mono<T>/Flux<T> return type for these functions.
The custom Thymeleaf configuration and TemplateSelectorModifier shows how to use the same fundamental controller action while using the built-in Spring content negotiation features with only minor differences in declarative statements between the action that responds with an HTML view versus the one that responds with a Turbo Stream view.
If you look at the eventual ping() function that does the work, you’ll see a number of nice Kotlin coroutine and other syntactic sugar that record the duration that code takes to execute, work with a Disposable resource, fabricate a delay in a non-blocking context, and for non-blocking code to handle exceptions thrown in blocking code.
