Naust JMAP

Editor’s note: As of writing Naust JMAP is still in pre-release and is subject to breaking changes.

Overview

Naust JMAP (repo) is a Go framework for building JMAP servers. It provides the protocol runtime required by RFC 8620 (Core), RFC 8621 (Mail), and related extensions, while leaving storage, authentication, search, delivery, and application-specific data models to the user. The runtime handles protocol correctness, including method dispatch, standard method semantics, state tracking, and push, allowing developers to focus on implementing their own services.

The distinction that matters is runtime rather than server. Most JMAP implementations are complete mail servers, where the protocol sits on top of storage, delivery and authentication decisions that were made for you. Here those decisions stay with whoever embeds it. When I started there was also no server-side JMAP engine in Go at all, only client libraries.

JMAP was written for mail and calendaring, but the core specification never mentions email. It describes typed objects, method calls against them, and a synchronisation model. Mail is the first datatype built on that, not the point of it.

Architecture

Naust JMAP separates protocol implementation from application logic. The core runtime manages JMAP concepts such as sessions, method dispatch, standard method semantics, state strings, change tracking, blobs, and push notifications.

Applications provide implementations through small interfaces: authentication, storage, blob persistence, search, notifications, and custom datatypes. This allows the same runtime to operate over different storage backends and deployment models without coupling the protocol layer to infrastructure choices.

A datatype starts as a declaration rather than a set of handlers. A mail plugin declares a Mailbox with a name, a role and four counters, and marks each property as client-writable, server-owned, or worth indexing. The runtime derives get, set, changes and query from that.

Because it generates those methods for any type it is given, it will also generate ones the specification never defined. Letting a type restrict which of them it wants caught a bug I had already shipped: RFC 8621 defines copy for Email alone, and Mailbox and Thread were both advertising it to clients.

The declaration stops well short of the behaviour. Threading, counter maintenance and search are all written as code in the mail module, because the specification defines none of them for you.

Storage sits behind ordered key-value operations with an atomic batch, which keeps a new backend small. The two that ship are SQLite and Postgres, so this is not a storage engine: the indexes, change tracking and query planner are bookkeeping built once above someone else’s engine, and they serve every datatype. A type that declares no indexes falls back to a scan rather than breaking.

Change tracking is the part that has to be exactly right, so a change log entry is written in the same atomic batch as the mutation that caused it, rather than derived from it afterwards; a log that subscribes to writes loses data the first time the subscriber falls over. Concurrent writers are handled by a single-writer lease per account, claimed with a compare-and-swap rather than a database lock, whose token is also the fence checked at commit. A holder that stalls and comes back has its writes rejected, so consistency comes from the lease and the batch rather than from any backend’s transaction isolation, and never from a timeout being tuned correctly.

The claim that correctness can live in the framework only holds if something checks it. Every backend runs one shared contract suite, and the object store is checked against a naive in-memory model with property-based tests, since the bugs worth finding are the ones only compound cases reach. The WebSocket codec is hand-rolled, so it runs against the Autobahn suite, which is how I found out that validating UTF-8 per message is not the same as validating it incrementally.

Adding JMAP over WebSocket was the test of whether the transport boundary was real. It needed no new core primitive, because request processing was already independent of transport, so a second transport became a separate module calling into the same place HTTP does. What it forced into the shared runtime was an admission path, since two transports competing for one request budget is not a question either of them can answer alone.

Design Decisions

Building this as a library rather than a server is only possible because of what JMAP is. IMAP is a long-lived session, so an implementation owns connection state and effectively has to be the process. JMAP is stateless request and response, so there is nothing to own between calls, an embedder can hold as many instances as they like, and a restart is invisible to clients rather than something they recover from. Synchronisation is a state string per type, which is why the change log rather than the object store is the structure everything here is built around.

The core module depends on nothing but the Go standard library, enforced by a test rather than a convention. This is a supply chain decision. Go resolves dependencies per module, so anything carrying one becomes its own module, otherwise it shows up in the audit output of everyone importing the runtime whether they use that part or not. Charset decoding lives with mail, pgx with the Postgres driver.

The constraint is occasionally productive and not free. RFC 8291 web push looked like it needed a third-party crypto library, until section 3.4 turned out to spell HKDF out as plain HMAC-SHA-256 steps, and the standard library version passes the specification’s own byte-exact test vector. On the other side, a vulnerability in the charset library still forced a Go version floor rearrangement across three modules, and the Postgres driver could not follow, because the pgx version that would have allowed it carries a SQL injection advisory. Quarantine decides which module has the problem, not whether you have one.

Optional parts of the specification live outside the core as separate modules, selected by import. The test for what belongs in the core module is whether the base specification demands it of every conformant server, not whether it happens to touch something already in there.

Most of the design work was deciding what not to build. There is no search index, only a two-method interface with a substring scan behind it, since a scan answers a text filter correctly and everything past that fragments into ranking, tokenisation, stemming and incremental reindexing. Interactive transactions on the backend interface were rejected because they would double every backend’s contract to defend against writers the lease already prevents. Erasing an account is not offered at all, since whoever wrote the storage layer can do it more directly than any generic operation could.

Challenges

Mail. Every mail feature would be easier with a table, an index or a state string that only mail is allowed to have.

1. Threading is server-defined, so there is no correct answer to check against. Messages join on a shared message-id and an equal base subject, with no merge, so a late linking message joins the first matching thread and threads that have split stay split. Merging would mean destroying and reinserting, because thread ids are immutable.
2. The first threading implementation was O(N squared). A 1600-message thread took 140 seconds to ingest, because each arrival loaded every candidate record to compare subjects. It now hashes the message-id and base subject together into a single composite index, so an arrival does one lookup per referenced id.
3. Two capabilities were added for mail: declaring a property whose value is a keyed object or a list, and indexing the individual members of one. A message carries a set of mailboxes and a set of keywords rather than single values. Neither is a mail concept, and the core specification defines its own types needing the first, which is the test each had to pass.

Scaling.

1. Most of what I initially read off the graphs was wrong. My harness took the maximum rather than the sum of per-process memory, so forking servers looked about six times leaner than they were, and it reused a single message for every delivery in a run, which turned a throughput column I had been reading for weeks into a measure of duplicate-message contention. One of the gaps I had been trying to close turned out to be a competitor storing one copy of forty identical messages rather than storing them faster.
2. What survived the corrections was a storage model difference rather than a code problem. Dovecot and Cyrus spool messages to files, so the bytes sit in page cache the kernel can reclaim, whereas a transactional key-value store generally has to materialise a value in the process before it can commit it. That is the price of a blob committing in the same transaction as the objects referencing it. A file-backed store ships too and measures better on both throughput and memory, and what it gives up is exactly that property, which is not something you can add back afterwards. The cost also belongs to the interface rather than to any one driver, since a backend write takes a byte slice; both shipped drivers could stream natively, but changing that changes something every backend implements, and the atomicity has to survive it.
3. Within that constraint there was still a factor of four available, found by changing how blobs are written, with the same binary, the same parser and the same messages. The chunked store only streamed blobs larger than a single piece, so ordinary mail was buffered whole, and it reserved a full piece per concurrent writer regardless of message size. Pieces now start at 256 KiB and double up to the cap. That was only legal because the manifest records a piece count and not a piece size, so nothing downstream can observe where the boundaries fell. It is still not the default, since it costs about three writes per blob against one and most mail is small.

Lessons Learned

The hardest part of implementing a protocol is not parsing requests, but defining the boundary between what the protocol owns and what the application owns.

Conformance and performance are separate claims. A substring scan answers a text query correctly, just slowly, so a real index buys speed and closes no compliance gap. That removed an entire milestone from the roadmap.

The rules only held once they were executable. Two design drifts happened by reasoning forward from a plausible premise past a rule I had already written down, and both were caught by review rather than by anything automatic.

What’s Next

• MDN send/parse (RFC 9007)

• S/MIME verification

• Quotas

• Further RFC 8621-family modules

• Streaming writes on the backend interface, if the atomicity survives it

• More breaking changes pre-1.0 :))

Visit Naust JMAP · Source