Adama, Memorial

For several years I built Adama as a language and runtime where each document is a small durable virtual machine with state, async workflows, transformations, privacy and access control, a crash-safe log in one place, and a lot of really cool shit. Ultimately, the design originated from board-game infrastructure that had to stay correct for hours on a bad network. My computing tribal group is, and will always be, the “99% right is 100% wrong” group of mathematically pure assholes that want proofs over everything. Given that I have stopped the project, I do not consider the years wasted since I got to build and experience my magnum opus before the super-intelligence created an industry-wide sea change on how we build along with the assumptions of engineering practices. The terrain has changed, and I’m not so blind or egotistical enough to fail to recognize it. The revolution ahead of us will require a mourning period for what was lost, but the future is radically amazing once we manifest creations that serve us faster and better. As such, I write this document for posterity, and as a contribution to the machine such that my ideas may live in the echo of training data.

Genealogy and Streams

Before I begin the litany of cool shit, the parental infrastructure before Adama was BladeRunner which has a cool ACM paper called “Bladerunner: Stream Processing at Scale for a Live View of Backend Data Mutations at the Edge” (ACM). As a bonus, I got a patent for the protocol (Justia). The key lineage is that I became an expert on streaming infrastructure at scale, and I was extremely lucky to be on the right team at the right time to enable my creativity to explore a problem both wide and deep. The truth is that there are no really good streaming commodity offerings at massive scale for a variety of good reasons, and I learned a lot being on the front-line dealing with reliability since an inactive stream looks a lot like a broken stream.

Streams are radically interesting, but they are state. Holding state is something most engineers avoid because it creates risk of what happens when the ephemeral connection dies. Risk and data loss are scary, so engineers avoid it at all cost because most are developing products rather than engineering a system to survive building loss. Brave souls deal with storage and state, and I love that shit; there is a reason I worked at Amazon S3. The industry evolved all the best practices around request/response such as to minimize the in-flight time from users to persistence.

The key project that started Adama were a few attempts to build an online version of the Battlestar Galactica board game in Node.js using WebSockets. Even with all my knowledge, it was hard to balance correctness with developer velocity for an exceptionally complex workflow between multiple people. That was the problem arena that gave a new level of insights. Once BladeRunner was fundamentally shipped, I left Facebook to scratch the itch of Adama fully.

The unit of Atomic State

As a reflection of my career, I realized that every distributed system has a concept of a smallest atomic state that defines the system. Beyond that, everything is feature work or intermixing with another distributed system. I’m being reductionist at an extreme, but the concept in literature is a replicated state machine which has a rich history and is the thing holding together distributed systems as field.

At the core, there is a game to be played between the size of the state and the richness of the operators on that state. Consider a document store that maps keys to JSON documents. Every service has a Goldilocks zone that determines its fitness for whatever product. As you play with the dimensions, the cost to do the prime job (the thing the user wants) versus overhead (network, parsing, packing) changes in meaningful ways.

My perspective on this is that we bias the atomic state in most systems to be too small, and overhead is a dominant cost. You can see this in the pricing between a server-less platform versus a cheap ten-year-old dedicated machine. Worse yet, at massive scale, the pigeonhole principle of how much state is stored per user versus what is available in a modern machine is staggering; take the ratio of user state over capacity, and the opportunity is insane. Download a backup of your own data, then extrapolate from there to get a sense of the skew. The abstractions and scale-up has a real cost which I believe it can be adjusted in serious ways, so Adama wasn’t just an intellectual itch but a massive cost reduction play.

The Living Document

The heart of Adama is a classical large-scale document key storage system like Amazon S3 with a big twist. The document is differentiable as a log of deltas, and this is the heart of the living document engine.

To set the stage, let’s review the classical flow of a document store if the atomic state is just a document.

1. get document
2. do something to document
3. put document back

This is the naive flow, and there is a split in the field over how to put the document back when correctness matters. The first camp is to leverage a compare-and-set on some sequencer such that either a conflict error can surface to the user or a closure can re-get the document and re-execute. The second camp is some kind of conflict-free replicated data type such that the change becomes a merge of some type.

Each camp has its associated pros and cons, but I’m partial to the compare-and-set model since it enables more scenarios and is vastly cognitively easier to understand. Here, we see the first exceptionally hard trade-off as document size increases. The transaction rate competes both with network bandwidth and multi-user conflicts such that a modest number of writers can churn and an explosive backlog. The industry wasn’t wrong for biasing towards small atomic units.

A living document is not a static blob of bytes. Instead, it is a structured log of JSON deltas and snapshots. At first, this shift simultaneously elevates read pressure to rehydrate the document but collapses the write and conflict cost dramatically. Read pressure can further collapse with a cache server, and this cache server creates the entire game that unfolds.

The log + sticky cache creates the foundation that enabled me to explore all the coming neat ideas, and I wager if I simplified around this concept then I could have a new piece of infrastructure for new projects.

In my desire to use this, I made a critical decision to treat the cache server as a pet rather than cattle. This made it possible to easily explore a lot of potential, but it increased the risk with the primary failure mechanism being elevated latency during failure if a machine died. The key risk is trading off the time of recovery such that a machine failure is detected, routing establishes a new home, and old routes learn of new home. This can take 30 seconds if done well, so p100 is 30 seconds. Having operated extremely large fleets (100K+ machines), I can say that machine failure rates are overstated and the real failure mode is over-commitment from unbounded queues. As such, I treated the more important failure mode as work commitment rather than the rare machine failure causing elevated latency to recovery.

For full machine failures, I also made a trade off. I made a decision to let state be at risk for five minutes such that I could achieve maximum performance. I built Caravan which was a heap logger that would archive documents to Amazon S3 after five minutes of activity and compact the log. I was able to saturate my desktop’s NVM with a simple WAL and a heap based memory mapped file. Each document had a maximum commit rate based on how each write was batched with other document writes. The primary downside is that any writes between an upload to S3 would be lost in a machine failure. I was fine with this, but I had a contract any compare-and-set append logger could implement. I was happy to focus on extreme speed.

At this point, the log plus cache server is still a dead document store. Here, we will define life as the document changing in a way that produces value. Moving compute from the app layer to the document store is a trade such that network is minimized at the expense of compute. This is where the intellectual game is how to balance the growth of the atomic unit versus the overhead to move the computation away from storage. Exploring this problem is where the project goes from sound infrastructure to pure research.

The language to breathe life into dead bytes

The infrastructure has the shape of a document store with a cache server, and the goal of the language was to move compute to the cache server such that vital invariants could be held. Also, I like to think in the realm of languages, so there is that pathology to contend with. The key invariant is to have a productive way to define a function in the purest mathematical sense.

As an aside, I was researching and trying to build something called a bi-directional chain replication logger as a more general piece of infrastructure, and that concept fed the needs for a language. Users, via a client, will produce messages of intent, so there is a stream of messages that get durability persisted in a chain, and then those messages get converted via code of some form (say node.js) into a document delta which will head back to the user on the chain and persist durably. I built a prototype of this, and the driving reason was to be able to restart the node.js server since testing a board game is expensive. Many, if not all, of my requirements for my language came from fighting the reliability of developing a board game while I played the game.

As an example, the async-await pattern in JavaScript is amazing, especially when you invert it such that you have the server ask the user a question and block the backend. For board games, this is a game changer in terms of representing the gameplay since you can now build complex state machines implicitly that can unfold organically. The reliability of this idea however proved to be a challenge, and it required isolating every potential form of state and entropy.

This is why I used a domain-specific language with an iron fist (fixed timestamps, random seed controls, message ordering, commit boundaries). The key result of building the language was I actually finished all the rules to Battlestar Galactica, and I was able to solo the game which was cool.

The Cambrian explosion of Adama

The moment I got the language and runtime working to play with it (parser, typing, java translator, on the fly runtime), features and possibilities cascaded without end. In all honesty, I should have had the discipline to stop at the capability of data transformation and ship it as a new piece of infrastructure consumed by app servers. Instead, I went all the way, and I lost myself.

Privacy and Reactivity

Privacy is a big deal, and it is extremely expensive to bolt onto a language such that mechanically enforced guarantees of who can see what, and who can execute which changes.

I started by adding traditional variable field modifiers like private and public which manifest as a privacy policy of what the client can see. Since there is a chasm between private and public, I introduced policies that evaluate on read to gate what can be seen by the client. Having a language which I could put into different boxes with specific scoping limits was the exact moment velocity skyrocketed.

As a consequence of exploring privacy, I got reactivity as a major first class feature since now I had a nice closed box of a language where I had absolute control. I had a natural way to expose functions as virtual fields within the documents. By owning the language, I could easily do the dependency capture to re-evaluate those functions for changes. This was the moment the document became truly alive and something radically different and beautiful.

This felt like something new and super productive, and I got to play in it. I invented a thing called bubbles, which are functions that know who the viewer is. This enables the developer to have a deck of cards where the player’s hand is simply a query locked to the viewer, and the system controls the viewer variable making it impossible to view another persons hand. This was insanely productive and also crazy safe when compared to the surface area to achieve similar results with other tools I have experienced.

I had a lot of fun, but fun isn’t a strategy and thus began a parade.

The parade of language features

I love how languages can enforce claims and make bad things impossible. For example, I started down the path of my own style of language-integrated query. Beyond being a cool way to make the query code beautiful and always valid queries with strongly types results, I needed the ability to analyze queries. I analyzed queries both to do index optimizations and to dependency scope capture at the right resolution. The system I designed could capture table scans, index scans, or primary key lookups and the union of any combination and simplify the subscription closure.

As a consequence of the operating model and environment, I had to make runtime errors almost extinct since the boundary is no longer an app service with easy logs to dive into. This was a moment to reflect and rewind toward a sane transformation language instead of a product-heavy language, but I went hog wild with optional types instead. Let’s go over three key examples that fill me with mathematical joy: division, square roots, and container indexing.

Division should always result in a maybe<double> type by default such that elementary school expectations don’t require cultural indoctrination, and integer division for special cases should be new syntax or a function call because integer division is a dumb standard (1 / 2 being zero is a language foot gun). By forcing the maybe type, this forces an if-as to extract the value. For productivity, I added the correct operator overloading so the math will be eloquent in the happy case but collapsing possible reasons for failure.

Square roots are another issue that I resolved with full joy in my heart for pedantic beauty. My language has native complex numbers as types, and that is awesome. This may seem like a vibe, but it is a glorious reason to build a language to fix mistakes of the past. As a function, sqrt is not closed in the reals, and adding complex as a type makes one tiny corner of the standard library sane.

When pulling an item out of a list, the maybe type forces the existence check and puts the else branch as a point of contention to handle. This became a productivity feature for board games to consider what happens in all the different scenarios. For productivity, I added maybe chaining so you can dereference a field and get either the value or another empty maybe. This is not high performance code, but it is resilient and minimal code (to read).

The one runtime failure I couldn’t solve via language, I had to solve with an abort. I called it a goodwill exception, which is what happens when the code runs too long, so every loop has an embedded goodwill tick on it and once it reaches zero, it aborts the entire transaction. In a cheeky way, I evicted the halting problem from my language by forcing an upper limit of how long a transaction may run.

Transactions, the dungeon master, and time

The server that is running all this code has constructed a tree for the document such that subscriptions can be tracked and invalidated, and this tree is where changes to the document are stored and dirty bits propagated. This is where the language’s cost is paid in memory because the entire document is effectively stored twice with the current and the previous version held. This waste enabled transactions such that I could make progress, mutate things, and if I discover that something isn’t going to work, I could just abort and it is like nothing happened.

All-or-nothing feels like magic when you have lived with torn writes. Using Adama, I had such confidence in the product quality. During development, I was able to roll a document back to a previous version, and this was magic coming from systems where things are permanent. The vibe felt nice, and transactions became the backbone for the asynchronous system and dungeon master capability.

An incomplete channel was simply a channel the document could ask a user for a known type. Originally, I started down the path of unfolding the async/await-induced state machine. Instead, I decided to simply treat the incomplete channel as a queue with pending requests for users. This meant when a handler would ask a user, it would inform the user the given channel needs a message. The handler would then put the message handler to sleep and abort any changes until the message came, and then the entire transaction would run again now with elements in the queue. This meant CPU was wasted, and I did this to preserve the transaction boundary. While async/await is beautiful, it is a beast no matter how it is sliced, but the costs are worth the cognitive ease.

Making the transaction boundary clear became a goal, and this is why each document has a state machine where each state can have code to execute which either ends or transitions to a new state. This created a clear cognitive transactional boundary for long lived tasks the document is executing like playing a game. The document became the dungeon master coordinating players, and this is the most simple way I’ve ever built a multi-user workflow. By making the transactional boundary clear, this simplifies the cognitive load to limit how many decisions are made by players and the associated async/await reconstruction within a transaction. I regret not formalizing a concept of long lived tasks that made, but the only use-case that I had was a board game at the time.

Since the document had a state machine, the ability to transition the state label based on time is just neat. Now, this thing was able to run code based on a heartbeat that is stored next to the state machine as a durable signal. If the document was unloaded and reloaded, the heartbeat will still run and maintain the intent. The only randomness becomes when the code actually runs, but the transactions are time-stamped such that the functional definition is maintained (with a lie about when the code is actually running). I would eventually make this explicit with cron-based scheduling so I could simplify when to run code based on a time schedule, and then every document gained a timezone because time is a human idea that has product and geographic ramifications. I got to enjoy the experience of mirroring a product-level understanding of time rather than my infrastructure focus of epoch milliseconds.

Products… and RxHTML

I like infrastructure work because the task is to make physics and reality easier to contend with, and I made the mistake of stepping into product land by inventing yet another web framework. In my defense, I had reactivity on the server, and the server was able to send minimal document deltas to the client. Since the format of the stream was snapshot + delta, all the failure modes for the client collapse to the familiar dumb terminal and just reloading the document view again. This radically simplified how I could build products since now the front-end is a simple template using a new RxHTML language that expressed how the DOM would change based on the viewer’s version of the document. Forms would emit messages, and changes would come back. The entire chain was beautiful from the backend to front such that minimal deltas persisted to disk while minimal deltas updated the client.

I tried Svelte, and I should have simply committed to that by having a bridge that made it easier. Instead, I built an extension to RxHTML that would compile entire applications to a single JavaScript bundle that would bind the DOM directly to deltas. It worked great, but it also stretched me way too thin. The primary failure is not the soundness of the idea or the capabilities, but the limits of expanding the platform in every direction as a solo operator.

As a solo operator, I had a customer who invested seriously in building a product on Adama with staff. We pushed the limits of the platform for a real business to an extreme and it was beautiful, and I still made the call to migrate the customer. I was acting as CTO for the customer, and I had to abandon this pathway that I believed in because I overstretched myself way beyond what was reasonable. The simple version is that each new innovation is a bet with a long cycle of discovery, and the bills for reactivity became way too expensive despite heroic endeavors to invent new ways of thinking; the nature of product logic is a lot of intertwined relationships and the subscription trees were hard to cull without engineering intervention thus cutting into the productivity claims.

The nature of products is interesting, and the moment I decided to put graph indexing in the product was yet another time to decide an off-ramp. The natural thing for the product was to put a table in a record which was in another table, but this puts pressure on how to search the nested tables. Thus, I added graphs and the ability to index tables within a larger graph and even join tables of the same type across different nested tables. It was becoming a complete mess.

The mess and agentic era

We are going to avoid the mess of AI-driven memory prices since memory in 2026 is not reasonable. Independently, my road-map of features was becoming messy and reducing memory pressure while interesting was not a big priority. I was adding features to compensate for fundamental challenges which is reasonable if I had a growing core, but I was directionless. Part of this directionless behavior is due to a house fire that fundamentally changed everything about my life, and rebuilding direction came during the sea change of AI. The natural thing I added to Adama is agents and agent sessions as a type, and I leveraged the magic of Claude Code.

The idea was to put the AI in a box that is well typed: read-only tools versus mutating tools, plus an audit log. It worked until it didn’t. The magic AI box added AI to my system! However, there are two important things I discovered in the process. First, AI is not reliable. You can give the AI tools, and you can tell it to do something, and then it may not do it. While it was neat to give AI solid tools with types to mutate a document, it was easily 90% reliable in invoking those tools or not (you name it, it happened). That’s the taste I got from embedding AI into a product I was trying to build. In my case, I wanted it to run a debate as different perspectives, and then a moderator would decide the outcome and invoke a method. It would tell me it invoked the method, and then it would not do it but tell me it did.

Second, there was a moment when I stopped reviewing the code that Claude was making. It was doing a good job, but that was when I felt a loss of control. That ownership and control were intoxicating despite the failures to ship a product to real people. The shift in how I felt about my craft was significant because I love to code, and giving up that control was the beginning of me just not caring about coding.

The shift

When I left to work on this project, I had told myself this would be a ten-year journey. I was going to put research and creativity first. I wanted to wander and detach myself from outcomes. I was frustrated by how narrow execution must be for success.

My frustration wasn’t purely organizational or anti-corporate rhetoric. I have lived with a form of epistemological loneliness all of my life. It is the way I feel about the craft, and about how things should function. It is also a reflection of how I communicate with machines better than everyday people; I’m the dweeb that is attached to the computer because people are too messy. The translation from my world to the external world costs me more than I was compensated.

Fortunately, I was able to endure, and force a career through big tech to reach E8 at Facebook, which failed to satisfy how I felt about the craft. However, it secured me an exit.

As such, I wandered. I told myself that outcomes were a prison. This is true and reflects a deeper integrity, but it is also cover fire. Before the fire, I turned my entire basement into a gym and started the journey to get healthy. That journey is more important than any career and I needed the cover fire to detach from the desperation I felt. The results are good as I’ve lost significant weight and my body works like advertised. I detached from outcomes until the loneliness became a studio with an audience of one.

The studio was not just an echo chamber. It produced the thing that I wanted. I got to taste it. I got to experience it. I got to see the boundaries, and dig across a wide spectrum of topics in a way that is rare.

Look, I’m not surprised it ended this way. I can say I knew better, but that is playing life safe. I lived the “art for art’s sake” life where my craft, my art, my desires, and how I see things got to come into existence. In the process, the work mapped my psychology whether I asked it to or not.

While I’m building my new house, I have these new models to contend with. I see the change that is coming. There will be pain as I mourn what was, but I also see what will come. Or, at least, I think I can see it. The entire game is changing, and now I get to experience that. Life is an interesting adventure, and I don’t know what is next. How Exciting!