Event Sourcing is a design pattern that ensures all updates to a model are done by writing events to some kind of durable storage. Your source of truth is the event store. Anything else is derived state from those events. This post goes over some definitions, related concepts, and my experience using it.
What Event Sourcing Is Not
Event Sourcing won't solve all your clustering problems. The append only nature of an event store can mitigate some classes of issues, but often when processing commands you'll need to inspect system state. If you're in a cluster or dealing with eventual consistency you'll still have to decide how to handle those issues.
Event Sourcing is also not a synonym for event driven architectures. You can have an EDA without Event Sourcing. Fowler has a piece on this here that collects some of the confusion in terms. At a previous employer I worked on a system that was event driven and the leads called it event sourced which led to some confusion and frustration. Events were a byproduct of stateful operations and event delivery was best effort and used more for state transfer.
Event Sourcing is also not CQRS. If you are doing Event Sourcing you will probably also do CQRS since the event store is your write model and your projections are building your read model. However, you can do CQRS without Event Sourcing as well.
Event Sourcing is not a mechanism to synchronize state across multiple systems. Much like how you generally don't want to pipe your persistence models directly to your API, you likely don't want to expose your internal domain events directly to outside systems. Events intended for external systems won't overlap 100% with your internal events. You may want to combine several events into a more useful bundle, enrich them in some way, redact values, or any number of things I didn't think of here. It's not always wrong to share your domain events externally, but you shouldn't do it blithely. A distinction I've seen used a few times is Domain Event vs Integration Event. It's worth considering which type of event you want in a given situation.
If replication, notification, or reliable background processing are more your concern then something like change data capture might be the tool you reach for before Event Sourcing. You can get some of the benefits and it's easier to bolt onto an existing system.
Event Sourcing does not require microservices. You could easily build an event sourced system as a monolith.
What Event Sourcing Is
Event Sourcing is a little like writing your application in the way a database is implemented. You append changes to a log (the event stream) and then update application state once the log has the change. If something goes wrong with application state, you can rebuild it from the log.
In practice this usually means that you will pick an identifier for domain models and append events for that model into a stream with that identifier. Some kind of mechanism runs those events, applies those changes to state, emits more events, and so on. Events might come as the result of a command from a user, external system, or in response to other events written to the stream.
You will find substantial overlap between Event Sourcing patterns and DDD (Domain Driven Design). They share some vocabulary, and rituals like Event Storming will give you better results when implementing Event Sourcing.
A common idiom is to think of processing the event stream into application state as a fold or reducer function. Sometimes I find it helpful to think of this as a
state machine. The common examples here are a bank account or a shopping cart. You don't simply store the items in the cart, you store the ItemAdded and
ItemRemoved events that lead to the state of the shopping cart. It might be useful later to see what kind of items users are giving up on so you can send a coupon.
An Example
At my current employer we have a variety of Task models (Aggregates) and some have a fairly complex lifecycle. A selection of events from one:
Created, Started, TaskCompleted, StepCompleted, UserRemovedFromTask, UserRestoredToTask, DeadlineExtended, StepAdded, StepRemoved, Aborted.
Our system had a representation of this task prior to introducing Event Sourcing but the current state of the system was difficult to explain if things went wrong or someone was simply confused about why they had to complete the task. We also didn't have a good accounting of how a task could get updated mid lifecycle and it was prone to error if the steps of a task were updated or dates changed while work was in progress. It was also difficult to get a view into how long a task sat in a particular state. If a user ignored it for a while we didn't really get a good idea of how long that was happening. We had created/updated timestamps but they gave an imperfect picture. Now when a step to the workflow is added and a user is notified they need to continue the task we are confident we can explain why.
One pass through that lifecycle looks like this. A user completes a step, the task appends StepCompleted and decides it has also
Started, so the aggregate is in progress. The user completes the last step and the task closes itself out with TaskCompleted. Later an
administrator adds a step to the task definition and the resulting StepAdded event moves the aggregate back to in progress. Nothing
updates a row and then writes a log entry about it, the events are the changes.
Note that events are past tense while commands are imperative. The command comes in asking for something to be done and the events record what actually happened as a result. This idiom matches up with DDD.
Aggregates
An Aggregate in Event Sourcing is analogous to an Aggregate in DDD. It's a (possibly) complex object that represents a part of your domain that must be updated as a single unit. In Event Sourcing an Aggregate is often identified by the stream identifier and it will have some kind of mechanism to replay those events, evolve, and emit new events. Aggregates might be snapshotted for long event streams so that only events since the last snapshot need to be applied to get to the current state.
The Decider Pattern
An alternative to the OOP approach to an Aggregate object that handles and emits events is the Decider pattern.
You fold the events from the stream into the current state with an evolve function, which takes a state and one event and produces the new state. You then pass
that state and a command to a decide function which emits 0 or many events. I tend to prefer this model over Aggregates since it has a clearer
separation of concerns over who controls state. I still tend to think of the thing the evolve/decide functions are working over as an Aggregate, thanks to
DDD terminology. I just prefer the FP implementation of the concept over the OOP one in this case.
Deciders and Aggregates are both much easier to test than a traditional system. Your domain logic has been pulled out into pure code that takes a command and emits events. You'll have some pure core that can reasonably be tested in isolation. Side effects don't go away, but the pattern pushes you to separate those side effects from the domain logic.
Projections
Once you have processed commands and appended the resulting events to the stream you usually need a query-friendly view of that data for your application to work with. In CQRS terms this is the read model. I've worked with MartenDB and Rails Event Store. Both provide a way to listen to events and evolve a read model from them, with synchronous and asynchronous projection tools.
I usually work with relational databases that have transaction support, so I do use synchronous projections where they fit. This avoids headaches with eventual consistency as you can read your writes from the projection after the commit. Synchronous projections aren't always the right call, however. If you have many complex projections, or projections reading from multiple streams you'll need to look into asynchronous projections instead.
Asynchronous projections use a background daemon to pick up newly written events and apply them to the projection. If you're using something like Cassandra to store events you're only going to have asynchronous projections. PostgreSQL has LISTEN/NOTIFY which can be used to trigger background work and tighten up latency on asynchronous projections. Marten offers an opt in setting that wakes the daemon as soon as events are appended, while still falling back to polling as a safety net. When creating a projection you should assess your latency and consistency tolerance. Adopting asynchronous projections early can save you scaling troubles later.
If you're doing Event Sourcing over a relational database you'll likely have a combination of the two.
Since my work project has many small event sourced aggregates with short lifecycles and simple read models, synchronous projections have been fine. I've also been working on a side project for a trivia metagame with fewer aggregates and longer lifecycles. These have more read models projected out of the event streams and I quickly reached for asynchronous projections there.
Choosing Aggregates
Choosing Aggregates in Event Sourcing is the same problem as choosing Aggregates in DDD. You're looking for the consistency boundary: the thing that has to be updated as a unit to hold its invariants. Event Sourcing adds additional feedback when you get it wrong. You'll be making tricky updates to multiple streams, a single stream is growing to an unwieldy length, or the provided concurrency protections are no longer helpful.
Some common advice when doing Event Sourcing is to try to keep event streams short. Rebuilding an aggregate or running a fold over hundreds of events is pretty fast. If it's thousands, you might be in some trouble. This is commonly fixed with Snapshots, and many frameworks provide them, but if you can avoid running into the problem in the first place then all the better.
The pitfall here is choosing too large of an aggregate when several smaller ones might do instead. In the trivia application I'm working on, Parties are event sourced.
Adding and removing party members, collecting achievements, etc. are all events on their stream. Parties can also declare rivalries amongst each other. Initially I modeled
rivalry as events on the streams of the two participating parties but found it awkward to send commands to both and it broke the built in stream level concurrency protections.
Rivalries can also be noisy with events as parties declare them, play games with the rivalry, end them, and revive them. Rivalry was lurking here as its own aggregate. A
deterministic stream ID is chosen from the two party IDs and a Rivalry aggregate is represented by that new stream. A party proposing a rivalry to another is written to that stream
instead of writing a RivalryProposed event to both parties. The rivalry can live out its own lifecycle without having to add a bunch of events to the Party streams. An Event
Storming session would have surfaced this at design time.
Another example is from this talk where they initially modeled money transfers as part of an Account aggregate but later realized it had enough of a lifecycle of its own to become a Transfer aggregate. This eventually simplified their design.
Concurrency
Both Marten and Rails Event Store provide optimistic concurrency out of the box on event streams. If you have two writers targeting the same stream one will get a concurrency exception prompting you to retry the command on the updated state. I find this to be a helpful nudge in choosing aggregates as well because if I'm trying to figure out how to manage concurrency between models and can't rely on a single stream's optimistic concurrency guards it is a sign that I might have another aggregate lurking in the design.
What to Event Source?
Another pitfall when adopting Event Sourcing is trying to model everything in the system this way. It's likely some entities in your system exist simply as configuration data to be referenced by other entities. You should consider which parts of your system really need to be represented by Event Sourcing and focus on those. There is overhead in choosing the correct events with the correct payloads and in projecting them to a read model. In some cases a traditional document or table with current state is just fine.
Event Versioning and Formats
Since events are only appended to the store, you'll inevitably find that you have modeled some of the events incorrectly and want to correct that. You shouldn't go mutate all of the events in place. Usually you'll handle this by introducing a new version of the event and keeping support for the old version. You can also provide an "up-casting" function to map an old event to the new format if you really need to adapt how the code handles the old events. You should be cautious with up-casting functions since they're going to be run every time an old event is loaded. It could become a performance issue.
Data formats like Avro and Protobuf provide explicit support for forwards and backwards compatibility for events. If you're building your own tools for Event
Sourcing that alone might be a reason to adopt a binary format like that. Versioning support aside, a binary format is going to be much faster and more
compact on disk. This is less obvious in the face of PostgreSQL's jsonb format. It won't provide you the compatibility guarantees but it is efficient
and offers ad hoc queries over the data.
Deletion / Undo
Much like how you cannot delete data from a log, you cannot simply remove events. If you delete a projection row then any rebuild is just going to bring it back.
You will inevitably need to undo or hide some data. Like an accounting ledger, you'll want to use compensating transactions. If a Party was
awarded an Achievement by accident, don't delete the event. Append an AchievementRemoved event that hides it from display. If someone comes asking later
where that achievement went, you'll have the data to explain what happened. As a benefit, you can now report on how often such a mistake happens.
You may also find that event data becomes irrelevant after a certain amount of time. In the easy case, the entire aggregate is irrelevant and you can archive a stream to cheaper storage and remove any related data from read models in response to an archival event. Some tools might provide a mechanism for stream compaction as well, which is like a Snapshot but removes all event history prior to the compaction and replaces it with the state at that moment. Use such a tool with extreme caution, you're deleting history.
GDPR and other right-to-be-forgotten rules provide a wrinkle to this. If you have PII in your system you should strive to keep it off the event stream. Reference user data by ID and try not to store it directly on the stream. Marten and Rails Event Store both provide a mechanism to help you comply, but you'll have to handle the fallout in backups and read models as well. These requirements are never simple to satisfy and I think Event Sourcing makes it slightly harder to comply with.
Introducing Event Sourcing to an Existing System
We have primarily added Event Sourcing to net new models. As new features are requested we evaluate whether their lifecycle would warrant the pattern and then make it so. This is of course the easy path.
For an existing model we treated the existing Rails model as the read model. We redirected writes as commands to a new Aggregate and then projected onto the existing model. The main issue here is that we'll never have the ability to rebuild the projection by replaying events. Many records have no event history. We already have some hairy legacy data to deal with so this didn't add much complexity overall, but it's not a mechanism I would use lightly.
My Experience So Far
I've become a fan of Event Sourcing in applications at this point. When I'm troubleshooting an issue, being able to look at the event history for a model is more informative than hunting for logs. On the trivia application I've been asked for several new views of the data that I was able to construct by simply adding a new projection over the existing event streams. At work, being able to explain and audit system state is invaluable. At a prior employer we relied on an Activity log concept for key models but it was always incomplete. New features were added and the log was forgotten about. That's not a mistake you can make with Event Sourcing.
The testing benefits are real. It's not magic, but the nudge to have a pure command -> decide -> evolve pipeline to model the core domain at least gives me some area of the codebase that feels useful under unit tests. You'll still want E2E tests to reason about side effects but it's an improvement over the default.
A CQRS style system built over Event Sourcing has excellent scalability characteristics as you can naturally produce optimal read models out of the event history and change those models cheaply when needed. Stream IDs provide natural partition keys either for table partitioning or database sharding. As is often the case, you should check application code for bad behavior before reaching for complex scaling tools. Before you shard your database make sure you aren't burying yourself under load amplification or N+1 queries.
Another interesting benefit I've encountered with Event Sourcing is that event streams provide a natural integration point for embedded scripting. In the trivia application the hosts had ideas for around 100 achievements out of the gate. I had no interest in coding all of those manually so I added Jint to the system. Hosts describe the achievement they want and can copy a prompt to their AI tool of choice. LLMs are good at JS so the results have been solid so far. The event streams provide a trigger for the scripts as well as a mechanism for testing. You can replay past events against the script to see what would have happened before marking it as active.
At work we're contending with conflicting definitions of scoring mechanisms and are likely going to use embedded scripting against the events we store there as well to generate scoring output based on customer preferences without having to modify core application code for each variation. It should integrate in a similar way.
The traditional model of application design where you read state, make a decision, and write the new state discards useful context about how you got to the current state. After having used Event Sourcing in a couple of projects I would find it difficult to go back to that mechanism. Being able to inspect decision history for troubleshooting and support is too valuable. The audit history you can produce is also valuable for compliance purposes.
Future Work
In both projects I've used Event Sourcing I have been doing so over PostgreSQL. The Aggregate/Decider concept maps nicely onto the Actor model of programming, and I would like to try building an event sourced application over either Akka or Orleans in the .NET ecosystem. Orleans provides Event Sourcing tools directly but has poor support for relational databases. It has an ADO.NET adapter but it saves opaque blobs into a database rather than giving you a touch point to save into a relational structure. Since Event Sourcing already encourages CQRS, I should be able to project out of a document store into a relational database for the read model. I'd have to give up most strong consistency guarantees though. I've also previously only streamed data from an OLTP database like PG or Oracle into a columnar OLAP database like BigQuery or Vertica. It would be interesting to see how replication based on events differs from simply attaching a replication slot to a PG database and doing a data transformation with dbt.
See Also
Designing Data-Intensive Applications touches on this topic a little when talking about working with logs. Greg Young also talks about Event Sourcing a lot.