A Quick Introduction to Event Sourcing

Preamble: This post is a brief introduction to Event Sourcing, a pattern that can significantly enhance the way we manage state in applications. I'll link to more detailed resources throughout the post.

Introduction

Event Sourcing is a powerful architectural pattern that stores the state of an application as a sequence of events, think of it much like a log stream - an immutable stream of events. Instead of persisting just the current state like a more traditional system, Event Sourcing captures every change that occurs, allowing for a complete history of the application's state. This has many advantages and disadvantages, which we will explore.

Event Sourcing is often used in conjunction with other patterns like CQRS (Command Query Responsibility Segregation) and Domain-Driven Design (DDD), but it can also be applied independently. It is particularly useful in systems where auditability, traceability, and the ability to reconstruct state are important, specifically in complex domains where business logic is rich and evolving which forces data to adopt new access patterns regularly.

Key Principles of Event Sourcing

This list is not exhaustive, but it covers the most important principles of Event Sourcing and what I'll be covering in this post.

Immutability: Once an event is stored, it cannot be changed or deleted. This ensures a complete audit trail of all changes. This is crucial for maintaining the integrity of the system and allows for historical analysis of how the state has evolved over time.
Event-Driven: Event-Driven is not the same as being Event-Sourced - It's an important distinction, for the system to be Event-Sourced it must be Event-Driven. An Event-Driven system reacts to events, events being the primary means of communication between components. This allows for a more flexible and scalable architecture, where components can evolve independently and react to changes in real-time
Reconstruction of State: The current state of the application can be reconstructed by replaying the events from the beginning. This allows for easy debugging and understanding of how the system reached its current state. This is particularly useful in scenarios where the state needs to be restored after a failure or when migrating to a new version of the application - it becomes native functionality of the system, not a disaster recovery plan.
Decoupling: Event Sourcing promotes decoupling of components, as different parts of the system can react to events independently. This leads to a more modular architecture, where components can evolve without affecting the entire system. It also allows for easier integration with other systems, as events can be published and consumed by different services. Unlike an API microservice architecture, where services are tightly coupled through APIs, Event Sourcing/Driven allows for a more flexible and scalable architecture at the cost of increased complexity.
CQRS (Command Query Responsibility Segregation): Further extending the decoupling; Often used in conjunction with Event Sourcing, CQRS separates the command side (which handles writes) from the query side (which handles reads). This allows for optimized performance and scalability. Instead of being fully command driven events will contain the full body of the entity being changed, allowing for a more flexible and scalable architecture. This separation of concerns allows for better performance and scalability, as the read and write sides can be optimized independently. A great write-up by Luke Popplewell on how to create a CQRS system using Event Sourcing can be found here.
Domain-Driven Design (DDD): Event Sourcing aligns well with DDD principles, as it allows for rich domain models that can evolve over time without losing historical context.
Aggregates: In Event Sourcing, aggregates are the primary units of consistency and encapsulate the business logic. They are responsible for handling commands and generating events. Aggregates ensure that the system remains consistent and that business rules are enforced, allowing for a more robust and maintainable architecture.
Correlation: Events can be correlated to track the flow of related events across different aggregates or components. This is useful for understanding the relationships between different parts of the system and for debugging purposes. Correlation IDs can be used to trace the flow of events through the system, allowing for better visibility and understanding of how the system behaves over time.

Causation: Events can be used to establish causality, allowing the system to understand the cause-and-effect relationships between different events. This is particularly useful for debugging and understanding how the system behaves over time. Causation allows for better visibility into the system's behavior, as it provides a clear understanding of how different events are related and how they impact the overall state of the application.
Eventual Consistency: Systems using Event Sourcing often embrace eventual consistency, where the system may not be immediately consistent but will converge to a consistent state over time. This increased complexity in the system allows for better performance and scalability, as the system can handle high volumes of events without being blocked by synchronous operations. From a frontend perspective this mean that the UI may not reflect the latest state immediately, but will eventually catch up as events are processed - instead it can adopt practices like optimistic updates, where the UI assumes the operation will succeed and updates immediately, while handling any errors gracefully. A fantastically illustrated article if you want to read more on eventual consistency can be found here.
Event Versioning: As the system evolves, events may change. Event versioning allows for backward compatibility and smooth transitions between different versions of events. It's important to note that this is not the same as schema evolution, which is a separate topic. Event versioning allows for different versions of events to coexist, enabling consumers to handle events in a way that is compatible with their version of the event schema. Ideally events should be atomic and evolve additively, meaning new fields can be added without breaking existing consumers. This allows for a more flexible and scalable architecture, as new features can be added without affecting existing functionality.
Event Schema Evolution: The ability to evolve the structure of events over time without breaking existing consumers. This is crucial for maintaining compatibility as the system grows.
Event Store: A specialized database or storage mechanism designed to efficiently store and retrieve events. It often supports features like event streaming, querying, and indexing.
Event Replay: The ability to replay events to restore the state of the system or to test new features without affecting production data.
Event Lake: A centralized repository for storing and managing events, allowing for easy access and analysis of historical data. This is particularly useful for analytics and reporting purposes, as it allows for a complete history of the application's state to be analyzed and visualized.
Projections: Derived views of the event data, often used to optimize read operations. Projections can be created to represent different aspects of the data, allowing for efficient querying and reporting.
Workers: Background processes that handle event processing, ensuring that the system can scale and handle high volumes of events without blocking the main application flow. Workers can be used to process events asynchronously, allowing for better performance and scalability.

Simple Event Sourcing Flow

An abstract simple example of the Event Sourcing flow, this is not a complete implementation but rather a high-level overview of how the components interact.

%% is-centered
graph LR
    Z[User]
    A[User Action] --> B[Command]
    B --> C[Event Store]
    C --> D[Event]
    D --> E[(Projection)]
    F[Query] -----> E
    Z --> A
    Z --> F

A user might typically perform an action that generates a command, consider this a typical write flow. This command is then processed and stored as an event in the Event Store, the immutable append only event stream. The event can then be used to update projections, which are optimized views of the data for querying. The user can also query the system to retrieve information from the projections.

A frontend that wants to display the current state of the system would typically query the projections, which are updated asynchronously as events are processed. This allows for a responsive user interface that can display the latest information without blocking the main application flow. It could use the event store to retrieve the latest events and update the projections in real-time, ensuring that the user always sees the most up-to-date information. Using properties such as an aggregate id would allow the frontend to filter the events and projections to only show the relevant information for the user.

Advantages and Disadvantages of Event Sourcing

As with any architectural pattern, Event Sourcing has its pros and cons. Here’s a quick rundown of the key advantages and disadvantages to consider when deciding if it’s right for your system.

Advantages	Disadvantages
Every change in the system is captured and stored, providing a reliable and complete audit trail.	Event Sourcing introduces more moving parts and adds architectural complexity.
The system can scale by processing events across multiple services or nodes.	Systems using Event Sourcing are eventually consistent, which can be confusing for users if not handled carefully.
Services can stay loosely coupled and evolve independently by reacting to events.	Rebuilding state by replaying long event histories can become slow without careful optimizations like snapshots.
The system is flexible and can easily adapt to new requirements without losing historical context.	Managing event versioning over time can be challenging and requires disciplined planning.
Events can be replayed to rebuild projections, recover from failures, or test changes safely.	Evolving event schemas without breaking existing consumers adds additional complexity.
The event log can be a powerful source of historical data for analytics and reporting.	A dedicated event store is usually required, which may introduce new operational overhead.
Projections allow you to build fast, query-optimized views of the data for different parts of the system.	Projections need to be carefully maintained and can drift out of sync if event processing fails.
The flow of how the system reached its current state is transparent and easy to trace.	Running an Event-Sourced system requires solid operational controls around retries, ordering, and failure handling.
The event-driven nature of the system promotes modular design and makes it easier to integrate new services over time.	Debugging across asynchronous event flows can be harder without good observability and correlation tools.

Building a Simple Event Sourced System using AWS

This section provides a high-level overview of how to build a simple Event Sourced system using AWS services. Treat this as a starting point for understanding how to implement Event Sourcing in practice.

Event Store

As mentioned before - the event store will be our append only event stream. In AWS, you can use services like DynamoDB, Kinesis, or EventBridge to implement the event store. For this example we'll use DynamoDB as it provides a simple and cost-effective way to store events.

graph LR
%%is-centered
    A[Commands] --> B[(DynamoDB<br>Event Store)]

We'll take advantage of DynamoDB's ability to store large amounts of data and its support for high throughput. Each event will be stored as a separate item in the DynamoDB table, with a unique identifier and a timestamp. We'll also use DynamoDB Streams to capture changes to the table and trigger further processing of the events, ultimately pushing them to SNS via EventBridge - EventBridge giving us room to create rules and extend if we need to.

graph LR
%%is-centered
    A[Commands] 
    --> B[(DynamoDB<br>Event Store)]
    --> D[EventBridge]
    --> E{{SNS}}

At this point, we have a basic event store that captures commands and stores them as events in DynamoDB. The events are then streamed to EventBridge, which can be used to trigger further processing or notifications. Using E

Projections

Projections are derived views of the event data that are optimized for querying. In AWS, you can use services like Lambda, DynamoDB, Elasticsearch, RDS to build projections. For this example, we'll use Lambda to process the events and update the projections in Postgres running within RDS and DynamoDB - both which would solve different use cases.

%%is-centered
graph LR
    A{{SNS}} --> B[Lambda]
    B -->F[RDS Proxy]--> C[(RDS Postgres)]
    A --> D[Lambda] ---> E[(DynamoDB)]

Fanning out allows us to update multiple projections in parallel, ensuring that the system remains responsive and can handle high volumes of events. The Lambda functions will process the events and update the projections in the respective databases.

Querying the Projections

To query the projections, you can use the respective databases directly. In this example, we have two projections prepared for different access patterns, so we can use SQL queries to retrieve data from RDS Postgres or use DynamoDB's query capabilities to access the data stored in DynamoDB.

%%is-centered
graph LR
    A(User) --> B[API Gateway]
    B --> C[[Lambdas for Queries]]
    C --> D[Query Projections]
    D --> E[(RDS Postgres)]
    D --> F[(DynamoDB)]

The API Gateway will route the queries to the appropriate Lambda functions, which will then query the projections in RDS Postgres or DynamoDB. This allows for a flexible and scalable querying mechanism that can handle different access patterns.

Conclusion

Event Sourcing is a powerful architectural pattern that can significantly enhance the way we manage state in applications. By capturing every change as an event, we gain a complete audit trail, the ability to reconstruct state, and a flexible architecture that can evolve over time. While it introduces complexity and requires careful planning, the benefits of Event Sourcing make it a compelling choice for many modern software systems.