Sunday, January 15, 2012

Events and Event-Driven Architecture: Part 2

In my last blog entry, I introduced Event Driven Architecture (EDA), compared it to Event Driven Programming and then spent most of the time diving into SEDA - Staged Event Driven Architecture, which is an intra-system design. In this blog entry, I will describe inter-system EDA, which is typically what people mean when they speak of "Event-Driven Architecture".

/* ---[ Inter-System Event Driven Architecture ]--- */

Event Driven Architecture describes a loosely coupled design between systems that communicate asynchronously via events. The format of the event is the API between systems and the bus between them is some form of messaging backbone, most typically a message queue.

Let's define "event" in the EDA sense (not necessarily the EDP sense) with a "business" definition and then dissect it a from a technical point of view.


a notable occurrence that is deemed important enough to notify others about. It can signal an action (such as an inventory change), a problem or impending problem (out of or nearly out of an inventory item), a threshold, a deviation or something else. It is typically a description of something that has happened, rather than a command to do something.

Event - technical breakdown

An event is communicated in a message with a header and a body.

  • The header can include: type, name, timestamp, creator and should be consistently defined across specifications
  • The body describes what happened. It should fully describe the event such that an interested party does not have to go back to the source system in order to get essential information.
    • This distinguishes it from a RESTful model (at least level 3 of the Richardson Maturity Model) where typically only minimal essential information is communicated. Instead URIs are provided for the client to go back and retrieve more complete information about an entity. REST is typically a request-reply interaction, whereas events in EDA are usually background asynchronous notifications broadcast to a messaging system, so a policy of providing all relevant details in the event message makes sense under those circumstances.

So what formats could be used for event info? Basically anything with a header and body. XML and JSON are reasonable choices. I'll discuss other options and considerations on how to choose a format in the next EDA blog post.

Interacting With a System

In an Event Driven Architecture, systems communicate with each other by sending events describing what just happened. These are generally not point-to-point connections and they are not synchronous request-reply connections. Instead, an event is broadcast to any and all listeners in an asynchronous fire-and-forget fashion. The broadcast is done via a queue or other messaging system. This is why EDA creates a more loosely coupled system than a traditional SOA design.

There are only two dependencies between systems now:

  1. the domain event format and content
  2. the messaging middleware

System A, publishing its events, is unaware of any subscribers to its stream of domain events. System A does not have any dependencies on subscriber systems. If downstream system B is currently down or overloaded, system A will continue publishing at its own pace. System B can get to its "inbox" whenever it is ready and it can do whatever it likes with the events it receives. None of that has any effect on system A, which can continue to go about its business as if it was the only system in the world.

Each system in an Event Driven Architecture can be carefree about other systems, but of course an Enterprise Architect still needs to ensure that all systems under his or her purview are interacting appropriately and meeting their SLAs. So with EDA there are still hard integration problems to solve. But there are circumstances where this loosely coupled design is a better and simpler choice. I'll cover an example later.

Note: there are some who would argue with the idea that an asynchronous eventually-consistent model is "simpler" than a typical SOA synchronous request-reply architecture.

As usual, whenever I use the word simple I (try to) always mean it in the sense portrayed by Rich Hickey in his Simple-Made-Easy talk - namely, simple means unentangled with concerns of others. Minding your own business is simple. Being involved and entangled in everybody else's is complex. Any repeat readers of my blog will probably get tired of me talking about it, but I think that was one of the most profound lectures I've listened to, so head there next if you haven't listened to it yet.

/* ---[ Some Formal Terminology ]--- */

Let's give some formal names to the players in a event-driven system.

First, there is the event generator. This is usually a software system that has just processed a request (say from a user) or an event (from another event generator).

Next, there is the event channel or event bus - usually a messaging queue such as a JMS or AMQP compliant broker.

Event subscribers are interested parties (applications) that sign up to receive events off the event channel. They then become event processors, possibly doing some action in response to the event or a series of events. The sum total of the downstream event processing is the event consequence.

/* ---[ Making It Real ]--- */

OK, enough formality, let's have an example. For this, I have found no better source than a presentation available on InfoQ by Stefan Norberg. I highly recommend watching this presentation. With apologies (or kudos) to Stefan, I'm going to steal his example here.

Suppose you have an online store where users purchase some goods from you. In your enterprise, your architects and developers have put together some front end systems like a shopping web site and a payment system. In the back end you have your databases, inventory and reporting systems. It is composed with SOA services through some enterprise server bus (ESB). Currently those systems interconnect with request-reply synchronous behavior.

Now management says to the IS team: "we want to institute two new features - a real-time fraud detection system and a loyalty program". If you stick with the current design, you will have to embed a fraud detection system in multiple systems - those systems will have to become fraud aware. And the fraud business rules may need to correlate across systems, building in even more complex dependencies. I'm going to steal two slides from Norberg's presentation and here is the first to illustrate:

(Image from slides available here)

This becomes an unscalable model. The fraud and loyalty systems are not only embedded in other systems, but they also will need to query the database(s) you have - thus creating more inter-dependencies, the opposite of a simple system. If a database schema were to change, the fraud and loyalty systems also have to be updated.

Event-Driven Architecture challenges us to look for opportunities where a more loosely-coupled design is possible. What if, instead, we pulled the fraud and loyalty systems out, made them standalone, perhaps give them their own databases (to keep just what they want for as long as they need) and had them subscribe to a domain event queue from the other systems?

If we modify the Shop and Payment systems to fire off domain events that describe what they do as they do it, the fraud and loyalty systems can subscribe to those in real-time. Remember also that events should ideally contain all information about the event so that subscribers do not have to go back to the source systems to get more information. This then also decouples the existing Customer and Reporting databases from the Fraud and Loyalty system, simplifying the design.

In fact, now that we are thinking this way, we can see that the Reporting database is a read-only database that could also be populated by subscribing to the event stream. So by rearchitecting now we have this:

(Image from slides available here)

In this slide there is also a generic Business Activity Module (BAM) to show that additional feature requests from management could be added here as well. This slide illustrates nicely how EDA and SOA can work together, allowing us to choose the right model for each subdomain.

/* ---[ Integration Modes ]--- */

Integrations, as you've seen, are not point-to-point in an EDA system. Thus, an EDA system is very amenable to an ESB or EIP (Enterprise Integration Patterns) distributed system, such as Apache Camel.

Unlike batch-based ("offline") modes of integration, EDA systems can stay up-to-date in real or near-real time. For example, instead of an hourly or nightly ETL job to synchronize datastores, one datastore can publish its new entries to a queue and the other system can subscribe and update its datastore immediately.

The flexibility at each layer the is strength of EDA. What gets generated, how it is processed/formatted, and who receives it and what they do with it are loosely coupled and each layer has dependencies only to the one whose queue/event stream it directly subscribes to. Even then, it is not a deep complex API-based dependency.

With CORBA, DCOM and RMI, the caller and recipient have to be in lock-step with each other through all version changes - they are hard-coded pre-compiled APIs that must match. But with Events, you need not have as much coupling. If event generators begin publishing new types of events, it will not break downstream systems and the subscribers can continue on as before, or adapt and be able to handle the new messages on their own update schedule.

You can evolve what types of messages you can process. For example, you could pass around the equivalent of HashMaps that contain the various data you need, rather than strictly formatted XML docs, for example. If you add more keys and values to a hash map, the downstream subscriber still can pull out the old ones it knows about.

The picture below shows how a reporting system can be upgraded at a different rate than the systems generating the events it subscribes to. As long as the basic “API”, i.e., format of the message, is retained, the consumers can be updated at a slower rate than the producers without breaking the overall system.

/* ---[ Next Installment ]--- */

Hopefully this installment in the EDA series gave you a sense of the power of EDA and promoted the idea that EDA and SOA are not competitors, but rather co-exist peacefully.

In the next blog post on EDA, I'll cover four additional topics: performance characteristics, domain event/message formats, complex event processing and guaranteed delivery. The latter is something you will want to have in many circumstances. Take our example above of having the reporting database populated by an event stream from the shopping and payment systems. In order to have accurate reporting, you will want to guarantee that eventually all events reach the reporting database, by some means of guaranteed delivery.

Key References:
1. Domain Event Driven Architectecture, presentation by Stefan Norberg
2. Event-Driven Architecture Overview (PDF), whitepaper
3. How EDA extends SOA and why it is important, blog post

Other entries in this blog series:
Events and Event-Driven Architecture: Part 1
Events and Event-Driven Architecture: Part 3

1 comment: