My recent post on BPM 2.0 and Complex Event Processing started quite a few discussions, some online (like this one with Bruce Silver and Paul Vincent), some offline with a few prominent CEP vendors, who seemed to agree with my position. Let’s give it another shot and see where this takes us.

First, some external validation that BPEL is perfectly suited for describing complex event patterns: Coral8, one of the leading independent CEP vendors, recently developed a very cool BPEL to CCL compiler that takes standard BPEL code and translates it into CCL, Coral8’s SQL-based programming language that offers a fast, natural way to develop CEP applications. This compiler is now being used by SAP to deploy CEP applications from BPEL code generated by their BPMN designer. Once again, the BPMN+BPEL duo is proving that standards matter (Cf. Why BPEL Matters).

Second, the fact that one vendor’s BPEL engine was not appropriate for CEP and led that vendor to embed a third-party CEP engine (Cf. Paul Vincent’s response to Bruce’s post) is by no means proof that one could not use a BPEL engine for this purpose. All it says is that this vendor’s BPEL engine just wasn’t good enough. I won’t name this particular vendor, for trashing my competitors usually is not part of my modus operandi, but I have no problem telling you what was wrong with their engine:

1. The engine cannot support very many process models deployed on the same server.
2. The engine is not capable of very high transaction volumes.
3. The engine is not real-time.

Any one of these limitations would make your CEP platform brittle at best. Get all three together, and you have to call it a day, which is precisely what this vendor did. But guess what? Other BPEL engines are not subject to such limitations, and so is the case for Intalio|Server. More on its performance can be found there.

Third, saying that a good BPEL engine could be used for CEP purposes does not mean that it’s enough. In fact, it’s not, but only if you want to cover the entire scope of CEP and ESP (Event Stream Processing, its close cousin). Where a dedicated CEP engine shines is when you have to process a very large number of events that cannot be related to any given process (or collection of processes), have different data structures for their payloads, and can occur over very long periods of time. In order to address such a scenario, not only will you want to store events in a datastore that looks nothing like the one used for process audit trails, but you will also need a data-centric language for describing your event patterns (either something like SQL or an object-oriented language). And make no mistake: Intalio’s Business Process Platform will most definitely provide such a dedicated CEP+ESP engine in the future.

Thinking about it, all this discussion reminds me of the process vs. rules discussion that was taking place four or five years ago. Back then, people spent a fair bit of time arguing whether a dedicated business rules engine was needed alongside the process engine, whether the process engine could be used for rule flow, or whether the rules engine could do everything on its own. The conclusion was that a process engine was needed, that synchronous rule flows should be implemented directly within the rules engine, and that asynchronous rule flows should be left to the process engine.

Another thing we can learn from the experience that was accumulated over the years trying to build general purpose rules engine was that you need a very different engine whether you’re dealing with a large rule set applied to a small data set, or a small rule set applied to a large data set. For the former, you will need a traditional rules engine like Fair Isaac Blaze Advisor or ILOG JRules. For the later, you will need something closer to what Versata had developed in the late 90’s, or what very many CEP engines do today. Very different solutions for very different problems.

But this is only the technical side of the story. Why does all this matter from a business standpoint? Well, for a very simple reason that can be spelled with just three letters: GRC (Governance, Risk Management, and Compliance). In its first incarnation (Virsa, Fermat), GRC was very much data-centric, after-the-facts. Post 2008 Financial Crisis, GRC 2.0 will become process centric, real-time, enforced by a whole new set of regulations and policies that will make things like SOX and Basel II look like a walk in the park. In order to address such a challenge, a true Business Process Platform will have to interweave events, processes, and rules in the tighest possible way, and this precisely what Intalio is up to.