Business Rules Management and Testing - Avoid the “Mystery Box”

Dietrich Kappe, Friday, June 30, 2006 @ 3:12 pm

I apologize for being tardy this week with my post. The truth is that I just got my hot little hands on the latest Blaze Advisor with the new RETE III engine and have been heads down testing it out. It's too soon for me to really say much about it other than "it's fast." Man is it fast. That's not a benchmark, mind you, just an impression from years of working with rule engines. OK, more on Blaze later.

OK, so over on The Road Hits the Rubber, Peter Lin muses on the shortcomings of Business Rule Management Systems:

• many users have very specific ideas about how to manage rules, but most don't bother using it. This applies to products from iLog to other vendors.
• users who actually manage rules actively have very specific ideas about how it should work. In this case, existing tools don't match what they want.
• an all purpose rule management system must be extensible for it to be useful. I'll go into this in greater detail
• many companies using rule engines and business rules have poor process in place. the typical horror story is the business analysts deploy a rule without testing it and the production system crashes. It's frightening how many companies do this.
• many business currently using rule engines are not using RETE algorithm and actually don't bother running compliance due to performance. this is actually very common and many businesses use performance as an excuse not to run compliance.

Here, here. Many companies look at the BRMS offerings of their vendor and say "we don't work like that." Rules end up being dumped out into an export format and checked into a source code repository. This is no way to manage rules.

Peter suggests a number of solutions to the above problems. There's lots to digg into here but I'd like to focus on one thing and take issue with him a little bit. The last two points on testing and compliance are, in my opinion, a very important part of what a BRMS should deal with. Peter doesn't agree:

Many sales guy ask for features like rule validation, spell checking and all sorts of stuff, but that isn't part of managing rules. That's the authoring of rules. To me authoring rules includes things like validation, profiling and application testing. Managing rules is a different discipline than writing rules, so it's best to separate the two activities and provide a clear integration between the two. When an user wants to save a draft of the rule, it sends the draft to the BRMS. If someone like his supervisor wants to review the rule, he should be able download it from the BRMS server.

From my experience, you are managing rule sets, not just individual rules. If you write your tests and run your validations at rule authoring time, you are vulnerable to someone deploying an invalid or nonsensical rule subset to a production system. Rule set validation and regression testing should thus absolutely be a part of the BRMS; adding, subtracting or modifying a rule into a large system can have many unforeseen consequences and unit and regression testing of rules and rule sets is one excellent way of addressing this problem.

Maybe I misunderstand what Peter means here by validation. I take it to mean that you are checking an entire rule set for correctness within a context; that context may be a particular dataset or a particular set of tests. A rule set that is correct for a population of Medicare patients may not be correct for a set of Medicaid patients. Thus the tests you perform on a rule set are specific to where you are deploying that set.

Taking a page from the OO world, writing unit and regression tests and testing for code/rule coverage should be built into the BR system life cycle. If you are authoring a new rule, you are likely doing so to change the behavior of the system. You should have a clear idea of what that change consists of and how it would be expressed in a test case or regression test. Otherwise, if you can't specify what sort of change in behavior you are trying to achieve, you have no business modify the BR System to begin with.

Lack of aggressive and rigorous testing in rule management leads to what I've termed the "Mystery Box." This is the rule based system whose inner workings are so mysterious that no one can tell how it arrives at it's conclusions or whether those conclusions are correct. Imagine a company where fixes in data quality or rule set correctness would result in newly correct behavior for a rule-based system; they would face a crisis because they had been feeding their customers incorrect information for years. Don't let the "Mystery Box" happen to you.

Technorati : bre, brms, business rules, business rules engine, rule management

business rules

Comments: 3 so far

• Leave a comment
• Comment feed for this post
• TrackBack URL

1. 

   Good comments. I’ll try to explain more clearly what I mean by validation. The short explanation is testing the rule, ruleset and rulebase to make sure what gets deployed is valid. You make a good point that a single ruleset may be valid for one dataset and invalid for a different dataset. I’ve seen this first hand, so in my mind, the validation should be flexible and modular. I’ve blogged about this in the past, but in random incoherent spurts. A few years back I wrote up a 40 page functional specification describing a design and architecture for rigorous rulebase validation. Clearly, leaving out validation is asking for trouble. My bias preference is to have that as part of the authoring. The reason for that is this. We the system allows a rule to become active and is set to be deployed in the near future, how should the system deal with it? Should it exclude the rule automatically? Should it try to fix the problem? Should it include the rule, but disable it? I would prefer to tell the rule author the rule is bogus or bad as they are writing it. That way, the user gets feedback sooner than later.

   peter

   Comment by Peter Lin, Friday, July 7, 2006 @ 7:51 pm

2. 

   I wrote up a quick blog about validation. hopefully this clarifies a bit what I mean by validation.

   http://woolfel.blogspot.com/2006/07/what-constitutes-validation.html

   peter

   Comment by Peter Lin, Friday, July 7, 2006 @ 8:14 pm

3. 

   Good to read topics about Rule Validation and the reference made to unit testing. There is another part of testing that was not mentioned and I thought this might be relevant; It is called rule verification.

   For the difference between rule verification and rule validation see the whitepaper:

   http://www.acumenbusiness.com/whitepapers/Agile%20Business%20Policies.pdf

   I agree that the rule validation part can only be done by the business analyst. Only he has the knowledge of what the intent was of the rule.

   The next version of the Rule Manager will have a rule validation component. This allows to create ‘unit tests’ for just business rules, by business analysts.

   The Rule Manager is available as a free download from http://wwww.acumenbusiness.com site

   Comment by Marco, Saturday, March 17, 2007 @ 10:24 am

Leave a comment