Implementing
Business Rules as Data—
A
Case Study
Introduction
Designing and building effective software systems is an extremely complex task in organized and well-structured environments. What do you do when the client does not have control of their business processes? When policy and procedure can change on a moment’s notice and can be mandated by organizations that are not under your control? This case study will examine the successful implementation of a system that faced exactly these problems. It will show that the only viable approach was embedding business rules in the data architecture. This solution allows changes that would have taken weeks to accomplish using traditional approaches involving complex modifications to system code to be implemented and tested in minutes by manipulating business rule data. The end result is a highly adaptive system with reduced maintenance cost for the client.
Project Background
The United States Air Force Recruiting Service (AFRS) is responsible for recruiting people to fill the manning requirements of the US Air Force. They currently recruit in excess of 35,000 people a year from an annual candidate pool of 4 million. In September 1997, AFRS started a modernization effort to replace a highly distributed data system that was expensive to maintain and no longer provided adequate support for their business requirements.
The legacy system focused on collecting data on potential recruits and almost totally ignored management issues that could make Air Force recruiting more efficient. The system provided no mechanisms for managing the recruiting process (meaning policy could not be enforced by the data system), and provided no means for evaluating the effectiveness of recruiting activities, both of which were serious functional deficiencies. Additionally, the distributed nature of the legacy system made day-to-day operations expensive and caused updates to the system to address changes in policy very time consuming and costly.
The functional deficiencies, and the complexity and cost of maintenance led to an AFRS decision to undertake development of a new system. Modernization would move the complete recruiting business to a centralized Oracle database, accessed by recruiters worldwide via a web-based application developed using Oracle tools.
Key Client Requirements
Analysis of the AFRS processes and data requirements identified the need to support the following capabilities:
· Acquisition and evaluation of test results for multiple tests. AFRS uses a well-defined set of tests to evaluate potential recruits. Each test has a defined domain for the scores (not all the same), and each test may have one or more scores as an outcome. Additionally, acquisition and use of all test scores follow the same management and documentation rules.
· Administration of multiple structured interviews. Candidate interviews are a key element of the recruiting process. Required interviews are defined (and changed) by policy. All interviews follow the same management and documentation rules.
· Management of internal processes. Recruiting processes are defined and managed to improve recruiting efficiency and ensure recruiting standards are met. Process rules are defined (and changed) by policy. All processes follow the same management and documentation rules.
· Enforcement of security issues. Users should be prohibited from seeing information and processes that are not required for the performance of their duties. Visibility of application content should not span organizational boundaries. The Air Force Reserve should not see Active Duty applications, for example.
· Process Management. Implementation of common processes and process rules was a primary objective of the client.
Architectural Strategy
The dynamic nature of the client environment prescribed a strategy for meeting the user requirements that went beyond just building a simple application. It was necessary to develop an architecture that addressed client needs while ensuring that the result of the effort would be a useable application. Political issues related to similar Department of Defense systems under development also influenced the solution strategy. The following objectives were identified
· Produce a system that could be considered for adoption enterprise (Department of Defense) wide.
· Use an enterprise level approach focusing on the development of a system that integrated into the enterprise and supported its mission.
· Develop an infrastructure that would allow an organization to define, or modify, its processes without requiring extensive and costly changes to application logic.
The Architectural Components
Once the goals and objectives of the effort were clearly understood it was possible to begin working with the functional experts to develop an understanding of the problem domain. Analysis of the problem domain relative to the architectural strategy and objectives identified key areas that needed to be addressed if the necessary degree of flexibility was to be achieved. The analysis identified three areas that needed to be managed dynamically: management of organizations and programs, definition of activities (which included data collection) and definition of process and process rules. These areas when combined provided a complete solution to the client’s problems.
Managing Organizational Structure and Programs
The management of organizational structure and the recruiting programs that support the organization is not particularly complex or interesting. It is, however, important to the overall implementation of the system as it provides the anchor for the system security model and the other more complex metadata driven structures, which eventually define the functionality of the system.
A variation of a commonly recognized data model pattern, the Party Pattern, sits at the heart of the structure and establishes the foundation for management of the organizational structure and applicants. One key difference is that the PARTY RELATIONSHIP entity was replaced by a set of associations each representing a class of relationships of interest to the client. This helped the client to understand the structure of the model. These associations included PERSON-PERSON, PERSON-ORGANZATION, and ORGANIZATION-ORGANIZATION.
In addition to the organizational hierarchies represented in the system, complex recruiting program definitions (some hierarchical) needed to be supported with the ability to apply any program to a subset of organizations. A program defines a set of validation rules that restrict and applicant’s ability to enter into a particular contract with the Air Force.
When an applicant begins processing he is applying to a particular program in a specific organization. It is possible that the applicant is processing for two different programs in two different organizations at the same time: Active Duty Air Force Officer and AF Reserve Health Professions, for example. In this case neither branch knows about the other application but personal history is shared across the applications to help identify and reduce fraud.
The PROGRAM to ORGANIZATION association provides the anchor for defining the rules and activities related to the processing of applicants to a particular organization’s program.
Managing Activities and Workflow
Activities and workflows provide the user visible definition of the systems functionality. The risks associated with data requirements being defined after system delivery (as a matter of evolving policy, not lack of requirements analysis), coupled with the repeated occurrence of apparent sub-types (each with distinctly different tuple definitions), led to the design and implementation of a single general data structure. This decision was reinforced by the need to enforce a consistent management process across all data in the areas being discussed.
Figure 1 presents the data structure that was implemented in the AFRS. Also included in the diagram are additional structural elements (PROGRAM, ORGANIZATION PROGRAM MAP, PROGRAM ACTIVITY MAP, CATEGORY, and ORGANIZATION) that were part of the data structure common to the definition and management of the business model supported through the generalized structure.
ACTIVITY allows for general definition of business objects, and the definition can occur at any stage of the systems lifecycle. A significant addition in this area is the inclusion of the MULT_RESP_ALLOWED, a control element to limit the defined ACTIVITY to one instance (PERSON ACTIVITY) when necessary.
Figure 1: The Activity Structure
ACTIVITY ITEM supports attribute definition. ACTIVITY ITEM TEXT provides the attribute name for the information to be recorded. In the AFRS system, this name actually became application data for the INTERVIEW class of objects defined in ACTIVITY. For interviews, the “name” in TEXT actually became the interview question being asked. A benefit of this structure is that, if AFRS needed to conduct tests, TEXT could contain the test question being answered and the new functionality would not involve any change to the structure of the database.
Attribute-Entity binding is achieved through the ACTIVITY ITEM MAP. It should be noted that this construct allows an “attribute definition” to be reused across multiple entities. This was valuable for interview processes, where it was desirable to ask the same question in more than one interview.
For AFRS, domain definitions were maintained as VALID ITEM RESPONSE. Additional application control elements were added:
· to control branching between modules in the application if particular domain values occurred in the collected data. In this case, FUNCTION BRANCH designated the application module executed upon selection of the domain value in question; and,
· to ensure required recruiter actions happened if specific domain values occurred in the collected data. For AFRS, ACTION REQUIRED = YES ensured a recruiter entered a remark for the RESPONSE using the domain value in question.
The actual data collection was implemented through the RESPONSE entity.
Note that several sub-types of ACTIVITY are represented in the model. The actual physical implementation uses a lookup table for the ACTIVITY TYPE so that new activity types can be added without modifying the structure of the database.
Process Control
Process control is paramount in the application environment. Policy and law determines how the user performs his job and both policy and law can change. A workflow is used to link activities together to form a business process. As such, the workflow is a critical concept in the implementation of the final system. (Note: Workflow is implemented as an ACTIVITY TYPE although it is not shown in figure 1.) Within a workflow some of the processes involved are actually accomplished as the result of information gathered and entered into the system by the user. For example when the user enters that a person has a driver’s license it triggers the process of collecting driver’s license specific information via a form. If the person has no driver’s license this form is never presented to the user.
Managing Process and Security Rules
Security and process control are tightly coupled in the system. Within each organization there are classes of users with different responsibilities relative to the overall process.
The Data Model
The data model for the process control infrastructure represents the processes
or activities being controlled, the statuses that describe the condition of the
processes or tasks, the relationships between the tasks or processes, the
controls that statuses exert on each other, and the relationships between
organizations, users, responsibilities and statuses.
Figure 2: The Status Control Structure
The model has three principal areas: the activity or process being controlled (light blue), the current state of the activity being controlled (light yellow) and the rules that govern the process (light green).
Process (light blue)
The process being controlled is represented in the diagram by the ACTIVITY, PERSON and PERSON ACTIVITY entities. The most important one is the PERSON ACTIVITY, which is the specific item being controlled.
Process State (white)
The current state of the process being controlled is represented by the STATUS entity derived through the STATUS DEFINITION and REASON DEFINITON look-up tables. A decision was made to implement all domains as entities within the model. This provided the ability to use the embedded control mechanisms to manage the choices that appear in the users lists of values.
Control Infrastructure (light green)
The strategy used for the development of the control infrastructure involves isolating those entities involved in the management of the control process from those that are actually being controlled. Each element of the control structure represents, and ultimately implements, a business rule.
Important Entity Definitions
STATUS
REASON MAP
The status reason map identifies valid status/reason pairs within the system.
VALID
PROGRAM STATUS
The valid program status identifies the programs for which status/reason pairs are valid.
STATUS
ACTIVITY MAP
The status activity map identifies which valid statuses apply to particular activities. It also identifies what responsibility level has control when this status is active.
STATUS
CONTROL
The status control states rules that describe the interactions between status activity pairs. Attributes in this entity determine whether the first status enables, inhibits or triggers the second status. If the second status is triggered the method used to trigger the new activity status pair is specified. The valid methods are insert new, update existing, insert or update, update – error if the activity doesn’t exist, and insert – error if the activity already exists.
STATUS
GRANT
The status grant entity specifies what user roles can exercise a status control.
As always there is an exception to any rule. We stated earlier that all domains were modeled explicitly. One domain, responsibility level, which identifies what organizational level has responsibility for an activity when it is in a specific status is shared by attributes in the STATUS ACTIVITY MAP and ORGANIZATION entities. This is particularly important since it is this pair of attributes that is used to manage access rights to the activity. One a role has access the normal controls based on the role take effect.
Rule Engine
Each status affects activities, process flow and other statuses. The system administrator must be able to modify system capabilities to account for changes in policy or procedure without having to modify application code. An application level screen provides access to the control structure for definition and maintenance.
The process control engine provides four services for the application. It uses the current state of a set of related activities to determine whether or not a user can perform a specific activity or workflow. It also performs a similar analysis to determine if a requested state change is valid. It manages inserting or updating states triggered by user generated state changes. Lastly, it manages the movement of responsibility for an activity from person to person, or organization to organization, based on the statuses being inserted.
The most complex process performed by the engine is determining what activities or activity/status pairs are currently valid based on the current state of a group of related activities. An activity can be performed if any of the activity/status pairs for that activity is currently “enabled.” An activity/status pair is “enabled” if a current activity/status pair enables it through the STATUS CONTROL structure and it is not “inhibited” by some other current activity/status pair.
Implementation
The engine for processing the status information consists of two packages and a series of database triggers that invoke the procedures and functions found in the database packages.
Processing the status information involves the analysis of incoming status values in the context of the existing state information. To avoid problems with mutating tables actions are queued in the BEFORE-INSERT-UPDATE trigger and processed in the AFTER-INSERT trigger.
LOOK_AHEAD
The LOOK_AHEAD function is the workhorse of the system. It is called by application level functions to determine whether or not a particular status can be set which in some cases determines when a specific activity can be performed.
Since one status can trigger the setting of another status the engine must examine the state of the system relative to the second status. If the second status is not “enabled” or is blocked by some other status then the original status is also blocked since it cannot perform all of its associated operations..
This analysis is accomplished through a depth first recursive search that fails whenever a blocked status is encountered. It should be noted that depending on the characteristics of the particular application, a breadth first search may be more appropriate from a performance standpoint.
SET_RESPONSIBILITY
Each status control record has a defined responsibility level associated with it. It defines the responsibility level within the system that gains responsibility for the activity when the status is set.
Whenever a new status is set its impact on the activity responsibility is analyzed and users are associated with, or disassociated from, the activity as necessary.
It is important to note that in managing the statuses it is possible for two statuses to set two different responsibility levels at the same time or for two different statuses to cause the same responsibility level to be set. Some care must be taken to ensure that this case is accounted for.
SET_STATUS
The SET_STATUS function is executed in the database triggers. Its purpose is to effect any status changes that have been triggered by the database activity. It verifies that the status changes can be initiated and applies the changes as necessary. It will return an error if any of the status changes would violate the rules defined in the database.
The Application
The end user application has two basic pieces: the end-user application and the maintenance application. The end-user application is comprised of a combination of hand written and generated forms and reports. The maintenance system is primarily composed of generated forms and reports.
Maintenance
Functional experts, developers and help desk personnel perform administration functions using the maintenance application. This involves managing the look-up tables, organizational structures, business rules, activities and workflows. This section describes the management of organizations and programs, business rules, activities and forms.
Maintaining organizations and programs
Maintaining organizations and programs were straightforward operations – both elements were concrete “concepts” that were implemented in an intuitive manner. These definitions, in conjunction with defined associations between programs and organizations, were the key elements for managing security and for classifying applicants to ensure correct validation rules were applied before they signed accession contracts. Defining an organization consisted of creating an appropriate PARTY (to include specification of responsibility level) and inserting the defined party into its organizational hierarchy in the ORGANIZATION-ORGANIZATION table. Program definitions were managed similarly in a simpler structure; hierarchical relationships were implemented in a simple recursive relationship on the PROGRAM table. Finally, programs were declared to be valid for specific organizations on the ORG PRGOGRAM MAP table. The screens for managing this data are uninteresting, and not presented.
Changing rules
Activity status information is maintained using the Status Control Maintenance Screen (Figure 3). This screen allows a status to be associated with an activity. As the status is associated with an activity it’s effect on the activity is defined. Specifically three questions must be addressed: is the activity required to pass through this status?, does the status end the activity?, and who is responsible for the activity once it attains this status?
Figure 3: Status Maintenance Screen
Once the status is associated with an activity its effect on other activities and statuses can be defined. In the example shown above the status of “forwarded to HQ USAF” for activity “Age Waiver” causes the status of “working at HQ” for that activity to be enabled and set.
Maintaining activities
“Activity” is the catch all term used to describe the tasks and processes whose controlled state interactions enforced the overall policy of AFRS. “Activities” spanned applicant data collection tasks such as interviews and tests, while others defined processes (state transitions were the only “data” collected). Maintaining activities consisted of creating a record on the ACTIVITY table, creating any required attributes on ACTIVITY ITEM table, and specifying the attributes membership in an activity on the ACTIVITY ITEM MAP table. Additionally, domain information for any ACTIVIY ITEMS was defined on VALID ITEM RESPONSE. Inserting Activities into the application for use in the field consisted of defining its interactions with other activities, as describe under changing rules, above.
Adding new forms
In general, data collection forms for gathering applicant data were accessed in the course of workflows, not through menu actions. As such, workflows included structure for registering all user accessible application executables and declaring their access sequence in each workflow. This being the case, adding forms to the system was a data management action, not a coding action.
The End-User Interface
When applied to user level applications the status control structures combined with activity ands workflow definitions provide a powerful and dynamic enrironment. Process control is precisely managed based on the rules defined in the database. Looking at a simple example, figure 4 shows that there is no work pending for the user in this case someone working at the Military Enlistment Processing Service.
Figure 4: MEPS Processing
An applicant enters the system and a recruiter sets an appointment with
him. When the applicant and recruiter
meet during the appointment several interviews (Figure 5) must be conducted to
start the process.
Figure 5: Recruiter Interview
As the application process progresses, the options available to the recruiter
change. The screens shown here (Figures 5 & 6) demonstrate the change in
options available to the recruiter as the application progresses.
Figure 6: Recruiter Processing
When the recruiter has completed his tasks relative to the application he
forwards the application to the MEPS for processing. The screen shown below shows the original MEPS processing screen
(Figure 4) with the addition of one applicant who has appeared on the screen as
a result of the internal systems processing.
Figure 7: MEPS Processing show new applicant to process
System Report Card
The rules implementation discussed in this case study had both significant strengths and weaknesses. The strengths and weaknesses define the trade-offs involved in implementing data-centric rules engines.
On the one hand, the implementation is exceptionally extensible, without modifications to the code-base or data schema. The behavior of the system can be altered dynamically, over a wide range of profiles, making system upgrades in a dynamic environment relatively easy.
Conversely, that very flexibility offers the owner of the application the opportunity to destroy the stability of the system. The apparent ease with which changes can be made can lead to sloppy maintenance practices that are less prevalent in code-based rules solutions. The effort, skills, and communication required to implement a code-based solution ensures a review cycle in rule definition that may not occur in a situation where the person defining the rules also implements them.
Strengths
· Very stable database structure
· All process rules documented in database
· Flexible and Adaptive
Weaknesses
· Rule definition and maintenance requires careful planning
· The limited rule set serves their current need but may not serve all future needs
The Future
Recent changes in AFRS philosophy has caused a need to evaluate the possibility of extending the supported rule set to include management of column level constraints. For some organizations and programs not all of the data is required or it is required based on the state of the applicants application. Extending the structure to support temporal column level constraints would increase the complexity of the software, but it would also increase the flexibility of the system significantly. It has not yet been determined whether or not to implement these changes.
Summary
The AFRS recruiting system offered the Air Force a dynamic and adaptable process management system. The flexibility of the design ensured that, with the correct commitment to developing skills in the operational community (to whom the maintenance of the data stating policy would fall) the system could rapidly evolve. Additionally, while the rules engine was restricted in its generality, in that it addressed only process-oriented rules (i.e., what processes exist, what tasks constitute the process (in what order), what states are allowed for each processes and task, and the security issues of who could take action at any point of the process) it addressed the most dynamic portion of the AFRS business and ensured processing policy could be enforced.
Having said that, there were significant shortcomings in the final outcome that will clearly color our application of this type of solution in the future. The ease of modifying the system during the development period, when highly motivated functional representatives were integrated into the team, resulted in a level of complacency about the system’s maintainability that was not warranted. Regardless of how business rules are implemented, correct analysis of the business must occur, and the appropriate translation of the rule to the data system achieved. This demands commitment, and it requires discipline.
In some manners, stating rules as data is not as intuitive as stating them as code, although evaluating the rule base is probably easier (if one knows what to look for). The complexity of the rule set is the same under either implementation strategy – however, the data centric approach, if coupled with the intent to empower the functional owners of the system with the evolution of the rule base, calls for great caution. The level of documentation required, and the availability of effective data administration training is critical when the sanity check of the communication process between the functional owner and the implementing coder is by-passed.
About the Authors
Roland Berg is a senior consultant with ThinkSpark in San Antonio, Texas. He has more than twenty years of experience as a software professional. He has published and presented several papers focusing on rules-based systems, architecture, design, and development techniques.
Kevin Holt has extensive experience in the systems analysis, modeling, and design domain. For more than sixteen years, he has been immersed in the IT world, either a user of IT products or a solutions provider ( analyst / architect / designer / developer). He has a particular interest in understanding complex business systems and designing flexible, extensible IT systems that support business requirements efficiently. He is currently a staff engineer with The Kernel Group, responsible for the design and fielding of a new commercial product-enhancing, continuous-availability management for enterprises where uptime and ROI for IT services are critical to business operations.