Tuesday, July 21, 2009

Complex Adaptive Systems: Why most Information Systems don’t Measure Up.

Complex Adaptive Systems, or Complexity Science, is about a lot more than Information Systems, we human beings being a prime example of such complexity. Wikipedia and other sources provide all the details you want on this topic, but one view of it that has stuck with me over the years is the difference between ‘complicated’ and ‘complex’.

Roughly speaking, a complicated system is what it sounds like, a system that is not simple and not intuitively understandable; however, given time and effort, you could analyze and identify all its parts and pieces and describe the whole system. The key is that it is a fixed system.

In contrast, a complex system cannot be analyzed and described as above because it is changing over time; analyze it last week, and again next week, and you will get two different descriptions. For decades, we have been fairly good at creating complicated information systems, but complex was and is rare, if even considered. Instead, what we have in IT is the Change Request, the Backlog, the up-coming Maintenance Release.

The problem has been our creating of information systems based on the needs of an organization at the time we asked. We were guilty of hard-coding the business process and rules (while probably not recognizing them as such) and delivered that code in working order, to be told that the business wasn’t doing things that way anymore; or worse, the business would keep this to themselves and start creating the first of many ‘workarounds’ needed to get past the system limitations and get things done. In either case, the situation would deteriorate and then the change requests would start coming.

Now, hind-sight is wonderful, and decades of hard, well-intentioned IT work should not be scoffed at; this has been a difficult realization to come to, common in many disciplines where complexity science has tussled with older paradigms; and the realization did come together in pieces over time as parts of the problem were recognized and new technologies & tools were developed to help. The now ubiquitous Database Management System came from realizing that lots of flat or indexed files processed in various batch jobs resulted in lousy information, and certainly weren’t going to work in on-line transactions, and even then we had to work our way through competing designs like hierarchical systems before the relational systems we use today emerged.

However, when it comes to complexity, data needed by a business is comparatively stable; it is how the data is used that changes more. Take a typical business process of handling a loan application; today the big loans go to Fred to process, but next week they go to Anne. That is the kind of change you cannot make to a hard-coded system fast enough to support the business. Beyond that, consider the rules that drive a process, such as loans over $100,000 need to be approved by a VP; and starting next week, it is changing to loans over $75,000.

This kind of complexity has finally led to some changes and new tools to help the business. First out of the block has been Business Process Management tools (BPM), which originated when imaging systems changed offices from moving paper around to moving units of work. More recently, Business Rule Management Systems are being implemented that move the changing of rules from the realm of the programmer to that of the business expert.

Will these tools be enough to move us from complicated, maintenance-heavy systems to truly flexible complex systems? There is no denying it will help; they will certainly solve a lot of old problems, so that we can move on to solving newer, better problems and what more can you ask for.

About Me

Ontario, Canada
I have been an IT Business Analyst for 25 years, so I must have learned something. Also been on a lot of projects, which I have distilled into the book "Cascade": follow the link to the right to see more.