Combining Random Events to Solve IT Problems and Finding the Blind Spots

Boy these are some genuine good points to follow and “not controlling variables” is a big one here. Dr. Halamka wrote in his blog about a scenario he dealt with and sometimes how information, issues and so forth come in from many areas and it can be a challenge sometimes to get all issues into one area to solve, but that’s kind of the way things work at times and every CIO around probably has this issue.

You can read the entire article at his post but the big message I got out of the post is the one that mentions “the blind spots”. We like to think we are on top of everything but sometimes software installs or other issues are done without being on top of what actually occurs. Last year I attended this conference and heard a lot about the “blind side” discussions mostly from big organizations and how the CIOs dealt with it. It happens with nobody at fault as Dr. Halamka mentions above as there’s no control on a lot of how it happens, but then again, having coordinated plan to put it all together and solve is yet one more challenge today. Sometimes others outside of IT expect the department to be 100% on top of everything but it’s just not so.

Institute for Health Technology Transformation Convention–Fall 2010 Convention - Insights From The Medical Quack

With all the varied software solutions out there today sometimes it really is an effort to get all the other departments to keep the IT folks up to date as non tech folks may not always understand the complete planning and coordination that goes into everything today. You miss one spot and it will eventually show up if not right away. BD

As CIOs we have significant responsibility but limited authority.   We're accountable for stability, reliability, and security but cannot always control all the variables.
Here's an example of random events coming together to create a problem, which is now well on its way to resolution.   However, there are many lessons learned that I'd like to share with you.

1. Policy and Technology need to be developed together.   No amount of hardware technology will satisfy customer needs unless there is some policy as to how the technology is used.    I should have focused on demand management in parallel with supply management, enforcing rigorous quotas and providing useful self-service reports while the chargeback model was being revised.
2. Governance is essential to IT success.   Although no one in the user community relayed any storage plans or issues to me, there should have been appropriate committees or workgroups established to coordinate efforts among research labs.    In administrative and educational areas, it's common for groups to coordinate efforts with enterprise initiatives. In research areas, it's more common for local efforts to occur without broad coordination.    Establishing governance that includes all research lab administrators would help improve this.
3. Approval processes for purchases need to include IT planning.   When grants are used to purchase equipment there is no specific oversight of the infrastructure implications of adding such equipment on firewalls, networks, servers and storage.   Purchases that generate data should require additional approvals to align infrastructure supply and demand.
4. Be wary of "Big Bang" go lives. Our high performance computing upgrade was a single event - 1000 cores to 4000 cores.   This should have been phased to better assess the impact of the expansion on application use, other elements of infrastructure, and customer expectation.

5. Know your own blind spots. As with the CareGroup Network Outage, there are aspects of emerging technology which are so new that I do not know what I do not know.    When storage demand increases by 70% and throughput accelerates by a factor of 4, what happens to an advanced storage infrastructure?   Bringing in a third party storage consultant would have filled in knowledge gaps.
In the world of healthcare quality, there's an analogy that error is like slices of Swiss cheese. If a stack of individual slices is lined up precisely, you can get a hole all the way through the stack.   In this case, a series of unrelated events lined up to create a problem.   I hope that IT professionals can use this episode to realign their "slices" and prevent infinite demand from impacting a limited supply.