Tuesday, 13 July 2010

Anatomy of an IT Disaster - How the IBM/SAP/Workbrain Queensland Health Payroll System Project Failed

There has been a lot of media interest in the failed SAP payroll project at Queensland Health recently. It has been termed as an "unprecedented failure of public administration''. Just today in the Australian Financial Review, it was stated that even the superannuation calculations have become a tangled web of manual overrides and inconsistency (due to the original payroll amounts being incorrectly calculated). There is also going to be an internal government hearing today to work out how this failure happened. Surprisingly though, the Queensland Minister for Health will apparently keep his job (as per the following news article in The Australian Newspaper http://www.theaustralian.com.au/australian-it/minister-keeps-job-despite-queensland-health-payroll-debacle/story-e6frgakx-1225886060838). Now disaster on such a large scale (like a large train crash) drew my curiosity and I just had to ask:

How did this massive project failure happen, and how did it go so wrong, so far,  for so long?

This blog article is something akin to "Air Crash Investigation" on TV - but from an IT software perspective. As the US philosopher George Santayana (1905) said - "Those who cannot remember the past are condemned to repeat it." - and I'd like to learn from such a systemic failure in the Australian IT context.


Project Statistics:
The project was large by anyones's measure:


More recently, blame has been levelled at problems sourcing from the management by the CorpTech Shared Services - as per this computerworld article:(http://www.computerworld.com.au/article/352346/corptech_called_account_shared_services_failing/).
I know some SAP developers who worked on the project and they had some explanations as to what the main reasons for failure. They bailed out themselves as they could see the trainwreck that would happen down the line. They identified that IBM wasn't the sole point of failure - they were simply the last company to try and come in and fix the mess.

The Queensland Government is now attempting to sue IBM even though it has signed the application off as satisfactory. In terms of fallout from the disaster, the 2 top people in Queensland IT have been sacked, and it is likely that CorpTech middle managers involved will be disbanded.

Problems with the Queensland Health Project (aka Project Post-Mortem):
  1. [Project Management Issue] There was NO contingency plan (aka "Plan B") in place in case the payroll system went belly up (and it did). Way too much trust was put into the contractors to deliver a perfect, bug free result (no real-world software is 100% bug free) and not enough common sense was used to mitigate risks. 
  2. [Project Management Issue/Testing and Reporting Issues] - Testing Plan and Numbers were fiddled (!) so the application passed testing criteria - According to the Courier Mail Newspaper article(http://www.couriermail.com.au/news/queensland/queensland-health-payroll-fallout-to-reshape-awards/story-e6freoof-1225885400871 - they (quite heinously) fiddled the numbers - "Instead of slowing the process, the project board agreed to revise the definition of Severity 1 and 2 defects – effectively shifting the goalposts so the project passed exit criteria."
  3. [Project Management Issue] - There was no parallel run for the payroll between the WorkBrain System and SAP Payroll. This is what was recommended by SAP itself. I've had the SAP QA team come out to my clients and they do a pretty thorough job.
  4. [Project Management Issue] - There should have been a Gradual Rollout (you can't do ANY large payroll system in one hit/using a "big-bang" approach).
  5. [Architecture Issue] - The Architectural design is questionable. The Integration between the 2 systems is wrong - as WorkBrain rostering is writing directly to SAP (using flat files to pump data into SAP) rather than using the timesheets as the intermediary entry mechanism first. SAP Payroll systems are effectively bypassed by using WorkBrain and a bespoke system for payroll calculation and generation.
  6. [Testing Issue - Government Due Diligence Issue]  - The system had been signed off by Queensland Government without proper checking on their part (they are subsequently trying to disavow themselves of this responsibility though the end decision to go live was theirs and done through their project board).
  7. [Architecture and Project Management Issue] - Whether WorkBrain should have been used at all as it is a rostering application. Other States have just SAP systems and they operate acceptably.
  8. [Project Management/Procedural Issue] A failure of a contractor [IBM] and CorpTech to Follow SAP's recommendations.
  9. [Change Management Issues/Lack of training] - The training plans for this project were very limited and didn't take account of the difficulty in operating a new payroll system. 
DDK
[NOTE: I have no affiliations to IBM/Queensland Government/SAP]

8 comments:

Jimmie Menon said...

It was like solving a mystery. Why did that big payroll project fail, I wonder if anyone has one idea like others. Everyone is exclaiming different probability as cause of such disaster. I loved the article. Thank you for writing it.

Regards,
Jimmie Menon
Payroll Providers Guelph

Plutus Payroll said...

Our Board is stacked with experienced industry professionals, who have come together to make the lives of Australian Business owners that little bit simpler, and to save them money along the way.
Contractor Payroll Services Australia

Affity Solutions said...

Thanks for your ideas. You can also find the details on Affity Solutions, at the software development company Australia. The main object of the Affity Solutions is to provide quality web services and is among the few software development company in Australia.

stephenn richardsonn said...

Thanks for sharing interesting post.


As a leading outsourcing payroll service provider London, we help you in opening bank accounts,Employing staff Surrey, International taxation advice & VAT services. Feel Free Try To Visit.

Stuart Spindlow said...

Thank you for such wonderful and interesting article. Please Visit:

accountants in barking
payroll services in London

Payslip 4 U said...

Fixed Price Payroll Service in the UK from just £1
Outsource Your Payroll Now and save money on your payroll

Payslip,
Electronic Payslips
P60
P45
PAYE Registration

Payslip 4 U is a online payroll bureau in UK

Zoya Khan said...

Payroll Service Provider is very important for small businesses today. The services provided by this service provider, generally ranges from processing of payroll works to managing the custody of the payroll of an employer. The processes that come within this segment are withholding, net paying and computing the employee’s salary.

Vxl Services said...

Great blog you've got here. It's packed full of lots of information!

payroll services new jersey
Cpa Consulting in new jersey