Raft of problems with IT systems overhaul revealed

The investigation and subsequent reports were completed by Deloitte under a process that was subsequently found to be inadequate and as a consequence NZTA no longer stands by the findings of those reports — which have been removed from NZTA’s website.  NZTA has also apologised to the manager who was the subject of the reports.

Documents show big problems surrounding the overhaul of the country's over-taxed IT systems for managing highway crashes, snarl-ups and roadworks.

The Transport Agency ditched a multi-million dollar contract soon after signing it, then the alternative system failed at its first deployment.

The agency has identified its "ageing technology infrastructure and applications as a continued risk", in the face of an increasing volume of "unpredictable incidents" on the roads.

So it set out in 2016 to build a national incident and event management system, or Niems, to consolidate the way its three big-city operations centres (Auckland, Wellington and Christchurch) manage traffic using highway cameras and sensors. The plan was to extend this to all councils later.

The system would be key, for instance, to providing the right information to traffic light operators - to ensure they can phase the lights if there is a crash, to keep intersections safe and quickly and safely direct traffic elsewhere.

One goal of Niems is cutting the time to get things moving again - each minute is worth a couple of million dollars to the economy.

The first hurdle was when Auckland Transport pulled out of a joint Niems in 2016, to build its own incident management system using a different IT system.

But NZTA went ahead, undertaking what it called "an exhaustive and rigourous process" to find a supplier.

It signed up Castle Rock Associates in Oregon - a "clear winner", documents say - after sending five agency staff on a $50,000, three-city trip to the US to vet the company.

Some initial systems tests were being done by November 2016, and there were numerous meetings with police, fire and transport operations centre staff in preparation.

But six months later, the agency scrapped the Castle Rock contract entirely.

"With four months left to deliver, $3.3m of the total $5m budget has been used, and there has been slow delivery of a usable product to date," the agency's IT governance group was told on June 5, 2017.

This advice came in a short presentation from the director of the agency's business unit, who said an independent assessment had also faulted the project.

NZTA has not released this assessment to RNZ.

Sources close to the project say Castle Rock's product was not state-of-the-art and did not have a great user interface, but it was good enough for what operators, working under pressure and not so interested in interfaces, need.

The NZTA unit was later castigated within NZTA for having "duplicate functions, a lack of clarity in the long-term direction, no workplan and with funding all set up for the short term". It had 40 projects on the go when it was disbanded this year, with others having fallen by the wayside, according to an investigation.

It also had very loose financial controls.

The governance group endorsed an immediate pullout from Castle Rock - even though, the documents show, the group did not know the scope, cost or benefits of an alternative system promised by the business unit.

By December 2017, the governance group still did not know the nature of what it had approved six months before.

But by then the agency had already told the incoming Transport Minister Phil Twyford it would begin the Niems rollout by mid-2018.

Having dropped Castle Rock, the agency adapted the old system to work in the Cloud.

After a series of delays, it tried to roll it out for the first time, via the capital's transport operations centre on Wellington roads on June 5, 2018 - even though a severe weather warning was in place.

It was dealing with real-time incidents on the roads, but the documents suggest it was a mess.

The launch debrief reads: "Details not saving or saving in the wrong event"; "Overall system slowness"; "Event initiator unable to reopen event".

And because of poor planning, repairs were held up so problems went on from the Friday until the Monday.

"The developer tried reducing the number of servers but this caused the SCATS team to lose access to ILS for a time."

This meant teams operating traffic lights could not access the crucial Incident Logging System.

"With a severe weather warning in place for the next 24 hours, alternative options were considered."

It was deployed without the usual systems integration, performance testing, rollback or disaster recovery plans.

With incidents or events being duplicated or overwritten, if police needed any information for investigation then that would have been at risk.

This was meant to be the system's big launch, but they had to pull out.

A debrief indicates they tried again in late June 2018, with better results, though the system's technology remained of limited use.

It is not clear just where the promised national rollout of the Niems system has got to, nor its integration with roadwork planning systems Myworksites and Forward Works Viewer, or crucially, Auckland Transport's separate system.

NZTA has not disclosed to RNZ what it has written off from Castle Rock, or if it is carrying it on its books even though it is not a usable asset.

Figure 'incorrectly entered'

Its annual reviews show the contract was worth $4.5m.

But after RNZ began asking questions, the agency said that figure "was incorrectly entered" into parliamentary documents, and it was in fact worth $1.5m.

The agency also told RNZ in recent days that the whole Niems project budget was $2.4m.

Yet its own documents give a figure of $6m, and show that by mid-2017 it had already spent $3.3m.

The agency has repeatedly put the caveat "to the best of our knowledge" around its answers.

The new NZTA chairperson Sir Brian Roche said he was not aware of the Castle Rock matter, that the agency had been reviewed enough and he was focused on the future.

"I could spend the next two years redefining the problems that we've already sought to address or I could spend the next two years trying to create a future which creates a different environment," he said.

He suggested RNZ ask the Auditor-General. The AG said it did not audit the business unit in its 2017-18 audit of parts of NZTA, which pronounced its financial controls as "good".

In that audit, the AG said NZTA had "a 'keep the lights on' programme to remediate ageing infrastructure ... This is a multi-year/ongoing programme of work to ensure that ... systems remain fit for purpose. A more strategic/long-term approach has been developed to remediate ageing technology infrastructure".

Numerous reviews have been held into the agency this year.

The Minister Phil Twyford told RNZ last month he had received no advice about hi-tech write-offs at NZTA.

At the controls

A national incident and event management system would coordinate: