Photo: RNZ / Anneke Smith
The Electoral Commission has found a crash of its eRoll on Election Day was "ultimately avoidable".
The electronic version of the electoral roll went down shortly after 10am on election day 14 October, and was not fixed until just before 2pm.
While it only affected those casting special votes, many voting centres were already under strain with massive queues, and this caused further delays and confusion.
How the eRoll works
The eRoll is a mobile app developed exclusively for the Electoral Commission and was developed and maintained by IT company Catalyst.
On election day staff primarily used it to search the live electoral rolls for voters who did not have their EasyVote card with them.
"It is an electronic option to do what can also be done by looking up the hard copy rolls in the voting place for electors enrolled by writ day and looking up the index to places and streets to work out which electorate a person lives in."
What happened on Election Day
Documents released in an OIA to RNZ showed at 9:59am on election day four connections failed - although it was later suggested these were not related to the subsequent outage.
From 10am, intermittent requests began returning a 502 bad gateway error code, which Catalyst later said in its incident report suggested an issue with an upstream (back-end) server.
At 10.15am it said it noticed the network was slowing.
Teams messages among Electoral Commission staff at 10.18am and 10.20am then advised that the server had gone down.
At 10.22am a text message went out to 6648 voting place phones from the national office confirming that the eRoll was down.
Another message at 10.30am advised voting places to follow page 18 of the voter assistants manual, which outlined the process for managing queues manually.
At about 11am connections were exhausted and queued requests timed out returning a 504 Gateway Timeout error code.
Catalyst support engineers began to diagnose the potential root cause.
Between 11.20am and 11.35am two changes recommended by Catalyst to increase capacity were implemented.
At about 12.30pm Catalyst suggested the root cause may be a specific piece of code within the application and a recommended change to the code was made.
However at 1pm when the full load was restored, the issues returned.
A change to bypass the code section completely was made and proved successful.
At 1.44pm a final text was sent out to voting locations confirming the eRoll was working again.
Between 5:15pm and 5:40pm there was another spike in errors, which then dropped away and the system ran without incident for the rest of the day.
Could it have been avoided?
An incident report by the Electoral Commission found "the degradation in the eRoll application performance on election day, was ultimately avoidable, with clearer and more accurate requirements and thorough load testing that was more representative of the environment and context on the day".
The volumetric assumptions used for the 2023 General Election were based on the 2020 election. But enrolment activities on election day itself turned out to be much higher in 2023.
An email from Catalyst on the afternoon of polling day did, however, note "the root cause of the problem is behaviour that has been encoded in the system since it was first developed and released prior to 2020. It has worked without issue for that GE and in the early voting period".
The email noted that the issue would not have been easily identified in standard load tests.
The incident report also said officials could have reacted faster if they kept a closer eye on the system.
Delay caused voters to turn away
Official voter turnout was 78.2 percent, down from 82.2 percent in 2020.
While the eRoll issues only affected those casting special votes, many voting centres were already under strain with massive queues, and this caused further delays and confusion.
A voter in Wellington told RNZ on the day that some people had left before voting because of the delays.
An email to Electoral Commission staff also reported queues of up to 45 minutes in several electorates, with people reportedly leaving voting lines due to the wait.
Issues with the eRoll prior to election day
The Electoral Commission said its eRoll system was used without issue in the 2020 general election, two by-elections and throughout the advanced voting period of the 2023 general election.
It was also load-tested prior to the 2023 general election.
However, the documents released showed there were other technical issues with the eRoll system in August, September and October, ahead of election day.
"These were all resolved or identified as user error and not system issues and are unrelated to the incident on election day," the Electoral Commission said in its response.
Timeline of prior issues
Voting place staff flagged they were receiving the error message "connection timed out" when trying to use the eRoll. Staff were advised to use the manual roll until the issue was resolved.
An incident report from the Electoral Commission found it was unable to access key systems due to Catalyst failing to get notification of planned maintenance by an upstream provider. Catalyst remedied the issue.
An issue flagged specifically in the Geraldine Electorate that voters being enrolled are showing up on the special roll when they should show up on the general roll.
Service desk were advised that the general roll wasn't showing up on the app, when voting assistants tried to enrol voters, they showed up on the special roll when they should have been on the general roll.
Advice was given to ensure voting assistants were issuing ordinary votes.
An individual's voting place device was unable to accurately perform eRoll searches, showing all members of the public as being a part of the "special vote" group.
An issue of duplicate entries being able to be made to the eRoll was flagged.
A person tasked with fixing the problem wrote in a email that possible "connectivity issues", voter assistant impatience and the eRoll not preventing multiple submissions of the same transaction, could all be part of the problem.
But he also noted "we won't be able to fix eRoll until after the election given it is being heavily used at the moment and it doesn't look to be a simple fix".
"Hopefully we'll see a decrease in these over the next few days as the message gets disseminated. The good news is its only affecting about 3 percent of the updates coming through."
It was reported that users were having trouble logging into eRoll and were unable to select electorates.
Some voter assistants training in Selwyn were reported to be unable to update address details in the eRoll and were having trouble logging into the eRoll.
ERoll errors during training of voting assistants were reported to be impacting the ability to train.
Although not a technical issue - emails also showed that instructions had been left out from an initial course on how to use the eRoll.
During some change and testing of the eRoll in August, it was flagged that some more updating was needed, although what exactly was redacted.
- 11 October - Timeout issues
- 10 October - Issues in the Geraldine electorate
- 9/10 October - General roll not showing up
- 4 October - eRoll searches not working for one user
- 4/5 October - MDA2 Multiple Entries (MIKE)
- 25/26 September - Logging in issues
- 26 September - Issues with eRoll during training
- 18 September - eRoll training issues
- 14 September - missing instructions
- 23 August - eRoll URL change
Steps the Electoral Commission has taken following the election
In its incident report, the Electoral Commission said several meetings and workshops had taken place between the Electoral Commission and Catalyst to better understand the events of the day and to ensure lessons were learned.
In its response to RNZ, the Electoral Commission said "while the system issues were identified and remedied on election day, the Board of the Electoral Commission subsequently requested a full report on the causes and lessons learned in relation to both the performance of the eRoll Application and the management of the incident," the Electoral Commission said in its response to RNZ.
The board had commissioned the following actions:
- a review of the current software development lifecycle to ensure appropriate checks and controls are in place to capture all functional and non-functional requirements, to be completed by end of Q1 2024;
- a review of how load testing of key business applications is delivered in future, including the use of external third party specialist providers to be completed by the end of Q2 2024;
- a review of performance monitoring that is in place across Electoral Commission IT systems including monitoring by our third party providers, to be completed by the end of Q1 2024; and
- further embedding of the major IT incident management process introduced by the Electoral Commission and Catalyst during Q1 2024.
The Electoral Commission will include an update on this work in 2024 as part of its statutory report to the Minister of Justice on the general election.
Robust testing was also done ahead of the Port of Waikato by-election, but ultimately decided that the eRoll would not be used.
