Background | Census Day Chronology
This is a case study of the Australian Bureau of Statistics’ 2016 online census project as a portrait of complete management failure by government, the ABS, and its principal contractor, IBM. The case study is split over several pages because of its length. The table of contents below offers quick navigation to the various sections. Scroll down to read the contents of this section.
The Census and Statistics Act 1905 gave the office of the Australian Statistician the power to collect statistical information, to conduct the census every five years, and to exercise care in the storage and release of information.
The Australian Bureau of Statistics Act 1975 established the Australian Bureau of Statistics (ABS) as a statutory authority, dealt with the appointment and removal of the Australian Statistician (the chief executive of the ABS), and created the Australian Statistics Advisory Council. The Council’s rôle in the online census project appears to be negligible.
The management and staffing of the bureau is conducted in accordance with the Public Service Act 1999.
The ABS has faced continuous budget cuts since 2007, justified by successive governments as measures to force greater efficiency.
The bureau responded with both staff attrition and a progressive reduction in the range of data being collected and analysed for use by policy developers and researchers.
ABS figures say that the 2011 census cost around $440 million, or about $19 per person about whom data was collected. The biggest cost was around $160 million in wages for 43,000 temporary employees responsible for delivering and collecting census forms.
The ABS considered introducing an online option for the 2016 census as an opportunity to save on the cost of manual data collection. It was adopted only after the Commonwealth Government of Australia rejected an option to extend the census time-line from once every five years to once every decade. Political resistance to extending census time-frames or reducing its scope is based largely on its importance in re-drawing electorate boundaries and parliamentary representation tied to population changes. Census figures are also important in assisting with calculating the share of GST revenue apportioned to the states and territories.
In October 2014, the ABS awarded a $9.6 million contract to IBM to design, build, and manage an online census system, with a further $415,000 contract for load testing awarded to Australian IT company Revolution IT. The ABS and IBM reportedly rejected the provision of special denial of service attack protections from their upstream cloud services provider, NextGen Networks.
A week before census day, Australian Statistician and ABS chief executive David Kalisch was interviewed on the Australian Broadcasting Corporation’s ‘7.30’ programme. Asked about the bureau’s readiness for the census, about undelivered online access codes and census forms, and about lengthy delays when using the ABS telephone help line, Kalisch responded by stating the ABS was better placed than it had been five years earlier, and that any delays in receiving mail or getting telephone answers were temporary inconveniences that would be addressed well within acceptable time-frames.
On 18 December 2015 the ABS issued a media release announcing that it would depart from customary practice of destroying records of names and addresses collected by the census after 18 months. As of the 2016 census it would instead maintain these records for four years. The press release included the following:
The decision follows a Privacy Impact Assessment which found that retaining names and addresses for the purpose of richer and more dynamic statistics and more efficient statistical operations has very low risks to privacy, confidentiality and security.
The Assessment concluded that the ABS is able to safely manage this data through extremely robust, best-practice data management and information security practices. These practices support the ABS’s ongoing commitment to maintaining community trust and protecting the confidentiality of individuals and businesses, as required by legislation.
In March 2016 the ABS issued a media release declaring that the decision to retain names and addresses had followed a ‘long, transparent public consultation and stakeholder engagement process. We invited public submissions and consulted directly with the Australian Privacy Commissioner, and State and Territory Privacy Commissioners.’
In May 2016 the Canberra Times reported that a former ABS employee called into question the veracity of privacy assurances and that he would not trust his former employer:
If the personal identifier address data-sets are to be over-written or replaced by data-sets from the 2020 census and so on, then to claim the retention of the 2016 data as only temporary is in fact a load of rubbish as it would have become continuing, updated data-sets,” Mr Hamilton said.
The former ABS-staffer added that the bureau’s hard-won reputation for excellence had taken a battering in recent years.
“Several quarters ago, the ABS had to admit that it was using inappropriate seasonality factors in its labour data, making its seasonal data frankly useless,” he said.
“The ABS that I knew and worked in would not have gone down that path, producing gibberish data via use of outdated methodology.
“Something has gone seriously wrong.
“This pattern of ABS behaviour which obviously comes from the highest levels of the ABS is such that I can no longer believe the ABS may be trusted with data such as retention of household personal identifiers.
“Unless the ABS is able to provide appropriate detail which shall allay my concerns, I cannot in good conscience participate in the 2016 Census of Population and Housing. I am aware of the potential consequences and frankly would love the opportunity to set out these facts before a magistrate.”
In July 2016, Sydney Morning Herald Economics Editor Peter Martin wrote a comment condemning the decision to retain name and address data for an extended period. His editorial included the following paragraph:
While there was little immediate public reaction to the decision to maintain the name and address data, a decision by several senators (the Greens, Nick Xenophon, Jacqui Lambie) to publicly declare they would risk fines by not providing name and address details on their own census forms created significant press coverage and social media dialogue.
Public comment focused on the bureau’s claims that adequate data privacy and security practices and technology were in place, and on the legality of demanding name and address details as mandatory rather than voluntary.
The sticking point is that a unique database key was developed to link name and address details to matching census data, and that this key is not to be destroyed at all. Using a database key to identify linked sets of information makes it possible to maintain long-term data-matching capability between a names and address data set and other census details.
The day after the Peter Martin editorial, ABS chief David Kalisch had his own comment published in the Sydney Morning Herald. Explaining the data retention decision, he wrote:
My decision followed community consultation, direct engagement with the Australian Information Commissioner and each State and Territory Privacy Commissioner, and a Privacy Impact Assessment (PIA). The ABS has transparently communicated its process and decisions every step of the way. We advertised our PIA process in the national media in November 2015 and received few responses.
Kalisch stated publicly that there had never been a security breach relating to census information, and hinted that the senators might be prosecuted for withholding name and address details.
Kalisch emphasised that names had always been collected in previous censuses. Responding to questions, he elaborated that in 2011 around 100 individuals had been fined, and that it was up to the Director of Public Prosecutions whether people withholding information would be summonsed.
He rejected the public statement by former ABS chief Bill McLennan that ‘compelling’ citizens to give their names on census forms was unconstitutional, arguing instead that legal advice to the ABS by the Australian Government Solicitor declared its actions lawful.
The Prime Minister responded to privacy concerns with a public statement that the ‘security’ of ‘personal details is absolute and that is protected by law and by practice. That is a given.’ Nationals MP Michael McCormack, promoted one month before the census to the small business ministry, and handed responsibility for the ABS two weeks from the census date, responded to the senators’ public statements by urging them to comply with the ABS demands, and proposing that the privacy risk was no bigger than belonging to a supermarket loyalty scheme or using a Facebook account.
Royal Melbourne Institute of Technology Senior Lecturer Dr Mark Gregory said that neither public nor private sector organisations could guarantee the security of the information, and there are no mandatory data breach reporting laws to back security assurances.
The ABS census website carried the following information on census night:
Are names and addresses compulsory in the Census?
Names and addresses have been collected in every Census.
Names and addresses are specified in the Census Regulations as Statistical Information, like all other Census topics. This requires the ABS to collect this information as part of the Census. The requirement for all topics, including names and address, on the Census forms to be filled completely and accurately is consistent with 105 years of Australian Census practice, the Census and Statistics Act 1905 and legal advice to the ABS from the Australian Government Solicitor. The only exception is religion, which the legislation specifies is optional.
However, this statement was removed from the site after 9 August 2016, possibly indicating it was incorrect.
Census Day Chronology
At just after ten on the morning of census day, 9 August 2016, the census website experienced unusually high traffic for 11 minutes, causing a five minute outage. ABS and IBM began investigating the cause.
A second surge in traffic was experienced at 11:46, leading to the activation of a denial of service attack strategy based on ‘geo-blocking’, or blocking all international internet traffic from reaching the census site.
By this time the ABS and IBM reportedly reached the conclusion that the surge in traffic was caused by a distributed denial of service attack (DDoS). This type of attack involves using public Domain Name Servers to mask the source of high volumes of UDP packets posing as normal network traffic, seeking out web addresses, but really designed to flood the victim’s server response capabilities, thus making them unable to process any requests at all.
At 11:55 the ABS sought advice from Australian Defence Signals, the nation’s military intelligence agency, on avoiding further attacks of this kind. Possible sources of the attacks were considered to be Russian or Chinese, and the ABS made a decision to block all international traffic to the site for the rest of the day.
Further suspected attacks at 16:58 and 18:15 were deflected by firewalls and unspecified other measures. Another attack at 19:30, apparently from within Australia, plus significant numbers of Australians logging in to complete the online census, caused the collapse of the geo-blocking safeguard, flooding the firewall router state tables; this means that so much information was being directed at the router that it could no longer respond at all. This led to a re-boot of the census website firewall (operating as a pair of routers), after which it was discovered that only one of the firewalls had been properly configured, rendering the pair ineffective in dealing with a heavy load, and leading to the collapse of the firewall.
At 19:45 the ABS took the online census offline while the ABS briefed the government. At 20:50 the system was restored for testing, but remained inaccessible to the public until the next day.
Australian IT blogger Patrick Gray suggests that IBM monitoring systems reported data interpreted to mean that the DDoS attack was cover for an attempt at data exfiltration (theft), leading to a decision to leave the system unavailable to the public.
There has been no confirmation or denial from the ABS or IBM about the precise details of emergency planning.
Subsequent commentary has cast doubt on whether any DDoS attack took place at all rather than outages being caused by misconfigured system components and misguided decision-making.
Prime Minister Malcolm Turnbull, reacted to the online census system outage by suggesting the ABS and IBM were to blame, and that ‘heads will roll’ following a review by his own IT advisor.
ABS chief statistician between 1995-2000, Bill McLennan, commented that the responsibility for the census rested solely with the chief statistician, and that his job included delivering required outputs regardless of budget cuts. However, former Secretary of the Department of Prime Minister and Cabinet, Terry Moran, suggested that successive governments must be blamed for foolishly cutting budget and enforcing outsourcing of ABS functions.
Aftermath: Make Census Great Again
On the weekend of 13-14 August, two undergraduate IT students at the Queensland University of Technology participated in a Code Network ‘hackathon’, using open source software and the Amazon cloud service to build and test Make Census Great Again – an online census solution they say was tested at 4 million page views per hour and 10,000 submissions per second, far outstripping Revolution IT’s testing for the ABS system of one million page views per hour and 260 submissions per second.
Online men’s lifestyle magazine EFTM suggested that hackathon participants Austin Wilshire and Bernd Harzer had therefore developed a better solution for $500 and pizza than the ABS was able to source for $10 million.