A MAJOR failure of the Census servers is simple: the numbers used to design the system just don’t add up.
Yesterday Prime Minister Malcolm Turnbull described a denial of service attack as predictable as “the sun will come up in the morning”. The Census server was only restored about 3pm, although many people reported it was still down for them.
Users are currently reporting patchy connection issues when accessing the website from their computers and mobile phones.
This is the third census that the Australian Bureau of Statistics has worked with IBM to collect census figures electronically and, when you crunch the numbers, the design of the servers appear to make some basic miscalculations.
In the 2011 census, the ABS reported 2.6 million households filed an eCensus form through a system, hosted at IBM’s servers at Baulkum Hills, Sydney, designed to handle a peak of 120 submissions a second.
This year, the ABS predicted more than two thirds of the 10 million households, or 15 million people, would file their census electronically on a system, again hosted by IBM, designed to handle 260 submissions a second.
The servers that went down on Tuesday night were designed to handle about twice the load a second than the system built five years but expecting about three times the traffic.
That scaling up seems close but not when you take in the other crucial factor in the equation: time.
In 2011, the ABS had a marketing campaign that spread the load.
“Although census night was Tuesday, 9 August 2011, the eCensus was open from Thursday, 28 July 2011 until Monday, 5 September 2011 to allow for a higher response rate,” the ABS says in its analysis of that year’s collection.
“Respondents were able to access the eCensus at any time during this period to complete and submit their form.”
In the 2011 census, Australians were encouraged to fill out their forms in advance and come back later either on or after census night and hit submit.
At its peak, in 8.43pm on 9 August 2011, the IBM servers handled 85 eCensus submissions, way below the load the system was designed to cope with.
Patrick Gray, who hosts the Risky Business podcast, yesterday reported IBM declined an offer for extra defences against a denial of service attack.
While nothing compares to the failure of this year’s census night, the history of electronic census collection in Australia has had some significant snafus.
In the lead up to 2011, the ABS issued a heads up to Australia’s internet service providers to expect a traffic spike on the night because on the night of the 2006 census, some misread the extra traffic of census filing as a distributed denial of service attack and throttled their network speeds to cope with what they perceived as an attack.
Then in 2011, the ABS had to remind 679,000 Australians to hit the submit button on their census forms, after they preloaded it with their responses for the Tuesday night and then forget to come back for the final stage.
The push to electronic submission of census papers reflects both an acknowledgment of the simplicity of the gathering process and the huge cost savings involved in cutting the number of field officers who did the old-school method of walking door to door.
After the 2011 census, the ABS then predicted it could save up to $80 million on field costs by switching to electronic submission.
The savings are more than just through manpower cuts. The ABS said this year it produced 327 tonnes of less material than in 2011, and the distance census staffed covered was cut by 230,000km as they delivered 21,151 few cartons of materials.
In 2010, the ABS announced it would end its arrangement with IBM after the 2011 census, instead switching to hosting it internally on the ABS’s Canberra servers which would alleviate fears of possible privacy breaches by hosting the data in-house.
But then it reversed its decision, and in 2014 it award IBM $9.6 million the contract for a third electronic census.
“Due to the peak volume of the online form during Census 2016 it was decided that contracting IBM would provide the best value for money and management of operational risk,” the bureau said at the time.
Australia is not the only country that has increasingly relied on the web for census data collection.
Since 2010, at least 32 countries have conducted a census at least partly online, with that figure set to rise sharply as more countries hold a census in the next few years.
The experience shows that people increasingly choose to submit their census electronically when given the option.
Back in 2000, when only 32 per cent of the population had the internet at home in Singapore, 15 per cent of census returns were done through the internet. In Portugal in 2011, half of the citizens submitted their census forms online.
The United States was a pioneer in online census collection, offering a small trial in 2000 although it only recorded 71,000 responses because it chose not to widely publicise the trial so that it was not swamped with electronic submissions.
The US dropped the online form submission in 2010, although it heavily used the internet to provide census information that year. It has held other population surveys including the Economic Census online since 2002. The US Census Bureau estimates half of the population will file their 2020 census forms through the internet.
Online Source
