Computer Glitch brings Melbourne traffic to a standstill | Hacker News Reader

Computer Glitch brings Melbourne traffic to a standstill

news.com.au

53 pointsstratos238 comments

Traffic is at a standstill in Melbourne , Australia. A computer glitch has shut down the automated traffic signs, and in turn caused the closure of main tunnels and traffic routes. What's worse, they still don't know what is causing the software glitch!

Open articleView on HN

I've done work at CityLink, and their IT shop is just totally clueless. This does not surprise me one bit.

This is the same company that was doing "mirroring" with Veritas Volume Manager across a metro fibre link, rather than using their TrueCopy licenses. For no reason. Two fibre cuts rendered their entire production database (tolling) environment inoperable and nearly irrecoverable, on two separate occasions. I worked about 36 hours straight getting everything back online and was given a pair of movie tickets as a thank-you gift.

I've worked with State Transit in NSW (buses & ferries) and they were awful. The most embarrassing part was the UNIX team and Windows team refusing to talk to each other even though they were only one cubicle partition away from each other so any communication had to go up through two levels of management and back down.

The scariest part was every single database of user information was transformed from another database of user information; there was no source of truth, just a great big never ending circle of import & convert that had reached equilibrium and was now stable, provided nothing changed.

This is unfortunately normal in most public and corporate bodies. They are afraid that if they talk then information will filter across and they'll lose their remit or hold over the solution at hand.

It's silly really.

Me too, they are the most imbecilic of IT organizations. And to think its the safetymongers whom ruled the day...

You got movie tickets! A long time ago (13 years) the production database was irrecoverable! I got backups working just a week before tolling finally started. Failure of that system would ultimately result in the tollway becoming a freeway (allowing for buffering of data at the gantries, etc. etc.)

Although I had some responsibility for the tunnel safety systems at the time, we never touched them knowing that any problems would result in the road being closed.

From what I remember, the gantries were (at the time, a few years ago) able to store about 2 weeks' worth of traffic before tolls would effectively become free.

"... growing whisper within the security industry in Melbourne that there weak security might have been breached from China. ..." ~ http://news.ycombinator.com/item?id=4605335

Any idea what type of systems they are running?

In IT security here in Melbourne, first I've heard of it. Also why would network security specialists need to fly over 30 hours from Europe? We also have some pretty capable teams on the ground here already.

Given the time frame of when it happened it's more likely a change that has been borked in some way with difficulties in rolling back.

thx for the insight @bennysaurus. That's what I dislike about rumours, especially with so little info and in areas I lack understanding.

No trouble and agreed - rumours flying around like that don't help the problem. If there was that level of security breech, outsiders would probably do more than lock everybody out anyway.

Finding further information about it: looks like a core switch died with the backup not kicking in. 12 hours to work around is a but much though.

I'm having a really hard time seeing how this is any different than a traffic light being out. If "me-go-you-go" fails, why don't they just have a traffic officer out there directing traffic manually?

It's more like 20km of traffic lights being out. These roads are our core east-west arterial roads and the landscape dictates that there aren't many other roads for the traffic to travel on. I can't say why there aren't traffic police out there, but these are once-public roads that have been privatised. Also, these roads are highways, not traffic intersections.

Point of trivia: before the current toll road company was given its mandate, our main east-west bridge (the Westgate) held the distinction of being one of the very few toll roads in the world that stopped being tolled when planned, when the cost was paid off. A few years of free travel, then this company was given the rights to the road just beyond the bridge...

It's not traffic light out, it's traffic lights out and if there is a problem everyone will die because none of the safety system are working.

It is the only uninterrupted motorway from the east to west sides of Melbourne, but not the only core arterial. There are other roads you can use to get from one side of town to the other. A lot of traffic using the tunnels is just commuter traffic from the east anyway.

The tunnels never were public roads, they didn't exist. You can still drive on pretty much the entire length of the West Gate Freeway, right up to the tunnel portals without paying tolls. There are some short sections of motorway though that were previously un-tolled but are now tolled, though there has been some pretty extensive upgrades.

That said I'm no way in favour of City Link, but the real core 'arterial' for urban commuting is our public transport system. :-)

Fair enough. I'm in the north, and the old Tulla freeway was definitely free all the way in, and now it's tolled, so that may be colouring my memory.

The ABC has an article with a quote that explains it better (http://www.abc.net.au/news/2012-10-03/tunnel-closures-cause-...):

  CityLink spokeswoman Kate Murphy says the tunnels had to close as a precaution.

  "It's not just overhead signs [that are not working], it's also our safety systems
  as well," she said.

  "For example, in the tunnels the deluge systems or the smoke extraction systems
  which we may use in an emergency, we can't actually operate those as well, 
  which is why we've had to take the dramatic measure of actually closing the
  tunnels."

I had to hunt the article to find that, since the first impression is just speed limit signs/traffic lights being out which is not major. (and plenty of posts here suggest other people thought the same thing)

The variable speed signs always have extra signs along the lines of "(80) if variable limit sign is disabled" as a fallback.

Not being able to handle leaks, fume buildup, fires and other emergency conditions is a completely different story.

The fact that their safety systems (Which anyone would expect to be separate from the traffic control systems) are also affected makes me wonder about the "computer glitch" angle.

Hmm... actually yeah that is a really good point.

As much as I don't want to sound like a conspiracy person, that level of connection between those components is a bit too much to believe. Especially in Australia where they have insanely high standards for construction/electrical engineering.

Like 99% of news reported computere glitch's it is some human mess up that caused it, either by design or error or a homer moment were somebody pressed the wrong button. Could even be somebody overloading the electrical phases in the datacentre and tripped the UPS into action for it to barth at the imbalance (seen that one).

This is why they say computer glitch and count on the majority of non technical people to not question it.

> Like 99% of news reported computere glitch's it is some human mess up that caused it

If the system allowed the user to do something that impacted safetey in such a way, then the computer glitch is that the computer should stop that sort of thing happening. Human error happens a lot and you have to account for it in your designs.

Of course there are times when you have to trust the user because it is not practical to do those tests (in which case the human should have procedure to follow that mitigates the risk), and instances where the user wil override the checks (when the machines say "I really don't think you should do X because Y, click 'I understand the consequences of my propsosed action' to proceeed") - then it is genuinly human error is something goes wrong.

(Ignoring all the safety stuff in other replies)

These are tunnels over 3km long. You can't have cops down there without breathing apparatus - even if the extractors were safely operating.

Why would it be safe for people driving vehicles, but not for police officers?

In a vaguely sealed container with air being supplied through filters for a few minutes vs. standing in the fumes directly for hours.

They have fume extractor fans in the tunnels. If they aren't on, CO poisoning for everyone.

One would hope that the "fail safe" scenario for fume extractor fans would be "Blow at full speed"

Or, indeed, that they wouldn't be linked to the traffic light system in the first place.

"... I'm having a really hard time seeing how this is any different than a traffic light being out. ..."

There are reports in the local press of these kinds of problems: "CityLink says it closed the tunnels because the IT problem affected incident detection and safety systems that are activated in the case of an accident. The network problem also affected the back-up system." ~ http://www.heraldsun.com.au/news/breaking-news/melbourne-tun...

And references are being made about inability to communicate to remote devices via the network, "lost the ability to communicate" with its incident detection and safety systems." ~ http://www.itnews.com.au/News/317694,citylink-glitch-shuts-m...

See, this is why we need to hurry up and deploy networked computerized automobiles. We won't need stoplights anymore, so this kind of stuff won't happen!

We will still need safety systems in tunnels though, and lack of those is why they shut the tunnels down. Ignore the misleading quotes about the variable limit signs not working, the real issue is the tunnel is not safe if there is a problem like a fire of fume buildup.

I think he was being sarcastic, hinting that network computerized auto-mobile's having a similar glitch would be truly devastating.

If mankind can't get a tunnels systems to work all the time, then what about our plans to introduce computer controlled cars? A glitch could wipe out millions of lives before being patched, and even then if they get patched that also introduces huge areas of vulnerability.

I'm sure a stealth program that loads onto a car and waits till a date to kill the occupants will be used by someone or a nation-state in the future.

A bicycle would be good in such circumstances!

Only if you're nimble enough to dodge all the cars trying to run you down.

>I'm sure a stealth program that loads onto a car and waits till a date to kill the occupants will be used by someone or a nation-state in the future.

If nation states want that they can do it right now:

http://www.autosec.org/pubs/cars-oakland2010.pdf

http://www.autosec.org/pubs/cars-usenixsec2011.pdf

According to a post on the citylink twitter account, "the Burnley & Domain tunnels will begin the re-opening process @ 4pm"

https://twitter.com/774melbourne/status/253371401212403712

Apparently the reason the tunnels were closed was that Transurban could not operate the emergency sprinkler system and smoke extraction fans. Surely it would have been cheaper for Transurban to pay a few MFB fire trucks to be stationed outside the tunnels and close a lane to guarantee them access in case of an emergency. That way traffic could flow, tolls could be charged, and nobody would be as upset. I am sure Transurban make more than the fire brigade would charge them per hour.

Man that is going to be one epic lawsuit when you factor in the cost for each of those commuter hours.

Bring every police officer on-duty, making a fair percentage of them as a manual fallback to traffic direction. Have the officers play the part of the traffic lights.

Reply on news.ycombinator.com