The addition of a ‘leap second’ to universal clock time on Saturday caused havoc on the web over the weekend – despite the fact that there was plenty of warning.
Sites including Reddit, LinkedIn and Mozilla crashed or experienced other technical problems.
Since 1971, the leap second has been added to keep the world’s clocks in synch with the world’s rotation. The ITU Radiocommunication Assembly recently resolved to keep on using it for another three years, despite concerns from the US that it might cause difficulties for computer communications.
And it seems the US was right to worry.
“We are having some Java/Cassandra issues related to the leap second at 5pm PST. We’re working as quickly as we can to restore service,” warned Reddit in a tweet on Saturday.
For most sites, the problem originated in a subsystem of the Linux kernel called hr timer, or high-res timer, which was unablee to handle the extra second and flooded CPUs with requests as a result. While the flaw was actually patched four months ago, not all versions of Linux had been updated.
Reddit says it also had problems with its Cassandra database, but that rebooting servers broght it back online within a coupole of hours.
Other companies that experienced problems were also able to fix matters relatively quickly. Airline reservation system Amadeus Altea went down, but was restored after an hour in which staff at Qantas were forced to check in passengers manually.
Some companies, though, saw the problem coming; Google, for example, handled it by adding the extra second in several small increments throughout the day.
And there was plenty of warning: senior system administrator at Opera Marco Marongiu wrote a detailed description of how the problem could be averted a month ago.
“After 40 years from the first leap second, it is still managed by many by a step back of one second; others do it their own way (as Windows, unsurprisingly); others had to invent yet other ways to avoid the traps of bad kernel implementations (ask Google),” he wrote.
“Yet, I don’t think it’s likely that Linux kernel developers (or developers of other systems and software, for that matter) will put much effort in handling the next leap second right. Brace yourself, and good luck.”