t’s an annoyance for the individual computer user: You’ve updated your operating system, and now you need to reboot. This is so the computer can switch to the modified source code.
Imagine, however, having to update and reboot hundreds or thousands of computers operating in large companies and organizations: It can have a significant impact in lost time and money as computers and online services shut down, sometimes for hours. To avoid downtime, organizations will usually wait for low-traffic periods to update — but this can leave the servers outdated or vulnerable to cyber attacks.
In 2008, Jeff Arnold ’07, MEng ’08, along with a team of MIT computer scientists and engineers, began solving this issue by developing and commercializing software, called Ksplice, that automatically applies patches (security updates or bug fixes) to an operating system, on the fly, without requiring a reboot.
Based on Arnold’s award-winning MIT master’s thesis, the novel software compares changes between the old and updated code and implements those changes into a running Linux kernel — an operating system’s core data-processing component. In essence, it does something that could normally be achieved only by shutting down the operating system.
The software also incorporates novel techniques that remove the need for programmer intervention with the code (a trademark of performing updates without Ksplice), which decreases the cost and risk of error, Arnold says.
“The aim is to allow administrators the benefit of the update while eliminating both the cost and downtime for the users,” Arnold says.
After winning the 2009 MIT $100K Entrepreneurship Competition for the software, Arnold co-founded Ksplice, Inc. — with Waseem Daher ’07, MEng ’08, Tim Abbott ’07, SM ’08, and Anders Kaseorg ’08 — in Cambridge to launch it as a commercial product. Arnold served as the company’s CEO.
In just 18 months, Ksplice accumulated 700 customers — independent firms, government agencies, and Fortune 500 companies that were running the software on more than 100,000 servers.
Then, the startup sold for an undisclosed amount to technology giant Oracle, which is now providing the software to its Oracle Linux customers, which include banks, retail firms, and telecommunications companies worldwide. (After the purchase, the Ksplice team joined Oracle to help the company integrate the software in its products.)
As of today, Ksplice has only ever run on Linux operating systems. But Daher says the code is written in a way that should make it “potentially expandable to other products,” such as Mac and Windows operating systems.
Object focused
The process of updating running kernels is called “hot updating” or “hot patching,” and predates Ksplice. But Ksplice’s novelty is that it constructs hot patches using the object code — binary that a computer can understand — instead of the source code, computer instructions written and modified as text by a programmer (such as in C++ or Java).
Hot patching a program without Ksplice requires a programmer to construct replacement source code or manually inspect the code to create an update. Programmers might also need to resolve ambiguity in the code — say, choosing the correct location in computer memory when two or more software components have the same name.
Ksplice, however, hot patches the object code, using two novel techniques invented by Arnold. The first, called “pre-post differencing,” creates object code before a patch (“pre”) and object code modified by the patch (“post”) on the fly. It then compares the “pre” and “post” code to determine what code has been modified, extracts the changed code, and puts the code into its own updated object file, which it will plug into the running kernel. Essentially, it makes changes to functions modified by the patch, and points to relocated, updated versions of those functions.
The second technique, called “run-pre matching,” computes the address in computer memory of ambiguous code by using custom computation to compare the “pre” code with the finalized, running kernel (“run” code). For example, by going back and forth between the “pre” and “run” code, and examining the “pre” code’s metadata values, it gains enough information to identify the code that needs to be relocated to a spot in the running kernel.
This technique also doubles as a safety measure, as the process will abort the patch if it finds any unexpected object code differences between the “pre” code and “run” code.
Although today’s computer technology risks becoming obsolete within a few years, Ksplice (now 5 years old) is still a novel product, says Daher, who was Ksplice’s chief operations officer. “It’s still going strong,” he says.
The source of Ksplice
Ksplice’s roots trace back to 2006, when Arnold was charged with implementing a security update, for MIT’s Student Information Processing Board, that arrived on a weekday. Trying to avoid downtime while the servers were in heavy use, he delayed installing the update until the weekend. This wait, unfortunately, resulted in a cyber attack that required reinstalling all the system software.
“That’s what motivated me to think about this problem: You can’t bring servers down right away, and can’t wait until you have a chance to update, which can be a week, a month, or a year, in some companies,” Arnold says. “We needed better technology for solving this impossible problem for updating on the fly, as the system runs.”
Under the tutelage of Frans Kaashoek, the Charles A. Piper Professor of Computer Science and Engineering, Arnold started developing Ksplice for his graduate thesis, which earned second place for the Charles and Jennifer Johnson MEng Thesis Prize in 2008.
From there, grants and MIT’s $100K competition funded Ksplice, and Arnold gathered a team to help develop the technology. As the company grew, Arnold and Daher took the reins: They dealt with customers, sales, marketing, and accounting — “challenging for people with strictly computer science backgrounds,” Daher says.
For help, they turned to MIT’s Venture Mentoring Service (VMS), “which was helpful in mentoring, guiding us through applying for grants, and thinking about the business in the right way,” such as understanding the customers and the market, Daher says.
“Something that was key to our success was we had a good network of mentors and advisors, and many were part of the MIT community,” he says. “It was valuable to hear their war stories and get their take on some of the challenges they were facing.”
“MIT is very much part of the Ksplice story,” Arnold adds.
Arnold and Daher are now working on another software startup at the Cambridge Business Center — and still keep in touch with the VMS, they say. Being in the early stages, they can’t say much about the startup. “But we’re happy to be back building a startup,” Arnold says.
