LAS VEGAS–The idea for a certification and testing lab for the security of software products has been kicking around the technology industry for a long time. But no one has really figured out a good model or methodology for doing it, until now.
Peiter Zatko, a longtime security researcher known as Mudge, has developed system that enables his organization, called the Cyber Independent Testing Lab, to measure the relative security and difficulty of exploitation for various applications. The data that Zatko has collected from the first few months of testing shows that there can be a wide variance between the perception of a product’s security and the reality.
The CITL, which is a non-profit independent organization with funding from DARPA, uses a variety if methods to assess the security of the products it test, and the result of each test is a scorecard that’s somewhat akin to the nutrition label found on food products. CITL only tests binaries, not source code, and uses fuzzers to look for potentially vulnerable code. Zatko, who is running the project with his wife Sarah, said the idea is to give software buyers a method for assessing which applications are the most resistant to attack. The scorecard that results from each test grades the target application on a number of things, including exploitability and potential disruption to a business.
“We as security practitioners tend to focus on exploitability. But as a consumer of a product, they’re almost always going to say disruptability is what bothers them,” Zatko said during a talk at the Black Hat conference here Wednesday. “If you’re running an offshore oil rig and one of your machines is being used to host warez or hide files, you’re not going to be as concerned as if something is exploited and it takes the rig offline for a year. Ask someone in a bank the same question and they’ll say exploitability, the integrity of the data, is what’s important. Because if they start propagating bad numbers out, it’s nearly impossible to get back to what the correct data is.”
Modeled after Consumer Reports’ work, CITL’s methodology includes looking at things such as known bad functions and how often a particular application uses them. It also looks for functions that are considered safe and compares how often the app uses good versus bad ones. Some of the testing that the CITL has done so far has focused on browsers, looking at the relative safety of each of the major ones in terms of the functions they use as well as the exploit mitigations they employ.
“Chrome has more good functions than the other browsers, but it almost always has bad functions too,” Sarah Zatko said. “There’s no consistent use of good functions, which partially defeats the purpose.”
Few people have been involved in the security community for longer or done a wider variety of things than Peiter Zatko. A founding member of the L0pht hacking collective, he later worked for a variety of technology companies, including BBN and Google. He also worked at DARPA for several years, working on the CINDER (Cyber Insider Threat) program and helping to fund a wide range of security research projects. He left Google last year and started the CITL with help from DARPA, something that he’s been trying to do since his days at the L0pht in the 1990s.
The goal for the CITL is to give buyers data on which apps are the most difficult for attackers to compromise. By the end of the year, the organization plans to release its methodology for static analysis of binaries and will allow access to its back-end testing system next year. CITL also will release large-scale fuzzing results by the end of 2017, he said.
“Now you’ve got choices for just about everything you buy. Choose the application that imposes more work for the adversary and doesn’t create more work for you,” Peiter Zatko said.
