Faster than a Flash

On May 6, 2010, stock prices in the United States jerked down and up with incredible speed. Within about five minutes, the Dow Jones Industrial Average plummeted about 600 points, only to regain most of that by a little later in the afternoon. Federal regulators later determined that this “flash crash” was triggered and then exacerbated by automated orders placed by mutual funds and other high-frequency traders. Stock exchanges had to cancel huge numbers of erroneous trades made during the crash.

Even before analysts began to understand what had gone wrong, some firms were able to stay out of the fray through real-time data analysis. For example, the high-frequency trading firm PhaseCapital, based in Boston, made no erroneous trades—an achievement that it credits to its use of complex event-processing software called StreamBase.

Market data comes from a wide variety of sources, such as the New York Stock Exchange, Nasdaq, and Reuters, and can sometimes be unreliable, explains Corwin Yu, director of electronic trading for PhaseCapital. For example, feeds can go down, or quotes can be incorrectly formatted or reflect unrealistically large changes. For companies to trade within seconds on such messy data, the information needs to be processed—in particular, scrutinized for potential errors before used to take any trading action. “It’s extremely fragile,” he says—even when there’s no crisis going on.

To deal with this information, PhaseCapital uses StreamBase, which is designed to accept large amounts of rapidly changing inputs and let organizations rapidly distill it into the insights they need to make decisions.

The typical way to deal with big data is by using databases. However, they aren’t good at processing data in real time; users have to wait until an entire data set has accumulated. StreamBase, however, can process a stream of data as it arrives, analyze it, make decisions about it, and take actions such as trading a stock or flagging a trend. “There’s a whole class of problems that are about real-time data analysis and real-time data processing,” says Richard Tibbetts, founder and CTO of StreamBase Systems.

The company offers its platform to customers, but, just as importantly, it gives them application programming interfaces that make it easy for them to develop their own software on top of it. That allows PhaseCapital (which won’t disclose how much it pays for StreamBase’s platform) to apply its own algorithms for scrubbing data and making trades.

Typically, PhaseCapital processes 30,000 to 40,000 “ticks,” or pieces of market data, per second. During the flash crash, that number jumped to more than 289,000 ticks per second—much of it representing stocks swinging wildly. “A lot of scrubs kicked in and realized that this data didn’t make sense,” Yu says. For example, some large publicly held companies, such as Accenture, traded for less than $1 during the flash crash. PhaseCapital continued trading, but filtered data that appeared suspicious. Because it didn’t act on that data, it was spared erroneous trades.

Even though many trades were later canceled, Eric Pritchett, PhaseCapital’s CEO, says there are several key reasons that it was important to avoid them anyway. For one thing, he says, it’s never certain what criteria regulators will use to cancel trades, so he wouldn’t want to have to rely on that mechanism. More importantly, he says, erroneous trades throw off a firm’s behavior for the rest of that day. For example, if a trade appears to have been profitable, algorithms may determine that the firm can afford to take riskier actions than it otherwise would. Pritchett says, “Not knowing where you really are with your book and your risk is the number one most dangerous thing that can happen to a trading firm.”

The example illustrates that anomalies in markets present both risks and opportunities, says Adam Honoré, who focuses on financial services technology as research director of the institutional securities practice for Aite Group. And the volume of data involved in trading is only going to increase, he notes.

Any time a company wants to analyze more data, there’s generally a price to pay—the extra processing reduces the speed at which a firm can take action. StreamBase has been a powerful tool in PhaseCapital’s eyes because it is structured so that new data can be filtered without causing significant performance hits, Yu says. “The trick to this game,” he says, “is to be fast and smart.”