Networking and high-frequency trading

The high-frequency-trading (HFT) industry is rather tight-lipped about what it does and how it does it, but PJ Waskiewicz of Jump Trading came to the Netdev 0x16 conference to try to demystify some of that, especially with respect to its use of networking. He wanted to contrast the needs of HFT with those of the traditional networking as it is used outside of the HFT space. He also has some thoughts on what the Linux kernel could do to help address those needs so that HFT companies could move away from some of the custom code that is currently being developed and maintained by multiple firms in the industry.

Secrets

Waskiewicz began by highlighting the secretive nature of the industry; it is sometimes amusing, but also rather frustrating, how little is known about HFT. For example, there is almost nothing on his company's web site beyond some office-location information; that can make recruiting difficult, for example. It is well-known that HFT companies do trading—stocks, options, securities, etc.—but the "how" is the secretive part; how do the companies decide on the trades to execute and how do they actually get executed? That is the secret sauce that HFT companies do not want to share with anyone—especially their HFT competitors.

He said that a Wikipedia definition describes HFT as "algorithmic-based trading"—data is analyzed in various ways in order to decide whether or not to execute a trade. Those trades are then made in "very very high volumes". While preparing for the talk, he found a study that attributes 60% of all of the trades in all of the exchanges worldwide to HFT firms. All of that volume is coming from algorithmic trading.

The trading strategies that determine what to trade and how to trade it are based on quantitative analysis. So HFT firms have teams of people looking at large amounts of data, extracting signals from the data, and using it to build models. The goal is to predict the future and then to automatically execute trades to take advantage of those predictions. The strategy algorithms can be implemented in either software or hardware; it is no secret, he said, that HFT firms use custom hardware to help accelerate their operations.

HFT firms have massive amounts of data stored that can be used by the teams to create their models. Market data for the last ten years or more amounts to petabytes of storage required. Moving that data around efficiently is important, but the primary concern for HFT networking is to have predictable latency; Waskiewicz had already mentioned that network jitter can cause HFT firms to lose a lot of money and he would return to that idea several times in the talk. Unexpected latency can change the timing of queries and actions so that the strategy is no longer doing what it is trying to accomplish.

The communication between HFT companies and the exchanges is subject to various differences between the exchanges and the protocols that they use. Exchanges, such as Nasdaq, Eurex, and KRX, each publish their own specifications of the protocols that can be used to do electronic trading. The specifications cover packet formats, how to query information, incoming and outgoing packet rates, and so on; each exchanges has its own nuances—and quirks. The exchanges generally run on standard 10Gbps Ethernet; there is no movement toward 100Gbps Ethernet that he knows of, though there is some talk about 25Gbps.

Inside the HFT firm, there is a need for high-performance computing (HPC) facilities to do the quantitative analysis. Those HPC environments require grid networks with lots of distributed CPU horsepower and storage, he said. Remote DMA (RDMA) is used on these internal networks, but predictable latency is still the main concern.

Latency

Out of the box, the kernel networking stack has poor performance with regard to latency, he said, though it can be improved with some tuning. Techniques like pinning workloads to particular CPUs, keeping NUMA locality in mind, and using interrupt affinity are generally well-known for reducing packet jitter. CPU isolation is perhaps a lesser-known feature that the HFT world uses to reduce the jitter even further; CPUs are isolated from the rest of the system and workloads are pinned to them in order to reduce or eliminate the jitter.

He put up some graphs from a simple benchmark that he did to show the effects of these techniques; it used netperf to measure request-response latency on a 10Gbps Ethernet network with a switch. As expected, the numbers generally got better for the minimum and mean latencies as each technique was applied; the values were reported as an average of ten runs of the benchmark. The unoptimized values were a minimum of 51.6µs and a mean of 68.7µs, which he said "wasn't terrible" though the maximum latencies were 250-600µs, thus "a bit of a mess".

He then showed the results for pinning the CPU with no interrupt-affinity change, which showed a small improvement (50.1/67.6). When he added interrupt affinity into the mix, so that the cache locality came into play, there was a more noticeable boost (45.4/53.1); "we're starting to get to this point of less jitter, which is the important part". He expected that isolating the CPU would make things better still, but was surprised to see the numbers get worse (47.8/61.1). He thought about that and realized that the interrupt was interfering, so he ran the benchmark without interrupts by putting the driver into polling mode. That was more in line with expectations with a 41.9µs minimum and a 56.3µs average latency.

But that was "a very synthetic benchmark", where he could mold the system and the application specifically to his needs; it does not really match the real world of traditional networking at all. In that world, there are other workloads that also need to be run so things cannot be statically partitioned as he was doing; but in the HFT networking environment, none of that matters. The synthetic-benchmark environment is what is used; a system where "everything is perfectly lined up is actually how things get deployed" for HFT.

Options

So he wondered if he could use express data path (XDP or, commonly, AF_XDP) as a way to improve things further. "Because if we throw eBPF at any problem that'll just fix it, right?", he said to some scattered laughter. While XDP is "not here yet" for HFT, he thinks it could be the right path someday and has some ideas on how to get there.

XDP allows for kernel bypass without actually bypassing the kernel, Waskiewicz said, which is really compelling. His vision is that the "hot path" data that is extremely latency-sensitive could be identified by the application and those packets would go directly to it, while the other traffic would continue to be handled by the kernel networking stack. "That's like the best of both worlds as far as I am concerned." There are some limitations that need to be dealt with (or worked around). The receive side must be done with polling since interrupts introduce jitter by their very nature. As far as he is aware, transmitting data requires making a system call and the context switches for system calls introduce jitter as well.

CPU isolation works well, he said, until it does not. His firm uses the isolcpus boot parameter to choose the set of isolated CPUs, but there are still some "random" inter-processor interrupts (IPIs) that occur, which are "fairly infuriating". At this year's Linux Plumbers Conference, there was a microconference on CPU isolation where the problems he has been seeing were discussed.

In some configurations, simply connecting to a CPU-isolated system using SSH will cause a cascade of events that eventually result in "TLB [translation lookaside buffer] shootdowns issued to every core on the system". Those IPIs cause jitter, but refilling the TLB causes jitter as well. He is trying to carve out some time to address that problem. Another CPU-isolation problem that he encountered was that an IPI is sent to all processors when someone executed "cat /proc/cpuinfo"; the system does that to get the operating frequency of each core. This was particularly a problem for systems that ran some kind of telemetry application that would check those values frequently. The bug has now been fixed upstream in work that his company did in conjunction with Red Hat, he said.

HPC side

As noted, the connection to the exchanges requires standard Ethernet, but the internal HPC grid, where there can be tens to hundreds of thousands of CPUs, can be (and is) rather more exotic. It turns out that HPC in HFT has been something of a niche market for RDMA. Quantitative analysis requires moving lots of data around and operating on it in parallel throughout the network.

Predictable latency is also important on the analysis side of the network, and RDMA works well for that, but there are some things that he thinks could be done better. For one, io_uring is showing great promise; it has expanded quite a ways from its initial genesis as a replacement for the libaio asynchronous I/O library. It is no longer only for I/O, as Josh Triplett's io_uring_spawn talk at LPC shows, Waskiewicz said. It would be interesting to see if the networking-hardware ring buffers could be used directly as buffers for io_uring operations so that data can get to and from the hardware using that mechanism; that would allow the HPC side to use something "much more kernel-standardized that would be able to replace RDMA".

But, for now at least, RDMA is the king in HPC networks; no mention of RDMA is complete without mention of RDMA over Converged Ethernet (RoCE), though, he said. Infiniband, which is the fabric used for these RDMA networks, is expensive, but that is not necessarily a problem in the HFT world as the industry is willing to spend money that allows it to make more money. Infiniband is something of a niche technology, though, which makes it hard to find technical people that can manage the network and keep it up and running.

RoCE (along with iWARP) allow the use of Ethernet equipment and management skills, but they come with challenges of their own. Converged networks still have jitter problems because they are not a dedicated fabric. That leads to a need for additional equipment and configuration to reduce that.

He said that he already had planned to talk about Homa for HPC before the John Ousterhout's keynote the previous day (which we covered: part 1 and part 2). Waskiewicz sees the remote-procedure-call-based approach of Homa as being similar to the RDMA Verbs API. Having Homa available both in user space and the kernel would allow for more flexibility. Being able to use standard Ethernet equipment throughout the network would be worthwhile from a maintenance and cost standpoint as well.

He wondered if there are other possibilities that the HFT industry should be looking at. If so, they must eliminate jitter, as mentioned multiple times, but they must also be low latency. If the latency is 100µs, even without any jitter, it still cannot be used for HFT because "it will lose every time".

Waskiewicz was running out of time at that point so he quickly reiterated his main points from the talk. Not surprisingly, jitter was the centerpiece; it is important to ensure that the algorithms can get the predictable latency that they need because the exchanges themselves can be damaged "when algorithmic trading goes haywire". It is not hard to find instances where this kind of trading has caused problems that made exchanges hit their circuit-breakers—or worse. He is pleased to see that there are various efforts to attack the jitter problem underway at this point.