Comment by HolyLampshade
1 year ago
At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there.
If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.
Wait, pcap as in wireshark packet capture?
I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!
The ultimate source of truth about what orders you sent to the exchange is the exact set of bits sent to the exchange. This is very important because your software can have bugs (and so can theirs), so using the packet captures from that wire directly is the only real way to know what really happened.
5 replies →
The result of bitter lessons learnt I'm sure. Lessons the fintechs have not learned.
That makes sense - but it's still somewhat surprising that there's nothing better. I guess that's the equivalent of the modern paper trail.
2 replies →
The storage requirements of this must be impressive
3 replies →
I would also really like to know that!
It generally seems to be a thing in trading: https://databento.com/pcaps
There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feed...
Look up Corvil devices by Pico.
Commonly used in finance.
https://www.pico.net/corvil-analytics/
Typically not a literal pcap. Not just wireshsrk running persistently everywhere.
There are systems you can buy (eg by Pico) that you mirror all traffic to and they store it, index it, and have pre-configured parsers for a lot of protocols to make querying easier.
Think Splunk/ELK for network traffic by packet.
Except it is literal “pcap” as they capture all packets at layer 3. I don’t know the exact specifications of Pico appliances, but it would not surprise me they’re running Linux + libpcap + some sort of timeseries DB
2 replies →
The Corvil devices used by Pico have IME largely been replaced by Arista 7130 Metamux platforms at the capture “edge”
1 reply →
Looks like tnlnbn already answered, but the other benefit to having a raw network capture is often this is performed on devices (pico and exablaze just to name two) that provide very precise timestamping on a packet by packet basis, typically as some additional bytes prepended to the header.
Most modern trading systems performing competitive high frequency or event trades have performance thesholds in the tens of nanos, and the only place to land at that sort of precision is running analysis on a stable hardware clock.
I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.
Looking at the order messages sent to and received from another trading system was not uncommon when I worked in that neck of the woods
The crypto firms are moving fast and breaking things. No need for that kind of safety shit, right? Would slow things down. Reminds me of Boeing.
So is this capture used to reconstruct FIX messages?
Yeah, FIX or whatever proprietary binary fixed-length protocols (OUCH or BOE for example) the venue uses for order instructions.
Some firms will also capture market data (ITCH, PITCH, Pillar Integrated) at the edge of the network at a few different cross connects to help evaluate performance of the exchange’s edge switches or core network.