Comment by mrfox321
1 year ago
At a big co I worked at, the lack of consistency between trading systems caused money to (dis)appear (into)out of thin air.
Prior to one of these hiccups, I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.
This led to an argument with my boss, who assumed things magically worked.
Days later, we received an email announcing an audit one one of these accounting discrepancies.
JPMC proposed using crypto, internally, to consistently manage cash flow.
Not sure if it went anywhere.
At all of the exchanges and trading firms I’ve worked with (granted none in crypto) one of the “must haves” has been a reconciliation system out of band of the trading platforms. In practice one of these almost always belongs to the risk group (this is usually dependent on drop copy), but the other is entirely based on pcaps at the point of contact with every counterparty and positions/trades reconstructed from there.
If any discrepancies are found that persist over some time horizon it can be cause to stop all activity.
Wait, pcap as in wireshark packet capture?
I'm not the commenter, but yes, often trading firms record all order gateway traffic to from brokers or exchanges at the TCP/IP packet level, in what are referred to as "pcap files". Awkwardly low-level to work with, but it means you know for sure what you sent, not what your software thought it was sending!
14 replies →
I would also really like to know that!
It generally seems to be a thing in trading: https://databento.com/pcaps
There is also this (though this page does not specify what pcap means): https://www.lseg.com/en/data-analytics/market-data/data-feed...
1 reply →
Typically not a literal pcap. Not just wireshsrk running persistently everywhere.
There are systems you can buy (eg by Pico) that you mirror all traffic to and they store it, index it, and have pre-configured parsers for a lot of protocols to make querying easier.
Think Splunk/ELK for network traffic by packet.
5 replies →
Looks like tnlnbn already answered, but the other benefit to having a raw network capture is often this is performed on devices (pico and exablaze just to name two) that provide very precise timestamping on a packet by packet basis, typically as some additional bytes prepended to the header.
Most modern trading systems performing competitive high frequency or event trades have performance thesholds in the tens of nanos, and the only place to land at that sort of precision is running analysis on a stable hardware clock.
I suppose Pre-Calculated Aggregated Positions, but I am not an expert in the field.
Looking at the order messages sent to and received from another trading system was not uncommon when I worked in that neck of the woods
The crypto firms are moving fast and breaking things. No need for that kind of safety shit, right? Would slow things down. Reminds me of Boeing.
So is this capture used to reconstruct FIX messages?
Yeah, FIX or whatever proprietary binary fixed-length protocols (OUCH or BOE for example) the venue uses for order instructions.
Some firms will also capture market data (ITCH, PITCH, Pillar Integrated) at the edge of the network at a few different cross connects to help evaluate performance of the exchange’s edge switches or core network.
Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow.
As a contractor, I helped do some auditing on one crypto exchange. At least they used a proper double-entry ledger for tracking internal transactions (built on top of an SQL database), so it stayed consistent with itself (though accounts would sometimes go negative, which was a problem).
The main problem is that the internal ledger simply wasn't reconciled with with the dozens of external blockchains, and problems crept in all the time.
> Fun fact, centralized crypto exchanges don't use crypto internally, it's simply too slow.
I know you're not arguing in their favor, just describing a reality, but the irony of that phrase is through the roof :-)))
Especially the "centralized crypto".
Yeah, that fact alone goes a long way to proving there is no technical merit to cryptocurrencies.
The reason they are now called "centralised crypto exchanges" is that "decentralised crypto exchanges" now exist, where trades do actually happen on a public blockchain. Though, a large chunk of those are "fake", where they look like a decentralised exchange, but there is a central entity holding all the coins in central wallets and can misplace them, or even reverse trades.
You kind of get the worst of both worlds, as you are now venerable to front-running, they are slow, and the exchange can still rug pull you.
The legit decentralised exchanges are limited to only trading tokens on a given blockchain (usually ethereum), are even slower, are still vulnerable to front-running. Plus, they spam those blockchains with loads of transactions, driving up transaction fees.
> JPMC proposed using crypto, internally, to consistently manage cash flow.
Yikes, how hard is it to just capture an immutable event log. Way cheaper than running crypto, even if only internally.
Harder than you'd think, given a couple of requirements, but there are off the shelf products like AWS's QLDB (and self hosted alternatives). They: Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log; then keep a searchable index on the log. So you can do bit-accurate audits going back to the first ledger entry if you want. No crypto, just common sense.
Oddly enough, I worked at a well known fintech where I advocated for this product. We were already all-in on AWS so another service was no biggie. The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements. In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
> The entrenched opinion was "just keep using Postgres" and that audits and immutability were not requirements.
If you're just using PG as a convenient abstraction for a write-only event log, I'm not completely opposed; you'd want some strong controls in place around ensuring the tables involved are indeed 'insert only' and have strong auditing around both any changes to that state as well as any attempts to change other state.
> In fact, editing ledger entries (!?!?!?) to fix mistakes was desirable.
But it -must- be write-only. If you really did have a bug fuck-up somewhere, you need a compensating event in the log to handle the fuck-up, and it better have some sort of explanation to go with it.
If it's a serialization issue, team better be figuring out how they failed to follow whatever schema evolution pattern you've done and have full coverage on. But if that got to PROD without being caught on something like a write-only ledger, you probably have bigger issues with your testing process.
Footnote to QLDB: AWS has deprecated QLDB[1]. They actually recommend using Postgres with pgAudit and a bunch of complexity around it[2]. I'm not sure how I feel about such a misunderstanding of one's own offerings of this level.
[1] https://docs.aws.amazon.com/qldb/latest/developerguide/what-...
[2] https://aws.amazon.com/blogs/database/replace-amazon-qldb-wi...
2 replies →
I'll just leave that here for no particular reason at all:
https://www.sec.gov/enforcement-litigation/whistleblower-pro...
2 replies →
> Merkle hash every entry with its predecessors; normalize entries so they can be consistently hashed and searched; store everything in an append-only log;
Isn’t this how crypto coins work under the hood? There’s no actual encryption in crypto, just secure hashing.
Theoretically they even have a better security environment (since it is internal and they control users, code base and network) so the consensus mechanism may not even require BFT.
It's all merkle trees under the hood. I feel like the crypto coin stuff has overshadowed the useful bits.
Is a Merkle tree needed or is good old basic double ledger accounting in a central database sufficient? If a key requirement is not a distributed ledger then it seems like a waste of time.
Merkle tree is to prevent tampering, not bad accounting practices
11 replies →
We launched Fireproof earlier this month on HN. It’s a tamperproof Merkle CRDT in TypeScript, with an object storage backend for portability.
See our Show HN: https://news.ycombinator.com/item?id=42184362
We’ve seen interest from trading groups for edge collaboration, so multi-user apps can run on-site without cloud latency.
8 replies →
Crypto/Blockchain makes it harder to have an incorrect state. If you fk up, you need to take down the whole operation and reverse everything back to the block in question. This ensures that everything was accounted for. On the other hand, if you fk in a traditional ledger system you might be tempted to keep things running and resolve "only" the affected accounts.
It's a question of business case. While ensuring you are always accounted correctly seems like a plus, if errors happen too often potentially due to volume, it makes more business sense sometimes to handle it while running rather than costing the business millions per minute having a pause.
It's mostly a different approach to "editing" a transaction.
With a blockchain, you simply go back, "fork", apply a fixed transaction, and replay all the rest. The difference is that you've got a ledger that's clearly a fork because of cryptographic signing.
With a traditional ledger, you fix the wrong transaction in place. You could also cryptographically sign them, and you could make those signatures depend on previous state, where you basically get two "blockchains".
Distributed trust mechanisms, usually used with crypto and blockchain, only matter when you want to keep the entire ledger public and decentralized (as in, allow untrusted parties to modify it).
6 replies →
If its for internal why not just use a normal append only log. x amount transferred from account y to account z. A three column csv oughta do it.
Any time your proposal entails a “why not just”, it is almost certainly underestimating the mental abilities of the people and teams who implemented it.
A good option is “what would happen if we” instead of anything involving the word “just”.
“Just” usually implies a lack of understanding of the problem space in question. If someone says “solution X” was considered because of these factors which lead to these tradeoffs however since then fundamental assumption Y has changed which allows this new solution then it’s very interesting.
1 reply →
Counterfactuals strike me as even less useful than underestimating competency would be. Surely basic double-entry accounting (necessarily implying the use of ledgers) should be considered table stakes for fintech competency.
Lots of threads on this here, most recently https://news.ycombinator.com/item?id=42038139#42038572 . I think this example is perfect, with the "oughta do it"
it literally ledger, its only show where money went but not showing "why" the money move
double entry with strong check that ensure its always balance fix this
https://news.ycombinator.com/item?id=42038139#42038572
> I hypothesized, given how shitty the codebase was, that they must be tracking this stuff poorly.
That is like half of the plot of Office Space
This sounds like a situation that I know about at the placed identified by name in your comment. It took months to track down the issue.