Comment by i_like_waiting
5 years ago
But can you stream data from lets say MS SQL directly into Postgres? Easiest way I found is Kafka, I would love some simple python script instead
5 years ago
But can you stream data from lets say MS SQL directly into Postgres? Easiest way I found is Kafka, I would love some simple python script instead
Stream? Unless you have really hight traffic, put that in a python script while loop that regularly check for new rows, and it will be fine.
If you want to get fancy, db now have pub/sub.
There are use case for stream replication, but you need way more data than 99% of biz have.
You don’t want stream replication for scale, you want it for availability and durability. The number of times replication has saved my ass is too damn high.
And you want Kafka (or something like it) the moment you need two processes handling updates which again you probably want for availability reasons so a crash doesn’t stop the stream.
You also don’t catch deletes or updates with this setup. But there’s a million db to Kafka connectors that read the binlog or pretend to be a replica to handle this.
Sure, if you need high availability + replication cross db, go for Kafka, but that's the thread point: it's not a common use case.
Most people don't need high availability, and most replication use the same db which comes with tooling for that.