Comment by dm03514

2 days ago

Still hacking on some data tools:

DuckDB for stream processing:

https://github.com/turbolytics/sql-flow

Lightweight kafka stream processing using DuckDB as the execution engine. 300MiB runtime can easily handle thousands of messages / second.

Working on a Kafka Connect alternative:

https://github.com/turbolytics/librarian

Right now mongo replication (through changestreams) is supported to kafka. Working on Postgres support right now.

0 comments

dm03514

Reply