Comment by aleda145
3 hours ago
As a data engineering person I can say that this is a great write up!
Some thoughts:
A "bubbling" topic right now is conversational analytics (i.e. talk to your data). There has been an explosion of tools in the last 6 months. YC is backing one too: https://getnao.io/
I feel like pandas is also somewhat frowned upon, the industry has moved on from that. Most SQL tools can now do everything that we could only do with pandas.
In my network everyone is talking about DuckDB. As long as you are under a 1TB it will have everything you need. I think most people should start with that vs locking themselves into something like Snowflake
I love what I am hearing. I still see a lot of engineers using pandas, but it is such a horrible tools. You usually find an abandoned notebook with 100s of ”df_final_2” with sequential wrangling, making it impossible to understand what’s happening. Notebooks are also horrible for the same reason IMO.
With chat-your-data you have Hex, Claude + MCP, snowflake, Databricks etc… everyone’s in on it.
Just to add, people in my network have been talking about polars (as an alternative to pandas) and other dataframe libraries. They're much easier to use now thanks to the Narwhals compatibility layer (for example, Narwhals was recently added as a dependency to scikit-learn).
What's different between Narwhals and Ibis? Why does the former exist when the latter has already (struggled along) existing for a while? Narrower scope / benefit of hindsight? No support for "cloud" data frames like Ray and Spark?
Also Pandas is very much still a great tool and it's only getting better. It has some fundamental limitations that are relevant for processing bigger datasets or running things with higher performance. But it's still my preferred data frame for interactive day-to-day work. I only switch to Polars (or DuckDB) when I want to maximize performance.