Comment by aarondia

5 years ago

After talking with Excel users for the past year to understand what makes it such a ubiquitous tool, I don't think that its the ability to write complex formulas, but instead the reactive, visual environment (read: IDE). Of course, for some users the ability to nest 100 VLookups or create UDF's is the important part, but for the vast majority of people, their workbooks aren't __that__ complex.

When it comes to data analytics IDEs there is a fundamental tradeoff between staring at the data or staring at the transformations. Excel makes sure you are brutally aware of each edit you make to your data at the expense of reproducibility and auditability of your transformations. Python takes the opposite end of the trade -- obscuring the underlying data, but bringing the transformations to the forefront. For non-programmers trying to learn Python (especially for data analytics), the biggest hurdle is losing touch with their data.

I've been building a Python package, Mito (https://trymito.io/hnc), to try to address this tradeoff for those who want to analyze data with the intuitiveness and data-first-ness of a spreadsheet, but with the power and traceability of Python. Mito is a Jupyter Lab extension which gives users an interactive spreadsheet that automatically converts your spreadsheet analysis to the equivalent pandas code. You can write Excel formulas, merge datasets, create pivot tables, etc.

7 comments

aarondia

airstrike 5 years ago

I've seen Mito around HN and reddit before and I think it's an absolutely brilliant, but its applications may be more limited than the vast use cases covered by Excel. Maybe that's fine, as your goal may *be to carve out a niche rather than compete with Excel more broadly.

The reason I say that is I think the major pain point in its hypothetical widespread adoption would be getting users accustomed to the ergonomics of accessing attributes in a programming language when they only understand noob-friendly Excel formulas, which is more functional in nature.

Looking at the demo, I think some pseudo-code such as the following could be easier for e.g. an office worker than pure python:

    ADDCOLUMN(Sheet='Train Stations', Name='Accepts Bags') # defaults to appending at the end

    SETFORMULA(Column='Train Stations'!'Accepts Bags', As=IF('Train Stations'!'Checked Baggage' = "Y", 1, 0))

    PIVOT(From='Train Stations', To='Pivot', Keys=('State'), Values=('Accepts Bags'), Formula=SUM)

    SORT(Target='Pivot', By='Accepts Bags', Direction='Asc', NA='Hide') # defaults to ascending, NA first

Clearly, it's not like I've thought this through carefully and am not claiming this particular example is really ergonomic, but hopefully this illustrates the point I'm trying to make.

It would not need to look like Excel, but I think the jump from spreadsheet -> Python may be a step too far for the average user than, say, spreadsheet -> some functional approach.

aarondia 5 years ago

Glad to hear you're a fan! You're right, Mito is not designed to cover all of Excel's use cases. Mito focuses on rectangular datasets that fit nicely into a pandas dataframe ... at least for now. So the types of use cases that we're looking for is data cleaning, pivoting, etc. If you're building an LBO, Mito is not the tool for you.
It seems like what your proposing is almost a wrapper around pandas functionality to make the language easier to read for Excel users. I think that's a super interesting approach which we honestly haven't thought that much about. As a rule of thumb for Mito right now, any spreadsheet formula gets generated as a Mito formula (ie: using an IF statement in the Mito spreadsheet generates the code IF(A > B, 1, 0) instead of the Pandas code) and anything else is raw pandas code (ie: pivot tables, merges, add column).
In general, we've been thinking about trying to move more of the code to the raw python approach since we've heard things like "not seeing the raw script makes the code unproductionizable" etc. But I also see your point that beginning Python users might prefer readable code over Python code. If we took that approach, users would still get the reproducibility, auditability, and ability to use a spreadsheet interface on large datasets, they'd just sacrifice any semblance of learning Python. That's great food for thought!

hermitcrab 5 years ago

>When it comes to data analytics IDEs there is a fundamental tradeoff between staring at the data or staring at the transformations.

That is an interesting point regarding Excel vs Python. But no-code, flow-based data transformation tools such as Easy Data Transform (my own product), Alteryx and Tableau Prep offer a different approach, by having a canvas of transformations and allowing you to see the data after each transformation with a click. This loses some of the massive flexibility of Excel, but also has a lot of advantages, including: no syntax to remember and the transformations are much more visible and easy to reverse.

tinomaxgalvin 5 years ago

So I have been using Excel every day for 13 years as an actuary. I'm also involved in the management of software project in c++.

You are correct. It's a combination of REPL like behaviour and being able to see the entire state at once that does it. Excel is the most agile programming environment known to man.

klelatti 5 years ago

Completely agree from my actuarial experience too. Do you find in practice though that often Excel is only part of the workflow and that a lot of the agility is lost because it's using output from a (often black box) modelling tool that is doing a lot of the computational heavy lifting?

ookdatnog 5 years ago

Just curious: Is the name "Mito" derived from "mitochondrion", as in the "mitochondrion is the powerhouse of the (ex)cell" meme? :)

aarondia 5 years ago

Sorry for just getting back to this, but yes it is! Almost nobody picks up on it, so color me impressed.