Comment by m101

1 day ago

What would be a revolution in excel is if we could have probability distributions in cells. Right now all spreadsheets are fairly deterministic and we think that tells us something about the real world. Having an ability to make cells distributions (and then all resultant cells would be distributions too) would represent the uncertainty in the world better (and expose lots of interesting things like distributions of tails)

Guesstimate (https://getguesstimate.com) has been around for years. I assumed Excel and Google would incorporate its features quickly, but it never seems to have taken off.

Features:

- All cells are named (auto named if you don't set them)

- Cells can have single values, many types of probability distribution, or sample data

- Formulas also produce distributions as first class outputs, because calculation is a Monte Carlo sim

- Each cell also has a pop-up box that invites you to explain your reasoning -- built in documentation

It got so much right. Sample data particularly useful because you can build a probabilistic bottom-up forecast and then update it with real-world data.

  • This is really amazing, and it reminds me of an Excel modification I did 24 years ago for an animation studio. They had around 3,000 digital artists and then about 200 support staff, and around 8 different productions moving through the studio at the same time. Each production was at some different stage of completion. The studio's production management staff spent a huge amount of time negotiating what staff was assigned to what productions, and the reasonings were all hazy and non-fact based. So I wrote what people would now call a deep learning system that mined all the accounting and production tracking records the studio had, and created probability distributions and confidence intervals for every individual and every task, and every combination of people and their combined tasks, and that all composited together into a giant predictive engine of real time whatif scenarios for the entire studio in different configurations.

    It was used for a series of feature films you all probably saw: the Chronicles of Narnia series, the Scooby Doo live action CG films, the Garfield films with the CG Garfield, and Life of PI.

    People both loved and hated it. They loved the insights into the studio's capabilities, and they hated thinking through how the thing operated, and they hated the thought process of constructing a new whatif scenario, because they had to THINK.

    • I'd say a broader "people has to WORK" about various accounting systems I developed. E.g. to make something warehousable, people need to enter various details about their products, and they hate it. And Amazon shows us that marketplace can be successful without even basic weight and dimensions.

What would this look like? Whenever I need something like that, I just stick each parameter in its own cell and use functions from there.

I would love an easier way to communicate to tech/math-illiterate CEOs that worst case, average case, and best case are different things, but must all be planned for.

When I did Process Improvement we used MiniTab for this.

I honestly was not a fan of that program and would have loved to be able to use Excel, but there are so many statistical things you just can't do in it.

Check out @Risk and crystal ball - both have been around for a long time.

I should also say what’s really nice is you can define the shape of the probability distribution and parameters for random variables, as well as the correlation of different random variables. Both are effective but also feel dated these days.

https://lumivero.com/products/at-risk/

https://www.oracle.com/applications/crystalball/

  • Off-topic: managing @Risk licenses in a fully on-prem, no-internet environment has been such a pain in the neck.

Its a paid add-on but @Risk gives the functionality you're looking for

  • As a number of the comments have suggested there are a number of Monte Carlo methods (including this addin) but I was hoping that someone would make one which calculated based on random variables and output a resultant random variable, over brute force monte carlo. Monte Carlo is also somewhat slow, but doing the maths analytically should be fast.

    • MC is used because there is no analytic solution. Only some likelihood functions have a conjugate prior. You have to rework the analytic solution for every model change (where most changes would result it in being non-analytic anyway), and they don't work well for high dimensional problems. Restricting the tool to analytic would mostly result in a toy app that just does Gaussians.

This has been my dream for decades. I even pestered the excel team when I worked at MS. Everyone thought it would be cool but not worth the effort and complexity.

With this new functionality, you could store 10000 samples of the distribution in the cell. Then broadcast your computations to the sample axis.

Would those then be tensors?

Matrices are Tensors but with the matrix product operator instead of the tensor product operator.

Pandas supports MultiIndex DataFrames but the pandas docs recommend xarray for 3D and N-Dimensional data.

xarray supports N-Dimensional data as for example NetCDF but not tensor arithmetic.

xarray_jax: https://news.ycombinator.com/item?id=41411280 :

> W3C CSVW supports per-column schema. ( with URIs for datatypes )

> Serialize a dict containing a value with uncertainties and/or Pint (or astropy.units) and complex values to JSON, then read it from JSON back to the same types. Handle datetimes, complex values, and categoricals

IEEE-754 specifies NaN (null), ±0, three infinities (positive, negative, and unsigned), but IEEE-754 does not specify a representation for categoricals, datetimes (like ISO8601), or complex numbers.

XSD (XML Schema Datatypes), which RDFS vocabularies often use to specify the rdfs:range of an rdfs:Property, does not specify how to specify abstract complex numbers; but OpenMath RDF, and QUDT (Quantities, Units, Dimensions, and Types) and OM Ontology all have a way to save complex numbers to disk, too.

  • The matrix product is not something specific to matrices, but it is a particular case of the contracted tensor product.

    (While now almost everybody uses the names "tensor product" and "contracted tensor product", historically these names are wrong. Better names, which do not misuse the word "tensor", would have been "Zehfuss product" and "Cayley product", after the names of the mathematicians who have introduced them.)

    The word "tensor" has become increasingly corrupted over the years. Originally, it had a meaning that matched its etymology, i.e. "tensor" meant "symmetric matrix", because a matrix could be decomposed through the polar decomposition into a tensor and a versor (i.e. a symmetric matrix and a rotation matrix).

    Then what is now called tensor theory was created by Ricci, but without ever using the word "tensor". The meaning of the word "tensor" was changed by Einstein in his theory of the general relativity, where for unknown reasons he chose this word to refer to arrays with an arbitrary number of dimensions that satisfy a certain kind of transformation relationships when the systems of coordinates is changed.

    After WWII, due to the huge popularity of Einstein, this weird usage of the word has become widespread.

    Then for many decades tensors were just a subset of arrays, those that are changed in a certain specific way when the system of coordinates is changed.

    Eventually, there was the second corruption of the word "tensor" which is completely unjustified, when "tensor" has become just a synonym for "array".

    Vectors are 1-dimensional arrays (though originally, like tensors, vectors were only a subset of 1-dimensional arrays), matrices are 2-dimensional arrays and nowadays "tensors" means just "arrays", i.e. of an arbitrary dimension.

    The tensor product and the contracted tensor product are defined for arrays of arbitrary dimensions. For 2-dimensional arrays the contracted tensor product is a.k.a. matrix product, and for 1-dimensional arrays it is a.k.a. scalar product.

    • yes. do you have a more useful meaning for tensors?

      MLIR distinguishes arrays and tensors by allowing array mutation, but not for tensors; which allows a large class of optimizations

      though the internal representation is one and the same

      i understand it is the same for SIMD aswell (modulo alignment etc)

      1 reply →

    • Normed tensor Gaussian splatters are useful too.

      From https://news.ycombinator.com/item?id=49855122 re: gravity from QED without any GR spacetime curvature, using the ~amplituhedron to solve n-body gravity with scattering amplitudes, and SQG/DDF:

      > normed tensor gaussian splatters work well as a simulation primitive for this too because they do conservation.