Comment by manucorporat
2 days ago
I started this about 9 years ago and never finished it. The idea comes from a course in my telecom degree called "Señales Aleatorias y Ruido" (Random Signals and Noise), I spent so many evenings writing probability by hand, and every time I wanted to check a result with a computer it was a ton of boilerplate.
The engine is Rust, the JIT is built on Cranelift, there is also a WASM backend so everything runs in the browser too.
Full disclosure, I could only finish it now because of AI agents. In my experience they are amazing at the runtime and the numerical code, but pretty bad at language design, so I kept that part for myself.
It's a toy language. Ask me anything!
I built a small one of these as a calculator back in the day, it was fun and I always wanted to see someone really run with the idea more, so your project is super cool!
Did you ever look at symbolic or exact operators instead of purely monte-Carlo?
I remember reading the paper for distr, they used a mix of Fourier transforms for convolution and symbolic reduction to build a probabilistic computing library in R. I attempted building a small python library for this, but for my problems the CLT ended up sufficient to approximate the results faster, so I went with that.
You may enjoy reading the paper, it’s not groundbreaking but is a nice presentation of relationships/operations. Maybe it’ll inspire some features for you.
https://arxiv.org/pdf/1006.0764
Very cool. I have a couple of questions:
1. It's obvious that you are a big fan of Cranelift. I'd be interested to hear more about your experience in practical terms. For example could you share any insight about use cases where it is best suited, and where it might be better to look elsewhere? Did you hit any pain points? What was its killer feature for NoiseLang?
2. You wrote: "My favorite trick is in the RNG. Generating random numbers is a serial dependency chain, so instead of fighting that, the kernel runs four independent streams at once and lets the out-of-order core overlap them. This trick ended up beating a hand-written SIMD kernel!" What does this mean exactly? you just ran a scalar kernel 4-wide using SIMD instructions? or you interleaved 4 scalar copies of the same algorithm? did you generate the code or just duplicate the streams by hand?
I was exploring ways to speed up this language, the naive implementation is just a interpreter executed in rust, which can just do so much. Once thing i explored was to compile the program graph into WASM, then execute WASM. The idea is the the WASM runtime would JIT program and run faster than any interpreter I could write myself. During this exploration, i found that I could use the JIT optimizer directly and skip the WASM step.
What noiselang does it parse, convert to a execution graph then carefully use the Cranelift api to describe the program, and under the hood, it will find different opetimization and generate byte code that runs directly in the host CPU.
The reason for it to be killer is that it allows NoiseLang to run as native speeds, with very little compiler/optimization work. it's a very simple repo.
For the RNG, this was a discovery myself, when profiling, i found the many benchmarks were limited by the speed of the RNG itself, ie, if i could genenrate random numbers faster, the simulation would be faster. xoshiro's next number is computed from its current state. So to get number N+1 you must have finished number N. It's a chain: A → B → C → D. Your CPU can run maybe 6 integer operations per cycle, but a chain only ever offers it one to run. Five of the six lanes sit empty.
I tried to use SIMD to speed this up, but still hit the limit, even if using SIMD, it still had to wait for the next number, a massive speed up came from realizing that i can keep four independent xoshiro256++ states and emits four samples per loop iteration, i += 4. Since the four state-update chains share no registers, the out-of-order core issues them in the same cycles instead of stalling on one serial chain.
SIMD gives you more work per instruction. But I wasn't short on work, I was waiting. A 2-wide xoshiro still needs state N before it can compute state N+1, so the chain is the same length and I wait at every link, I just get two numbers per link instead of one.
And each link costs more. xoshiro rotates a 64-bit word every round, and NEON has no 64-bit rotate, so that becomes three instructions instead of one. Twice the numbers, three times the wait.
Four streams wins because it leaves the chain alone. It just runs four of them at once, and the CPU was already idle enough to overlap them for free.
Surely SIMD combined with multiple streams would beat both approaches. (This would be separate streams in each SIMD lane and separate streams in different SIMD variables.) There are multiple SIMD execution units, just like the 6 scalar units you mention. The latency of SIMD ops will be similar to scalar, except in cases you mention like shifts.
Definitely going to play around with this, thanks for posting.
I know MCMC isn’t your goal, but seems like this could be used for ABC-MCMC (as is?)
Would also be nice to have an option to plot using a KDE vs histograms.
(Also your FM example seems to be technically PM)
oh! that is very interesting. I was not aware of I could simulate markov chains with Approximate Bayesian, I have some good reading to do this weekend! indeed, expressions like P(D == 8 | D > 3) are already natively supported: https://noiselang.com/play/#x=conditional_bayes
Fair! My thinking was that PM of a single tone signal (the one i use in the demo is equivalent to FM, but shifted a bit). And implementing real FM for decoding is a lot more noisy, but I will add some callout in the article.
Truth be told, you motivated me to write the exact FM with the differenciation, maybe. Could be interesting to simulate PM vs FM for non single tone signals, to see how FM does even better!
Be warned - by using AI like this you've made yourself a lightning rod for the people who really really really dislike AI.
I know haha I wanted to be transparent about this, I have been coding since 9 years old, 32 years old now. I have nothing to prove other than it would have been impossible for me find time to complete this project without help, also a Toy language. Not trying to replace anything people use today :) it's a cute project
Yeah. :)
Since the advent of AI I have been delighted by the number of old project ideas I've been able to execute. There are just too many ideas to implement them all by yourself - but AIs don't seem to mind in the least.
It's a brave new world.
how much did Ai cost for you?
Claude Max Plan honestly is kind of endless, I have a old game written in MRC Objectice-C that i am porting to something more modern, and still within the limits of the flat subscription... not sure for how long this will last
so 200$