← Back to context

Comment by lhorie

4 years ago

I have to echo somebee about that benchmark suite. It's fairly well known in the framework-bulding community that this is not a very good benchmark (just ask Boris Kaul of Ivy.js, Ryan Carniato of Solid.js, Leon Sorokin of domvm, etc etc).

Some "frameworks" achieve good numbers there by being utterly unusable in real life, others miss the spirit of the benchmark completely (e.g. submitting a non-keyed implementation as keyed thereby gaining an unfair/misleading advantage), and as somebee said, the benchmark itself largely measures repaint time (and does so in a less than ideal way, by using setTimeout macro queue as a proxy for repaint time measurement, in band, instead of instrumenting performance via CDP). It lacks rigor in many ways (the most blatant was that initially it considered keyed and non-keyed implementations on par, but there are other issues such as putting a lot of weight into DOM creation compared to e.g. array diff, or as somebee said, not measuring low repaint load diffs)

IMHO, it only has two things going for it: a) it has a lot of frameworks listed b) it does at least attempt to measure repaint times unlike other benchmarks that only measure JS time (which has become somewhat irrelevant since V8 et al now offload repaints out of the JS thread)