Comment by servo_sausage

Big studies have an important role. Especially for dramatically different approaches, such as the different approaches to teaching reading. The differences are so acute, that careful A/B, or A/B/Control studies are the best approach.

But most improvements in any complex system happen iteratively, and benefit from clusters of subtle changes found to work well together. At some point enough experience is gained to characterize the change, and give others a chance to consider it.

I suggested incremental adoption, and organic adoption, of successful changes, precisely because of this need for significant bottom up testing before spreading something widely. Success at one scale, and location, doesn't always translate directly to another context, or might not work in another context at all.

> You also run into issues with teaching skills and standards, you need a high level of planning and adherence to the supplied plan in order to measure outcomes; otherwise it's just vibes based on individual teachers.

You point out a very important concern. Good measurement doesn't make bottom-up improvement impossible, unnecessary, or any less important. The point of measurement is to make improvements easier to see, and not get in there way.

This is what I was referring to when I said at the top, the job is to create a context where improvements can happen.

One of the simplest ways to balance top-level and bottom-up concerns, is to communicate the actual top-level needs (not just current practices) clearly, then let front line educators propose changes to measurement practices, where they feel the current practice is holding them back. That gets both scales working together to enable improvements to happen, and to be seen. There is no (competent) conflict here, the opposite.