Comment by inaseer
21 hours ago
I've been working on a framework for writing executable specs in .NET called Accordant, developed at Microsoft and open sourced recently.
Github: https://github.com/microsoft/accordant
Docs: https://microsoft.github.io/accordant
Every API has a contract - the rules for how it should behave. You can't withdraw more than the balance. You can't delete a resource with active references. You can't re-create what already exists. But usually these rules are never written down in one place. Accordant lets you write the contract directly, as executable code. Not documentation that drifts, but code - if the implementation stops behaving according to the contract, you get immediate failures. Not only can you use the executable spec to validate _arbitrary_ scenarios, you can also use the spec - a first class construct - to mechanically explore the state space of a system and generate interesting test sequences. The docs above have examples.
Also worth calling out that we've used the framework to model a number of complex, distributed real-world systems: those involving async processes, concurrency, retries and crash consistency. These are non-trivial specs (and they pair quite well with techniques like deterministic simulation testing). Great care has been taken to ensure the specs remain readable and concise despite that richness of behavior. For those of you old timers who might be familiar with Spec#/SpecExplorer and NModel, this model-based testing library is a descendant of that line of work.
With the rise in AI-assisted software development, I feel we need richer ways of specifying and validating software and I feel quite excited and bullish about the possibilities here. There's a lot more to say on the topic - follow my twitter feed if interested in more updates ;)
This is interesting; I see the docs mention that polling support is built in for asynchronous background tasks. What about event-driven systems where a message will be published when a task completes (such as from a message broker/pubsub system)?
There are really two separate concerns here.
The first is that some effect happens asynchronously, potentially interleaved with other operations. Whether a client observes completion by polling or by receiving an event from a message broker is orthogonal to the specification itself - the model looks essentially the same in both cases. The built-in test executor uses polling, but that's an execution strategy, not a specification construct.
If you have a trace containing both requests/responses and observed events, you can use the model to check that the trace conforms to the specification. In practice, it helps if the events can be localized to some interval in the execution (e.g. "this happened after A and before B"); otherwise the checker has to consider many more possible concurrent interleavings.
The conformance testing docs hint at how this can be done, but don't yet show an event-driven example. It's a good enough question that I'll write a dedicated doc page on it.
Conformance testing page: https://microsoft.github.io/accordant/docs/concepts/conforma...
Is this property based testing? Or if not how does it differ?
Property-based testing and model-based testing are closely related. Both ask the developer to state the expected behavior of a system (whether you call it a property, invariant, model, specification, or contract) and then validate that behavior over arbitrary inputs and arbitrary sequences of operations. Property-based testing frameworks also typically provide fuzzing and shrinking.
Where we felt there was a gap was in expressing rich stateful behavior: models involving non-determinism (e.g. a timeout where the write may or may not have committed), concurrency, and eventual asynchronous completion, and then checking that an observed execution trace conforms to that model. Accordant aims to make those kinds of specifications concise and readable.
Once you have such a model, it's possible to integrate it with the fuzzing and shrinking capabilities of existing property-based testing libraries. We'll have documentation on that integration soon.
[dead]