Comment by wahern
2 hours ago
I once had a statistics professor (teaching a programming class using, IIRC, SAS) in college who worked as a data analyst in the federal government, possibly the Labor Department. Part of his job was working with polling and form data. After class he once tried to explain a paper he wrote arguing for an additional null-like type in the formal analytics system in his domain. I was young and stupid (took it as an easy programming course; was not skilled at abstract math) and impatient (wanted to meet up with friends at a pub) so didn't follow along well enough to even properly remember his argument, let alone understand it. But I've always distinctly recalled that he was arguing for a 4th, or possibly 5th type, yet whenever I survey the literature I've only found at most such 2 types. But I've probably been looking in the wrong places.
For context, NULL/missing was a type. N/A was a type (i.e. a question irrelevant to someone). I can't remember the 3rd, and of course not the additional (4th or 5th) type he was proposing. If anyone knows a good resource that describes this problem domain, that would be much appreciated. I've been haunted for over 20 years by the regret of not fully appreciating what he was trying to explain to me, both because I could've learned something that day, but also because I almost immediately felt like an a*hole for not showing enough interest in and respect for a piece of scholarly work he was clearly proud of and wanted to share with me.
EDIT: I think the third pre-existing type was for a situation where someone abstains from answering a question like, "Sex: male/female", because they consider themselves neither. So something like an Other type. In retrospect this was presumably related to set theory, and he felt there was a gap in the pre-existing formal models.
> For context, NULL/missing was a type. N/A was a type (i.e. a question irrelevant to someone). I can't remember the 3rd, and of course not the additional (4th or 5th) type he was proposing
“Don’t know”: this has a value, but we do not know it could be one. Also, within N/A, one _could_ discriminate between “not available yet”, “will never be available”, “was available once, but was lost”, etc.
Depending on the domain, others could be
- “Can’t tell”: this has a value, but you are not allowed not know it (unlikely, as you likely also shouldn’t be allowed that the value exists)
- incomputable: this has a value, but it isn’t possible to know it
I think generic systems shouldn’t try to capture such domain specific things, but allow for implementing them. SQL shouldn’t even have null, but have enums and product types on top of which you could create it and functions supporting it.
This reminds me of the sixteen or so different flavors of null used in HL7! https://terminology.hl7.org/en/CodeSystem-v3-NullFlavor.html though in practice I've only seen a handful of these. Notably:
NI No Information - the default, it is unknown and why it is unknown is also unknown
UNK Unknown - the most common other than NI
ASKU Asked but Unknown
NASK Not Asked - we don't know because we didn't ask
MSK Masked - hidden due to privacy or legal etc
NA Not Applicable
NAV Not Available - for example patient might be unconscious or doctor might be unreachable, temporary