There's something very satisfying about how this style seems to "climb the abstraction ladder" very quickly, but all of those abstractions he creates are not wasted and immediately put to use. I think much of the amazement and beauty is that there isn't much code at all, and yet it does so much. It's the complete opposite of the bloated, lazy, lowest-common-denominator trend that's been spreading in many other languages's communities.
I was surprised at how verbose and commented the code was. Then I read the note saying that Whitney's original code was in `ref/` and the two files in the root were annotated by other kparc members.
I know you're joking but people who write C like him would argue it's not obfuscated. It's certainly not intentionally written to be more difficult to read, and those who are practiced in the art often say that it's _easier_ to read.
I recall a chap called Geocar demoing kOS, in a very impressive fashion. (I additionally recall an individual had to awkwardly hold the mic for the whole presentation)
I am interested, but one thing I have still not quite understood is where to use this family of languages.
As far as I can tell they are best for data analysis but not for heavy numerical computation because most or all of them lack GPU support. is that right or have I got it wrong?
q/kdb+ is used in finance (banking + funds) for heavy numerical computation every day. high-volume realtime data straight from markets, and petabyte/trillion-row historical DBs. it runs on CPU but computation easily parallelizes over cores/clusters.
I have a longstanding fascination with K and other "modern" APL derivatives.
There are a few intersecting truisms about coding that I believe: one is that people's working memory varies: some have an immense amount, some less. Humans definitely process spatially better than in time series (e.g. comparing side by side rather than turning over a page.)
This implies you should prefer succinct code and languages because they are less memory load for engineers working on them.
At the same time, a corollary is that a smaller standard library / language is generally better, in that less needs to be learned by an engineer for full coverage of the language.
Another truism is that some people's processing speed is higher than others, and in general I think of the combination of working memory + speed as roughly equivalent to "g", general intelligence.
K occupies this weirdo place though, because it's absolutely succinct, a very small language as counted by number of atoms supported by the interpreter, and also incredibly hard to scan.
One of the K intros I read mentioned that the language is designed to be something that takes down your thinking; essentially the idea is that the workflow is "drink coffee with fellow PhDs, annotate on the chalkboard, and then when ready, capture it directly." This seems about right to me with my own K/J/Q experiences -- the bulk of the time is spent thinking about structuring a problem solution.
I compare this to go, a language I love for its long-term readability and maintainability, where I spend a lot of time writing boilerplate and dealing with errors in-situ.
At any rate, somehow there's a sort of event horizon of terse solution making where you come out the other side and need a 170 IQ to feel comfortable, and Mr. Whitney lives where he lives, and I live where I live. :)
People complaining about how ugly the C code is here are definitely missing the point: he has bent C's preprocessor to his will in order to encapsulate how he thinks about coding: essentially functional, vectorized. It's using C to write a DSL for solving programming problems interesting to Arthur Whitney.
I think it's fascinating on those terms. In a world where you have to read 10,000 lines of code from 100 developers, the C is terrible, and hard to parse. In a world where you will mostly write code to a style you've honed over 40+ years, it's super expressive, minimal, pared down to what matters, and probably fits his brain perfectly.
One analogy I like to tell people who are overcome with shock and horror at APL-family languages is to compare it to someone used to Latin-family human languages looking at something like Chinese for the first time --- it's likewise totally "unreadable" at first glance, but then you realise that over a billion people can read and write that language fluently every day, many of which may also struggle with a Latin-family language as they've never seen one before.
I'm not convinced that APL is "incredibly hard to scan" for someone who is familiar with it; it's just a matter of experience. While I'm by no means experienced in APL either, a visually similar thing I did frequently in my younger days was reading x86 instructions not in a disassembler nor hexdump, but displayed as CP437. It was not hard, and I can still remember ┤, PQRS, ═!, and ├ as "MOV AH", "PUSH AX; PUSH CX; PUSH DX; PUSH BX", "INT 21", and "RET" respectively. Here's an example:
ò║•═!├Hello world!$
Edit: looks like HN swallowed a byte, but you can sort of see what I mean.
y=x/5; / this would be considered "noisy" by C programmers, relative to COBOL
The first is COBOL (designed to make code easier for "normal" people to read. The second is C/Java/Python/Javascript (which looks more like the math that we learn in grade school).
k/APL/J simple moves further in the direction of the algebraic notation you already know. The difference is more operations/algorithms.
When you read the one-character symbols in K as algorithms versus characters, it makes much more sense. In addition, you can read "faster" in k than in other languages, relative to the functionality being expressed.
When I review C/Java/C++, I print out the source code and write the equivalent k code in the margin. The compression ration is typically 10-20X. Doing so speeds up my work significantly when I go back over the reviewed code.
As a moderately experienced K programmer, I find K easy enough to "scan". Common idioms immediately stand out as recognizable "words" that are suggestive of what a routine does before you fully parse it:
@& (filtering)
@< or @> (sorting)
@\: (folding)
,/ (flattening)
+\ (a running total)
etc.
It's also easy to notice, e.g. in K3, a reserved name like "_f" and immediately know you're looking at a recursive procedure.
Another is regular expressions. They look entirely incomprehensible if you see one with no prior knowledge, but once you get used to them they are fairly easy to understand.
Oh absolutely. Pg talks at some point about how LISP macros can create a sense of godlike power / mania, and I think there's some of that in APL-land too. The true titans don't need to debug in the traditional sense - a whole program fits on a single screen, they can load it in their head, reason about it, and see what happened.
Non-titanic engineers accidentally generate bugs in these languages with high "floor" IQ requirements, and debugging someone else's K code is terrible. Debugging your own K code is terrible, not less because when you get help you will feel very, very stupid indeed :)
In my opinion, the language that comes closest to K's functionality and is also understandable by mere mortals is K itself. It is obviously extremely close to K's functionality and is a very simple language, the only reason it doesn't seem simple is that most people are used to verbose languages. A couple of days of practice is enough to make K readable, in my experience.
Also, I am constantly amazed at how concise K is, easily rivaling not only conventional languages but also much larger array languages like APL or J. Arthur Whitney's taste in selecting primitives is out of this world.
I'm extremely biased in recommending it, but Lil is semantically very similar to Q, entirely free, and intended to be beginner-friendly: https://beyondloom.com/tools/trylil.html
It's not as powerful or concise as K, but it gives you some of the flavor of an array language tucked inside what resembles an ordinary imperative/functional scripting language.
I recommend checking out uiua.org for fun. The docs are well written and the concepts, while foreign to most, are ultimately accessible and interesting.
k and uiua are in different branches of the APL family.
This is super accurate, and interesting in the context of the repo. It’s clearly not the most accessible language or style - I wonder if someone who has hyper optimized for themselves in the way Arthur has can write an effective teaching language. It requires a theory of mind for minds that are probably quite different in the general case.
I don't know about Golang but there are languages where this sort of highly domain-optimized, super terse syntax could be embedded as a domain-specific sublanguage, in a significantly less hackish way than what C allows.
I'm not that knowledgeable, and def not a C apologist, but what language are you thinking of? To my eyes, that is some heavy abuse of the preprocessor in a way that I don't think almost any modern "safe" language could possibly countenance.
I read that code like he got the expressivity benefits of a lisp macro system with close-to-the-metal C speeds in like 100 lines of code. I'm curious what else could do this.
The K language seems to be limited to CPU only. Yes, it can call out to GPU (https://code.kx.com/q/interfaces/gpus/) but K only runs on the CPU. This strikes me as odd for an array language.
It's around, and they recently stopped providing free download links for recent versions, from which I take it that they have a reasonable enterprise sales program rolling now.
The feature split on the free / enterprise edition holds a lot back; my vibe on the free version was it was just enough to validate that shakti is performant, and then they want you to pay.
Heavy use of the C preprocessor and C defaults to embed a functional programming language. Language with a small number of core functions and ability to apply functions to lists of atoms. Aesthetically favoring short identifiers and minimal whitespace to create high semantic density. Eschew comments.
Whitney and I both worked in the 1970s for I.P. Sharp Associates, which used an email system written beautifully in APL by Leslie Goldsmith. Most user names were simply our initials. Ian Sharp was IPS, Leslie LHG, and Arthur ATW. (Middle name Taylor.) I’ve been SJT ever since (including to two wives) but was not smart enough to grab the domain: see 5jt.com.
Arthur Whitney is showing how to build an array language using C in array language form.
AW is known for kdb+ which is often used in finance due to its extreme performance properties and ability for quants to quickly explore ideas.
Personally, to get a better handle on how array languages worked, I implemented KlongPy which is a python implementation of Klong, which descends from K (which AW wrote).
You have to play with this stuff to understand it intuitively.
Ha! For those of us who were in the Morgan Stanley Fixed Income group for many years in the 1990s and 2000s AW is known for APLus which was used extensively for modeling and application work.
FWIW, kdb+ is not that extremely performant - there's a lot of things that could be faster, and a lot of limitations that mean that you often would be better of not using a DB at all (or to use another DB and just pull everything you might need into memory). There is/was a tradeoff in that many things that would make it faster would require more code, and a cool thing about q/kdb+ is that it takes so little code you don't have I$ issues, but I think that's a tradeoff that doesn't make as much sense anymore in 2023.
What it's really great for is that it's really neatly integrated into the q language, which is great for exploratory programming, and it's fast enough not to get in the way.
Basically it is an APL-like language that is used (or was, I think it has moved on to a successor language called q now) in the proprietary kdb+ system used by some financial companies. Even if you aren't into finance it is a fun language to play around with.
array languages are very fascinating and I'd love spending more time learning about them
I read that they are (K in particular) used in finance...would you reckon it would be easier to find job in that field having that in the CV? I know I cannot compete on the C++/Python side but maybe as a skilled K developer I could sneak in
Might not, since for this kind of thing most likely either you give up in couple minutes (which it is not a heinous since it does not waste your time anyway) or you just read it.
Ha! The guy who wrote this is the same guy who invented ~~APL~~ a number of APL inspired languages (Edit: He did not invent APL. Thanks for the corrections!), so I suspect he may just be built different.
APL was invented by Kenneth Iverson, the other person mentioned on the page, not Arthur Whitney.
I was not convinced of the readability of APL, but compared to its successors which tried to stick to ASCII, I've learned to appreciate the merits of an extended character set.
J, the predecessor to K, also by the same author, is written in nearly the same style, but shorter:
https://news.ycombinator.com/item?id=25902615 )
There's something very satisfying about how this style seems to "climb the abstraction ladder" very quickly, but all of those abstractions he creates are not wasted and immediately put to use. I think much of the amazement and beauty is that there isn't much code at all, and yet it does so much. It's the complete opposite of the bloated, lazy, lowest-common-denominator trend that's been spreading in many other languages's communities.
Whitney seems to have an obsession, and a real knack, for writing or rewriting array languages.
I was surprised at how verbose and commented the code was. Then I read the note saying that Whitney's original code was in `ref/` and the two files in the root were annotated by other kparc members.
Whitney is an unrealized IOCCC champion.
I know you're joking but people who write C like him would argue it's not obfuscated. It's certainly not intentionally written to be more difficult to read, and those who are practiced in the art often say that it's _easier_ to read.
> Whitney is an unrealized IOCCC champion.
no, that'd be fabrice bellard, who is actually a "realized" ioccc champ.
atw doesn't do obfuscated c. you are led astray.
2 replies →
Pity that he writes in C; if he wrote in JavaScript, he wouldn't need a minifier.
2 replies →
Ah, thank you for clarifying this. I was confused why it said "by Arthur Whitney" when the contributors do not seem to be Arthur Whitney.
"contributors" is essentially me. what needs to be clarified?
it is by arthur whitney.
anything else?
I recall a chap called Geocar demoing kOS, in a very impressive fashion. (I additionally recall an individual had to awkwardly hold the mic for the whole presentation)
Did anything ever come of kOS?
Everything I've found that's public is documented here:
https://gist.github.com/chrispsn/da00835bb122c42f429a084df83...
The kparc.com links are down though.
For people discovering k for the first time, I wrote https://xpqz.github.io/kbook as a gentle intro.
I am interested, but one thing I have still not quite understood is where to use this family of languages.
As far as I can tell they are best for data analysis but not for heavy numerical computation because most or all of them lack GPU support. is that right or have I got it wrong?
Are there any other use cases?
q/kdb+ is used in finance (banking + funds) for heavy numerical computation every day. high-volume realtime data straight from markets, and petabyte/trillion-row historical DBs. it runs on CPU but computation easily parallelizes over cores/clusters.
regarding use cases, see https://kx.com/resources/use-cases/
3 replies →
I have a longstanding fascination with K and other "modern" APL derivatives.
There are a few intersecting truisms about coding that I believe: one is that people's working memory varies: some have an immense amount, some less. Humans definitely process spatially better than in time series (e.g. comparing side by side rather than turning over a page.)
This implies you should prefer succinct code and languages because they are less memory load for engineers working on them.
At the same time, a corollary is that a smaller standard library / language is generally better, in that less needs to be learned by an engineer for full coverage of the language.
Another truism is that some people's processing speed is higher than others, and in general I think of the combination of working memory + speed as roughly equivalent to "g", general intelligence.
K occupies this weirdo place though, because it's absolutely succinct, a very small language as counted by number of atoms supported by the interpreter, and also incredibly hard to scan.
One of the K intros I read mentioned that the language is designed to be something that takes down your thinking; essentially the idea is that the workflow is "drink coffee with fellow PhDs, annotate on the chalkboard, and then when ready, capture it directly." This seems about right to me with my own K/J/Q experiences -- the bulk of the time is spent thinking about structuring a problem solution.
I compare this to go, a language I love for its long-term readability and maintainability, where I spend a lot of time writing boilerplate and dealing with errors in-situ.
At any rate, somehow there's a sort of event horizon of terse solution making where you come out the other side and need a 170 IQ to feel comfortable, and Mr. Whitney lives where he lives, and I live where I live. :)
People complaining about how ugly the C code is here are definitely missing the point: he has bent C's preprocessor to his will in order to encapsulate how he thinks about coding: essentially functional, vectorized. It's using C to write a DSL for solving programming problems interesting to Arthur Whitney.
I think it's fascinating on those terms. In a world where you have to read 10,000 lines of code from 100 developers, the C is terrible, and hard to parse. In a world where you will mostly write code to a style you've honed over 40+ years, it's super expressive, minimal, pared down to what matters, and probably fits his brain perfectly.
One analogy I like to tell people who are overcome with shock and horror at APL-family languages is to compare it to someone used to Latin-family human languages looking at something like Chinese for the first time --- it's likewise totally "unreadable" at first glance, but then you realise that over a billion people can read and write that language fluently every day, many of which may also struggle with a Latin-family language as they've never seen one before.
I'm not convinced that APL is "incredibly hard to scan" for someone who is familiar with it; it's just a matter of experience. While I'm by no means experienced in APL either, a visually similar thing I did frequently in my younger days was reading x86 instructions not in a disassembler nor hexdump, but displayed as CP437. It was not hard, and I can still remember ┤, PQRS, ═!, and ├ as "MOV AH", "PUSH AX; PUSH CX; PUSH DX; PUSH BX", "INT 21", and "RET" respectively. Here's an example:
Edit: looks like HN swallowed a byte, but you can sort of see what I mean.
Another analogy
Would you rather write/read:
DIVIDE X BY 5 GIVING Y
or
y=x/5; / this would be considered "noisy" by C programmers, relative to COBOL
The first is COBOL (designed to make code easier for "normal" people to read. The second is C/Java/Python/Javascript (which looks more like the math that we learn in grade school).
k/APL/J simple moves further in the direction of the algebraic notation you already know. The difference is more operations/algorithms.
When you read the one-character symbols in K as algorithms versus characters, it makes much more sense. In addition, you can read "faster" in k than in other languages, relative to the functionality being expressed.
When I review C/Java/C++, I print out the source code and write the equivalent k code in the margin. The compression ration is typically 10-20X. Doing so speeds up my work significantly when I go back over the reviewed code.
As a moderately experienced K programmer, I find K easy enough to "scan". Common idioms immediately stand out as recognizable "words" that are suggestive of what a routine does before you fully parse it:
@& (filtering)
@< or @> (sorting)
@\: (folding)
,/ (flattening)
+\ (a running total)
etc.
It's also easy to notice, e.g. in K3, a reserved name like "_f" and immediately know you're looking at a recursive procedure.
1 reply →
Another is regular expressions. They look entirely incomprehensible if you see one with no prior knowledge, but once you get used to them they are fairly easy to understand.
For me Haskell, point free functions, and lenses are a happy medium :)
my fascination with Q/kdb/K/... disappeared once I had to debug it in production
you want a stack trace? tough luck, you'll get back:
`type
and that's it
Kx has much improved Q/k's debugging capabilities - including stack traces.
see: https://code.kx.com/q/basics/debug/
backtrace has been supported for some time. the debug facilities are nice
https://code.kx.com/q/basics/debug/#stack-frames
1 reply →
Oh absolutely. Pg talks at some point about how LISP macros can create a sense of godlike power / mania, and I think there's some of that in APL-land too. The true titans don't need to debug in the traditional sense - a whole program fits on a single screen, they can load it in their head, reason about it, and see what happened.
Non-titanic engineers accidentally generate bugs in these languages with high "floor" IQ requirements, and debugging someone else's K code is terrible. Debugging your own K code is terrible, not less because when you get help you will feel very, very stupid indeed :)
Awesome comment.
In your opinion, what would be a language that comes closer to K’s functionality but at the same time be understandable to mere mortals?
In my opinion, the language that comes closest to K's functionality and is also understandable by mere mortals is K itself. It is obviously extremely close to K's functionality and is a very simple language, the only reason it doesn't seem simple is that most people are used to verbose languages. A couple of days of practice is enough to make K readable, in my experience.
Also, I am constantly amazed at how concise K is, easily rivaling not only conventional languages but also much larger array languages like APL or J. Arthur Whitney's taste in selecting primitives is out of this world.
I'm extremely biased in recommending it, but Lil is semantically very similar to Q, entirely free, and intended to be beginner-friendly: https://beyondloom.com/tools/trylil.html
It's not as powerful or concise as K, but it gives you some of the flavor of an array language tucked inside what resembles an ordinary imperative/functional scripting language.
1 reply →
I recommend checking out uiua.org for fun. The docs are well written and the concepts, while foreign to most, are ultimately accessible and interesting.
k and uiua are in different branches of the APL family.
2 replies →
…fits his brain perfectly.
This is super accurate, and interesting in the context of the repo. It’s clearly not the most accessible language or style - I wonder if someone who has hyper optimized for themselves in the way Arthur has can write an effective teaching language. It requires a theory of mind for minds that are probably quite different in the general case.
I don't know about Golang but there are languages where this sort of highly domain-optimized, super terse syntax could be embedded as a domain-specific sublanguage, in a significantly less hackish way than what C allows.
I'm not that knowledgeable, and def not a C apologist, but what language are you thinking of? To my eyes, that is some heavy abuse of the preprocessor in a way that I don't think almost any modern "safe" language could possibly countenance.
I read that code like he got the expressivity benefits of a lisp macro system with close-to-the-metal C speeds in like 100 lines of code. I'm curious what else could do this.
1 reply →
The K language seems to be limited to CPU only. Yes, it can call out to GPU (https://code.kx.com/q/interfaces/gpus/) but K only runs on the CPU. This strikes me as odd for an array language.
shakti has GPU support now, i believe.
Anyone know what is happening, or happened, with shakti db[1]?
It's been years since Arthur started this other k-variant.
1- https://shakti.com/
It's around, and they recently stopped providing free download links for recent versions, from which I take it that they have a reasonable enterprise sales program rolling now.
The feature split on the free / enterprise edition holds a lot back; my vibe on the free version was it was just enough to validate that shakti is performant, and then they want you to pay.
Here's the code
https://raw.githubusercontent.com/kparc/ksimple/main/a.c
https://raw.githubusercontent.com/kparc/ksimple/main/a.h
What do they mean by "atw-style" ?
Arthur Whitney
https://en.wikipedia.org/wiki/Arthur_Whitney_(computer_scien...
https://queue.acm.org/detail.cfm?id=1531242
https://hn.algolia.com/?q=arthur+whitney
I'm guessing it's his initials; Arthur T. Whitney or perhaps ArThur Whitney, and possibly his account name.
Sounds right.
Additional examples at
https://github.com/louyx/aplus/blob/master/src/a/k.h
https://code.jsoftware.com/wiki/Essays/Incunabulum
How would you characterize that?
Heavy use of the C preprocessor and C defaults to embed a functional programming language. Language with a small number of core functions and ability to apply functions to lists of atoms. Aesthetically favoring short identifiers and minimal whitespace to create high semantic density. Eschew comments.
4 replies →
Whitney and I both worked in the 1970s for I.P. Sharp Associates, which used an email system written beautifully in APL by Leslie Goldsmith. Most user names were simply our initials. Ian Sharp was IPS, Leslie LHG, and Arthur ATW. (Middle name Taylor.) I’ve been SJT ever since (including to two wives) but was not smart enough to grab the domain: see 5jt.com.
This looks like a cult I would happily dive into. Does anyone care to ELI5? I tried to read the code but did not understand a single line
Arthur Whitney is showing how to build an array language using C in array language form.
AW is known for kdb+ which is often used in finance due to its extreme performance properties and ability for quants to quickly explore ideas.
Personally, to get a better handle on how array languages worked, I implemented KlongPy which is a python implementation of Klong, which descends from K (which AW wrote).
You have to play with this stuff to understand it intuitively.
Ha! For those of us who were in the Morgan Stanley Fixed Income group for many years in the 1990s and 2000s AW is known for APLus which was used extensively for modeling and application work.
1 reply →
FWIW, kdb+ is not that extremely performant - there's a lot of things that could be faster, and a lot of limitations that mean that you often would be better of not using a DB at all (or to use another DB and just pull everything you might need into memory). There is/was a tradeoff in that many things that would make it faster would require more code, and a cool thing about q/kdb+ is that it takes so little code you don't have I$ issues, but I think that's a tradeoff that doesn't make as much sense anymore in 2023.
What it's really great for is that it's really neatly integrated into the q language, which is great for exploratory programming, and it's fast enough not to get in the way.
6 replies →
Basically it is an APL-like language that is used (or was, I think it has moved on to a successor language called q now) in the proprietary kdb+ system used by some financial companies. Even if you aren't into finance it is a fun language to play around with.
https://en.wikipedia.org/wiki/Kdb%2B
One of us.. one of us..
But seriously, if you haven’t given it a try you should. You won’t be disappointed.
array languages are very fascinating and I'd love spending more time learning about them
I read that they are (K in particular) used in finance...would you reckon it would be easier to find job in that field having that in the CV? I know I cannot compete on the C++/Python side but maybe as a skilled K developer I could sneak in
I started with the header file. It was a gentle warning of the madness in "a.c". At least it's well commented.
I believe that all one letter names for a language are already taken... Even Ć
I've just learned that Ć is no longer a viable name (thankfully). Now it's Fusion Programming Language (or fut?) https://github.com/fusionlanguage/fut
My god, AW is even willing to #include <stdio.h> now!
[dead]
[dead]
[flagged]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
Might not, since for this kind of thing most likely either you give up in couple minutes (which it is not a heinous since it does not waste your time anyway) or you just read it.
The distinction between “seeing” code and “reading” it is right on point.
Ha! The guy who wrote this is the same guy who invented ~~APL~~ a number of APL inspired languages (Edit: He did not invent APL. Thanks for the corrections!), so I suspect he may just be built different.
https://www.jsoftware.com/ioj/iojATW.htm
Have you seen anything written in K itself? Here's a program to calculate primes:
I'm sure ATW knows a lot of APL, but APL itself was created by Ken Iverson when ATW was just a small child. https://en.wikipedia.org/wiki/Kenneth_E._Iverson
APL was invented by Kenneth Iverson, the other person mentioned on the page, not Arthur Whitney.
I was not convinced of the readability of APL, but compared to its successors which tried to stick to ASCII, I've learned to appreciate the merits of an extended character set.
A program written in Unix dc(1) would look as weird as that.
How would you implement the same functionality?
you must have seen a lot of code.
Your code better? What metric?
[flagged]