Comment by regus

8 hours ago

Jq's syntax is so arcane I can never remember it and always need to look up how to get a value from simple JSON.

56 comments

regus

Reply
iLemming  14 minutes ago

It's because .json itself has so much useless cruft it's often annoying to deal with. I am forever indebted for younger self forcing me to learn Clojure. Most of the time I choose not even bother with JSON anymore - EDN semantically so much cleaner - it's almost twice compact (yet lossless), it's far more readable (quotes and commas are optional), and easier to work with structurally. These days I'd use borkdude/jet or babashka and then deal with data in Clojure REPL - there I can inspect it from all sorts of angles, it's far easier to group, sort, slice, dice, map and filter through it. One can even easily visualize the data using djblue/portal. Why most people strangulate themselves with confusing jq operators unnecessarily, I would never understand. Clojure is not that hard, maybe learn some basics, it comes handy a lot. Even when your team doesn't have any Clojure code.

d35007  7 hours ago

That’s interesting! Can you say a little more? I find jq’s syntax and semantics to be simple and intuitive. It’s mostly dots, pipes, and brackets. It’s a lot like writing shell pipelines imo. And I tend to use it in the same way. Lots of one-time use invocations, so I spend more time writing jq filters than I spend reading them.

I suspect my use cases are less complex than yours. Or maybe jq just fits the way I think for some reason.

I dream of a world in which all CLI tools produce and consume JSON and we use jq to glue them together. Sounds like that would be a nightmare for you.

  • randusername  6 hours ago

    I'm not GP, I use jq all the time, but I each time I use it I feel like I'm still a beginner because I don't get where I want to go on the first several attempts. Great tool, but IMO it is more intuitive to JSON people that want a CLI tool than CLI people that want a JSON tool. In other words, I have my own preconceptions about how piping should work on the whole thing, not iterating, and it always trips me up.

    Here's an example of my white whale, converting JSON arrays to TSV.

    cat input.json | jq -S '(first|keys | map({key: ., value: .}) | from_entries), (.[])' | jq -r '[.[]] | @tsv' > out.tsv

    • nh23423fefe  3 hours ago 

          <input.json  jq -S  -r '(first | keys) , (.[]| [.[]]) | @tsv'
    <input.json  # redir
    jq
    -S           # sort
    -r           # raw string out
    '
    (first | keys) # header
    ,              # comma is generator
    (.[] |           # loop input array and bind to .
    [                # construct array
     .[]             # with items being the array of values of the bound object
     ])           
     | @tsv'        # generator binds the above array to . and renders to tsv

      2 replies →

    • figmert  4 hours ago

      Here's an easier to understand query for what you're trying to do (at least it's easier to understand for me):

          cat input.json | jq -r '(first | keys) as $cols | $cols, (.[] | [.[$cols[]]]) | @tsv'

      That whole map and from entries throws it off. It's not a good use for what you're doing. tsv expects a bunch of arrays, whereas you're getting a bunch of objects (with the header also being one) and then converting them to arrays. That is an unnecessary step and makes it a little harder to understand.

      2 replies →

  • rzzzt  29 minutes ago

    I'm often having trouble with figuring out in advance what the end result will be when processing an input array: an array of mapped objects or a series of self-contained JSON objects? Why? Which one is better? What if I would like to filter out some of the elements as part of the operation?

  • attentive  40 minutes ago

    > I dream of a world in which all CLI tools produce and consume JSON and we use jq to glue them together.

    that world exists and mature (powershell)

marginalia_nu  4 hours ago

I think the big problem is it's a tool you usually reach for so rarely you never quite get the opportunity to really learn it well, so it always remains in that valley of despair where you know you should use it, but it's never intuitive or easy to use.

It's not unique in that regard. 'sed' is Turing complete[1][2], but few people get farther than learning how to do a basic regex substitution.

[1] https://catonmat.net/proof-that-sed-is-turing-complete

[1] And arguably a Turing tarpit.

  • jasomill  2 hours ago

    I was just going to say, jq is like sed in that I only use 1% of it 99% of the time, but unlike sed in that I'm not aware of any clearly better if less ubiquitous alternatives to the 1% (e.g., Perl or ripgrep for simple regex substitutions in pipelines because better regex dialects).

    Closest I've come, if you're willing to overlook its verbosity and (lack of) speed, is actually PowerShell, if only because it's a bit nicer than Python or JavaScript for interactive use.

ivaniscoding  7 hours ago

Shameless plug, but you might like this: https://github.com/IvanIsCoding/celq

jq is the CLI I like the most, but sometimes even I struggled to understand the queries I wrote in the past. celq uses a more familiar language (CEL)

  • xpe  7 hours ago

    CEL looks interesting and useful, though it isn't common nor familiar imo (not for me at least). Quoting from https://github.com/google/cel-spec

        # Common Expression Language

    The Common Expression Language (CEL) implements common
    semantics for expression evaluation, enabling different
    applications to more easily interoperate.

    ## Key Applications

    - Security policy: organizations have complex infrastructure
      and need common tooling to reason about the system as a whole
    - Protocols: expressions are a useful data type and require
      interoperability across programming languages and platforms.

    • ivaniscoding  7 hours ago

      That’s some fair criticism, but the same page tells that the language wanted to have a similar syntax to C and JavaScript.

      I think my personal preference for syntax would be Python’s. One day I want to try writing a query tool with https://github.com/pydantic/monty

dcre  2 hours ago

Funny that everyone is linking the tools they wrote for themselves to deal with this problem. I am no exception. I wrote one that just lets you write JavaScript. Imagine my surprise that this extremely naive implementation was faster than jq, even on large files.

    $ cat package.json | dq 'Object.keys(data).slice(0, 5)'
    [ "name", "type", "version", "scripts", "dependencies" ]

https://crespo.business/posts/dq-its-just-js/

epr  3 hours ago

To fix this I recently made myself a tiny tool I called jtree that recursively walks json, spitting out one line per leaf. Each line is the jq selector and leaf value separated by "=".

No more fiddling around trying to figure out the damn selector by trying to track the indentation level across a huge file. Also easy to pipe into fzf, then split on "=", trim, then pass to jq

raydev  2 hours ago

Like I did with regex some years earlier, I worked on a project for a few weeks that required constant interactions with jq, and through that I managed to lock in the general shape of queries so that my google hints became much faster.

Of course, this doesn't matter now, I just ask an LLM to make the query for me if it's so complex that I can't do it by hand within seconds.

janderland  5 hours ago

JMESPath is what I wish jq was. Consistent grammar. It only issue is it lacks the ability to convert JSON to other formats like CSV.

hilti  6 hours ago

LOL ... I can absolutely feel your pain. That's exactly why I created for myself a graphical approach. I shared the first version with friends and it turned into "ColumnLens" (ImGUI on Mac) app. Here is a use case from the healthcare industry: https://columnlens.com/industries/medical

NSPG911  8 hours ago

I also genuinely hate using jq. It is one of the only things that I rely heavily on AI.

  • amelius  7 hours ago

    At that point why don't we ask the AI directly to filter through our data? The AI query language is much more powerful.

    • latexr  7 hours ago

      Because the output you get can have hallucinations, which don’t happen with a deterministic tool. Furthermore, by getting the `jq` command you get something which is reusable, fast, offline, local, doesn’t send your data to a third-party, doesn’t waste a bunch of tokens, … Using an LLM to filter the data is worse in every metric.

      6 replies →

    • imcritic  7 hours ago

      Because the input might be sensitive.

      Because the input might be huge.

      Because there is a risk of getting hallucinations in the output.

      Isn't this obvious?

      1 reply →

    • Shorel  7 hours ago

      You really need to go and learn about the concept of determinism and why for some tasks we need and want deterministic solutions.

      It's an important idea in computer science. Go and learn.

      4 replies →