Jitter is a very interesting project! It's a kind of generic interpreter that is—as the name suggests—almost a JIT compiler. It's something more or less "in between". Technically it's more an interpreter + VM. But it uses all kinds of (I think novel) tricks to remove the overhead of interpretation, which makes the code at runtime almost look like what a JIT compiler would have produced.
Some more internal info than on the GNU page on how this works can be found in the following slides:
I'm curious as to whether the techniques used here are safe against Spectre-like attacks. There appears to be a heavy reliance on indirect branching. The slides here are dated just before Spectre was published, so it would not have been a concern at the time.
Oh, this is one I hadn't heard of. There are several different takes on this idea, and I've had the notion to make my own for some years, too (not enough free time...)
I feel like a good "how powerful is this tool, really?" test would be to model the Adobe PSD format with it. That format is bananas, and is what got me into this topic in the first place.
But for those interested, here are some other tools in this vein:
Thanks for this compendium. I originally wrote and released binspector, but have since moved to HexFiend, because I like the power Tcl affords as a template language. I too have rewritten specs for PSD too many times to count; it’s one of my favorite nerd-sniping pastimes. I’ll have to explore some of these other links, to find a good editor for my Windows colleagues.
I am not affiliated Synalyze It!, but have been a customer for several years. Generally, I've been happy with it, though it crashes occasionally. The support has been responsive when I've had licensing issues.
This is why I keep reading these threads, looking for something I can wrap my head around.
I started looking at Kaitai Struct not too long ago. I purposely started with a pretty simple format, but I think I'm not smart enough to figure out the input format. Also, I'm pretty sure I hate YAML.
I tried building Poke just last week, having resolved to try it since 1.0, but hit problems with the Boehm GC on my old Mac, and then what looks like Readline problems on my new Mac. I haven't given up, but I'm off to a slow start.
Hachoir, mentioned in another comment, looks promising, although the input looks suspiciously similar to a program I'd write myself.
Haven't watched the full video, but this looks similar ImHex (https://imhex.werwolv.net/), which also includes a pattern language thing to describe the structure of data. I used it once for a project, and it was useful when it worked, although I ran into some limitations when trying to model container formats.
Maybe it could do that and I just couldn't figure it out at the time, but if you have say a zip file with different file formats, you couldn't tell the language to switch between different structures based on like an index or a header that tells you the format of a subsection. It was a limitation of the pattern language.
I wonder if GNU poke is more advanced in that regard? A tool like this would be super useful for debugging custom binary formats, but some formats can get pretty complex.
I always liked Winhex for this type of thing. It could dissect a structure with a c-like syntax. I found it as shareware as a kid and have never used a hex editor I liked more.
wxHexEditor has a similar look and feel to WinHex, and similarly supports editing of extremely large files or raw devices. It's nowhere near as featureful but still very useful, and doesn't have the proprietary license.
It may not be ideal to praise commercial software in a GNU software thread, but I really enjoy using 010 Editor. It's one of the few programs I pay for. Its C-like template syntax (looks similar to GNU Poke syntax) makes reverse engineering binary formats easy and fun!
I love/hate that "pickle" is defined through usage. Curious to know if that name has any meaning (e.g. python pickles are for long-term storage of ephemeral data), but not curious enough to watch a video just in case that's covered.
In my opinion the core on which poke was made is actually much more interesting than the tool as such.
Poke is actually not only an editor, it's a domain specific language.
This language is built on top of something called GNU Jitter.
https://www.gnu.org/software/jitter/
Jitter is a very interesting project! It's a kind of generic interpreter that is—as the name suggests—almost a JIT compiler. It's something more or less "in between". Technically it's more an interpreter + VM. But it uses all kinds of (I think novel) tricks to remove the overhead of interpretation, which makes the code at runtime almost look like what a JIT compiler would have produced.
Some more internal info than on the GNU page on how this works can be found in the following slides:
http://ageinghacker.net/talks/jitter-slides--saiu--ghm2017--...
I'm curious as to whether the techniques used here are safe against Spectre-like attacks. There appears to be a heavy reliance on indirect branching. The slides here are dated just before Spectre was published, so it would not have been a concern at the time.
>I'm curious as to whether the techniques used here are safe against Spectre-like attacks
Because people will be compromising your GNU poke or Jitter code?
1 reply →
Oh, this is one I hadn't heard of. There are several different takes on this idea, and I've had the notion to make my own for some years, too (not enough free time...)
I feel like a good "how powerful is this tool, really?" test would be to model the Adobe PSD format with it. That format is bananas, and is what got me into this topic in the first place.
But for those interested, here are some other tools in this vein:
* fq - like jq for binary data: https://news.ycombinator.com/item?id=34434249
Thanks for this compendium. I originally wrote and released binspector, but have since moved to HexFiend, because I like the power Tcl affords as a template language. I too have rewritten specs for PSD too many times to count; it’s one of my favorite nerd-sniping pastimes. I’ll have to explore some of these other links, to find a good editor for my Windows colleagues.
Also
* Synalyze It! - Hex Editor for Mac: https://www.synalysis.net/
I am not affiliated Synalyze It!, but have been a customer for several years. Generally, I've been happy with it, though it crashes occasionally. The support has been responsive when I've had licensing issues.
This is why I keep reading these threads, looking for something I can wrap my head around.
I started looking at Kaitai Struct not too long ago. I purposely started with a pretty simple format, but I think I'm not smart enough to figure out the input format. Also, I'm pretty sure I hate YAML.
I tried building Poke just last week, having resolved to try it since 1.0, but hit problems with the Boehm GC on my old Mac, and then what looks like Readline problems on my new Mac. I haven't given up, but I'm off to a slow start.
Hachoir, mentioned in another comment, looks promising, although the input looks suspiciously similar to a program I'd write myself.
It's a bit overkill for only building Poke, but you can use Pkgsrc [1] to do so on macOS:
Bootstrap Pkgsrc:
git clone https://github.com/NetBSD/pkgsrc.git
cd pkgsrc/bootstrap
./bootstrap --unprivileged
And then build and install Poke:
cd ~/pkgsrc/editors/poke
bmake install package clean clean-depends
[1] https://pkgsrc.org
See also the recent talk the author gave at FOSDEM last month[1] for an introduction to the tool and its use cases.
[1]: https://fosdem.org/2023/schedule/event/bintools_poke/
IMO one of the most humorous talks at the event! :)
Haven't watched the full video, but this looks similar ImHex (https://imhex.werwolv.net/), which also includes a pattern language thing to describe the structure of data. I used it once for a project, and it was useful when it worked, although I ran into some limitations when trying to model container formats.
Maybe it could do that and I just couldn't figure it out at the time, but if you have say a zip file with different file formats, you couldn't tell the language to switch between different structures based on like an index or a header that tells you the format of a subsection. It was a limitation of the pattern language.
I wonder if GNU poke is more advanced in that regard? A tool like this would be super useful for debugging custom binary formats, but some formats can get pretty complex.
I always liked Winhex for this type of thing. It could dissect a structure with a c-like syntax. I found it as shareware as a kid and have never used a hex editor I liked more.
https://www.x-ways.net/winhex/
Interesting! I've heard of winhex before, but I never knew it had a templating language.
I've been using 010 Editor, which similarly lets you define templates with c-like syntax, run it against a file, and edit the fields from there.
(Cross platform too, which is nice. It is a paid product though)
https://www.sweetscape.com/010editor/
It's nice to see it's cross platform now. It was windows only back when I needed it.
wxHexEditor has a similar look and feel to WinHex, and similarly supports editing of extremely large files or raw devices. It's nowhere near as featureful but still very useful, and doesn't have the proprietary license.
https://github.com/EUA/wxHexEditor / http://www.wxhexeditor.org/
C-like syntax is a poor choice for data description. It's not flexible enough. This is why many newer DDLs were created.
It may not be ideal to praise commercial software in a GNU software thread, but I really enjoy using 010 Editor. It's one of the few programs I pay for. Its C-like template syntax (looks similar to GNU Poke syntax) makes reverse engineering binary formats easy and fun!
Check out ImHex! It's a FOSS alternative to 010 Editor.
What are the pros and cons of 010 Editor compared with GNU poke?
That would be a fun CLI for retro/tiny computers. Imagine something like that for a Z80 or 68hc11 vs the typical boot rom.
Can anyone comment on how this compares to Hachoire?
https://hachoir.readthedocs.io/en/latest/
Not the same thing, but I am reminded of Rez, Derez and the Resource Editor of ancient Mac daze.
Rez and DeRez still ship with xcode (or did the last time I checked)
Can it be taught to read serialized protobufs?
Yes. Go for it.
How fun!
Reminds me of ResEdit, from classic Mac OS, and its template resources.
I love/hate that "pickle" is defined through usage. Curious to know if that name has any meaning (e.g. python pickles are for long-term storage of ephemeral data), but not curious enough to watch a video just in case that's covered.
A pickle is a cucumber preserved in preservatives for long term storage, hence that name I think (and it start with P like python).
Just to expand on this:
> Pickling is the process of preserving or extending the shelf life of food by either anaerobic fermentation in brine or immersion in vinegar.
https://en.wikipedia.org/wiki/Pickling
[dead]
It was about time!
The capabilities of hex editors on Linux is strongly underwhelming when compared to, say, WinHex almost 30 years ago!
Yet another half-assed custom language that nobody wants to learn? What happened to Guile being THE GNU extension language?
Love the website design!
What a great name!