Comment by networked

6 days ago

I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:

  gzipt \
      --corpus data/tinyshakespeare.txt \  
      --prompt $'MENENIUS:\n' \
      --length 200 \
      ;
  
  MENENIUS:
  MtLUMSeptuttyyyxyxyxyxyvyyyxyxyxyxyvyyyxyxyxyxywyvzyxyxyx
  yyxyyyxyxyxyxyxPlyxyxyxyxyxyxyxyxyxtoxzfTUS.zxzzzyzzzvzzz
  vzzzxvzyvyxyxyxyvyxyxyxyvy--,Vdvyxyxyxyxyxyxyxyxyxxy!zFlx
  zzyyxyxyxyvyxyxyxyvyySPffuyuy

Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?

Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."

9 comments

networked

maxidog 6 days ago

Did you check MiMo correctly performed this unfamiliar task before posting this comment?

networked 6 days ago

I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.

This was the main change for bzip2:

  @@ -33,19 +34,16 @@ def candidate_lengths(
       level: int = 9,
       pool: ThreadPoolExecutor | None = None,
   ) -> list[int]:
  -    """Compressed length of ``context + seq`` for each seq, sharing the context.
  +    """Compressed length of ``context + seq`` for each seq.
  
  -    Compresses ``context`` once into a ``compressobj``, then clones its encoder
  -    state per candidate and feeds only that candidate. Identical to
  -    ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
  -    match search over ``context`` happens a single time.
  +    Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
  +    snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
  +    whole block, so every candidate recompresses the full context. Threads
  +    still scale because ``bz2`` releases the GIL.
       """
  -    base = zlib.compressobj(level)
  -    head = len(base.compress(context))
  
       def length_for(seq: bytes) -> int:
  -        clone = base.copy()
  -        return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
  +        return len(bz2.compress(context + seq, level))
  
       if pool is not None:
           return list(pool.map(length_for, sequences))

jrmg 6 days ago

Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?
(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)

1 reply →

jeremyjh 6 days ago

So, you had an AI write code you don't understand, then posted output you don't understand in a comment on the internet for other humans to read?

Dylan16807 5 days ago
No, they used AI to write code they do understand, then posted interesting results they (partially) don't understand for other humans to see.
- jeremyjh 5 days ago
  
  Then why attribute the work to MiMo? People mostly only attribute the model when they've outsourced the understanding along with the coding. If I understand the code I don't bother to mention the tools I used to create it.