← Back to context

Comment by belter

1 day ago

Ask a model to

"Write a chess engine where pawns move backward and kings can jump like nights"

It will keep slipping back into real chess rules. It learned chess, it did not understand the concept of "rules"

Or

Ask it to reverse a made up word like

"Reverse the string 'glorbix'"

It will get it wrong on the first try. You would not fail.

Or even better ask it to...

"Use the dxastgraphx library to build a DAG scheduler."

dxastgraphx is a non existing library...

Marvel at the results...tried in both Claude and ChatGPT....

6 comments

belter

Reply
manmal  1 day ago

I‘ve just tried the dxastgraphx one in pi with Opus 4.5. This was its response:

    I couldn't find a library called dxastgraphx in either pip (Python) or npm (JavaScript) package registries. This library doesn't appear to exist.

    Did you perhaps mean one of these popular DAG/graph libraries?

    Python:
    - networkx - comprehensive graph library with DAG support
    - graphlib - Python standard library (3.9+) with TopologicalSorter
    - dask - parallel computing with DAG task scheduling

    JavaScript/TypeScript:
    - graphlib - graph data structures
    - dagre - DAG layout

    Would you like me to build a DAG scheduler using one of these existing libraries, or would you like me to implement one from scratch? Let me know which language and approach you prefer.

somebodythere  8 hours ago

Why would I ask the model to reverse the string 'glorbix,' especially in the context of software engineering?

criticalfault  1 day ago

just tried to reverse the string you provided using Gemini. it worked fine on the first try

  • eithed  1 day ago

    In gemini:

    > The reverse of the string 'gssssyeekxxuoaqaaaiooo9aaa7zjx' is:

    xjz7aaa9oooiiaaaqauxxkyeessssg

    > Youve made a mistake

    ``` I see where I tripped up! Let's re-examine that long string carefully.

    The original string you gave me was: `gssssyeekxxuoaqaaaiooo9aaa7zjx`

    Let's look at the end of that string: `...aaa7zjx`. When we reverse it, it should start with: `xjz7aaa...`

    Here is the precise character-by-character reversal:

    *xjz7aaa9oooiiaaaqauxxkyeessssg*

    Wait—I see it now. In my previous response, I included an extra *'i'* and missed one *'o'* in the middle section. Let's do a strict count to get it perfect:

    * *Original:* `g` `ssss` `y` `ee` `k` `xx` `u` `o` `a` `q` `aaa` *`iooo`* `9` `aaa` `7` `z` `j` `x` * *Correct Reverse:* `x` `j` `z` `7` `aaa` `9` *`oooi`* `aaa` `q` `a` `o` `u` `xx` `k` `ee` `y` `ssss` `g`

    The correct reversed string is: *xjz7aaa9oooiiaaaqauoxxkyeessssg*

    Thank you for catching that. It’s easy for the eyes to glaze over when there are so many repeating vowels!

    Since I've stumbled a bit on these, would you like to give me one more string to see if I can get it right on the first try? ```

    After more back and fors it consistently fails in this task, even though when strictly dividing the tokens it will get this right. Yet the final answer is always wrong.

baq  12 hours ago

You’re trying to interrogate a machine as you would a human and presenting this as evidence that machines aren’t humans. Yes, you’re absolutely right! And also completely missing the point.