Comment by lettucegoblin
2 days ago
I think the coherence might be coming from the filtering, not the fibonacci intervals. my thinking is that the process looks like this: your fibonacci script finds a big list of candidate words -> your grammar rules arrange them -> sentencetransformers aggressively filters this list and picks the single most semantically coherent sentence. so is it possible that sentencetransformers is doing all the heavy lifting for creating coherence? have you ever looked at the raw, unfiltered candidate words/sentences to see if they have any coherence on their own? on the "french toast" example, could this be a case of language pareidolia(e.g. seeing faces in clouds)? the model selects "piney connubial produit" because its math is the closest to the input -> your brain, being a great pattern matcher, creates a story to connect the words. so is the meaning actually being generated by the fibonacci part, or is it just being found by the filter and then interpreted by you? with the punctuation model, i'm guessing it's just learning simple surface-level patterns, right? like, it learns: text contains "but" -> place comma before "but". how does it handle complex sentences that don't follow the simple patterns in your 4mb of training data? does that break down? the comparison to bert seems off because they're solving the problem in fundamentally different ways. tofigure out if the fibonacci part is actually meaningful, have you thought about running some controls? for example, what happens if you replace the fibonacci intervals with random intervals? does the quality of the final, filtered output get worse? what if you just looked at the raw word lists themselves? is a list generated by your fibonacci model measurably more coherent than a list of words just pulled randmly from the text?
Thank you for all this! I ran some controls in a separate study more recently - the (tiny!) results seemed interesting to me:
https://github.com/henrygabriels/FMLLM/blob/main/improving_l...
This compares the impact of Fib-interval 'word clouds' appended to prompts for creative writing tasks to randomly-selected 'word clouds' appended, to 'no words appended to prompt'.
Checking semantic coherence between Fib intervals vs random intervals is a super interesting idea, thank you. I'll make this priority 1, because you're right - that changes everything.
And yes, I think the sentence generation is the least interesting part; as other commenters have pointed out, it's basically Markov Chain (a term I'd not heard before) + sentencetransformers. It wasn't working - to generate those sentences - with a large volume of available words, however. Those were capped at 50 words available + stop words. I'm not sure if this changes anything.
I didn't want to oversell the sentence-generation aspect, but it seems I may have. I did state twice in the doc "I do think sometimes the human mind sees what it wants to see, especially with language", but I should have definitely made this more clear!
On 'how does punctuation model handle longer sequences', I depunctuated and repunctuated your comment:
>i think the coherence might be coming from the filtering not the fibonacci intervals. my thinking is that the process looks like this your fibonacci script, finds a big list of candidate words. your grammar rules, arrange them. sentencetransformers aggressively filters this list and picks the single most semantically coherent sentence, so is it possible that sentencetransformers is doing all the heavy lifting for creating coherence have you ever looked at the raw unfiltered candidate wordssentences to see if they have any coherence on their own on the french, toast. example, could this be a case of language pareidoliaeg seeing faces, in clouds. the model selects, piney connubial produit because its math is the closest to the input your brain. being a great pattern matcher creates a story, to connect the words. so is the meaning, actually being generated, by the fibonacci part or is it just being found by the filter and then interpreted by you with the punctuation model im guessing its just learning simple. surfacelevel patterns right like it learns text. contains but place. comma before but how does it handle complex sentences, that dont follow. the simple. patterns in your 4mb of training data. does that break down the comparison, to bert seems off because theyre solving the problem. in fundamentally different ways, tofigure out if the fibonacci part is actually meaningful have you thought about running some controls. for example, what happens if you replace the fibonacci intervals. with random. intervals. does the quality of the final filtered output get worse what if you just looked at the raw word. lists themselves. is a list generated, by your fibonacci model measurably more coherent than a list of words. just pulled randmly from the text.
It doesn't have any punctuation characters more 'complex' than what you see in that output, hence it smashing 'wordssentences' together. I guess a next step = add these characters, see results.
I'm going to come back to your comment, and respond to more of your points, but I also want to reply to some of the others. I really appreciate the extensive input!