Comment by yorwba

8 hours ago

You can build an unlimited-order Markov chain by, instead of pre-computing a table of counts for all possible contexts, using a substring-search index on the training data to count possible continuations on the fly: https://arxiv.org/abs/2401.17377 That paper uses suffix arrays, but more compact indices are possible: https://arxiv.org/abs/2506.12229

2 comments

yorwba

Reply
anthk  2 hours ago

Could that be implemented in Hailo for Perl (it's two commands away from a base install:)

- Install Cpanminus for Perl, some C compiler and sqlite3 just to be sure.

      cpanm -n local::lib 

     cpanm -n Hailo

     ~/perl5/bin/hailo -t corpus.txt -b chatbot.brn

    ~/perl5/bin/hailo -b chatbot.brn