Hey everyone, I’m looking for a way to use an open source local large language model (LLM) on Linux, particularly on low-spec hardware like Raspberry Pi, to generate lengthy, coherent stories of 10k+ words from a single prompt. I recall reading about methods described in scientific papers such as “Re3: Generating Longer Stories With Recursive Reprompting and Revision”, announced in this Twitter thread from October 2022 and “DOC: Improving Long Story Coherence With Detailed Outline Control”, announced in this Twitter thread from December 2022. These papers used GPT-3, and since it’s been a while since then, I was hoping there might be something similar made using only open source tools. Does anyone have experience with this or know of any resources that could help me achieve long, coherent story generation with an open source LLM? Any advice or pointers would be greatly appreciated. Thank you!
Using a raspberry pi seems very underpowered, best case you will be limited to something like 4-7B models on a 8GB RPi4. You may need to configure it with very long timeouts and expect it to output something like a token every few minutes.
I ran a 6B model on a i7 without a GPU and it didn’t give good results before I got CUDA up and running. Probably because of timeout.
Raspberry Pi
This is a famously high-compute problem and you want to chuck it on a pi? Most llm models require a good GPU.
Just want to piggyback this. You will probably need more than 6gb vram to run good enough models with a acceptable speed and coherent output, but the more the better.
This sounds suspiciously like “I want AI to write my school essay” to me.
More generically, this sounds like “Let’s not put any effort into anything: the machine will do it for me.”
That is my opinion and I’m aware it’s a minority point of view these days. That’s why I fully expect to be modded down immediately after posting this. Because arguing with people who have a different opinion is also not in fashion anymore.
Let’s not put any effort into anything: the machine will do it for me
So you are not using a calculator, I presume? Only math done on abacus is not being lazy?
-
you need a gpu or gpu resources to run a GPT level model
-
stop thinking of it as “make me a story” and start thinking of it as “let’s make a story together”
First talk back and forth over the basic idea.
Then get a short outline.
Then work on each chapter of the outline.
Then repeat and refine.Try to keep overall queries under a certain character limit. Google what some good ranges are.
Make sure to save each conversation, and use the outline and previous chapter as the start of your conversation for making the next chapter.
The AI won’t write it for you, but it can be an amazing accelerator if you’re willing to put in the work.
-
