to get the ball rolling usually yeah

I've always been a little skeptical of the up/downvote mechanism on any social media platform. It makes it so easy to weaponize the human bias towards group conformity, by doing exactly what you described.

I won't try to argue the voting has no benefits. It can be helpful to reduce the reach of truly bad faith posts for example, and everyone loves the little dopamine hit of seeing one's own posts upvoted - me included. Just that it also has real drawbacks, and I'm not sure whether the good outweighs the bad.

At this point, I hope Nv etc realize that even if selling AI cards to data centers gets them 10 times the profit per unit, it really is best for them in the long run to have a healthy and vibrant gamer and enthusiast market too. It's never good to have all your eggs in one basket.

Or be a 90s computer text adventure

Zork on steroids!

Thanks for your comments and thoughts! I appreciate hearing from more experienced people.

I feel like a little bit of prompt engineering would go a long way.

Yah, probably so. I tried to write a system prompt to steer the model toward what I wanted, but it's going to take a lot more refinement and experimenting to dial it in. I like your idea of asking it to be unforgiving about rules. I hadn't put anything like that in.

That's a great idea about putting a D&D manual, or at least the important parts, into a RAG system. I haven't tried RAG yet but it's on my queue of matters to learn. I know what it is, I just haven't tried it yet.

I've for sure seen that the quality of output starts to decline about 16K context, even on models that claim to support 128K. Also, I feel like the system prompt seems more effective when there are only let's say 4K context tokens so far. After the context grows, the model becomes less and less inclined to follow the system prompt. I've been guessing this is because as the context grows, any given piece of it becomes more dilute, but I don't really know.

For those reasons, I'm trying to use summarization to keep the context size under control, but I haven't found a good approach yet. SillyTavern has an auto summary injecting system, but either I'm misunderstanding it, or I don't like how it works, and I end up doing it manually.

I tried a few CoT models, but not since I moved to ST as a front end. I was using them with the standard llama-server web interface, which is a rather simple affair. My problem was that the thinking output seemed to spam up the context, leaving me much less ctx space for my own use. Each think block was like 500-800 tokens. It looks like ST might have an ability to only keep the most recent think block in the context, so I need to do more experimenting. The other problem I had was that the thinking could just take a lot of time.

What was your setup for this experiment?

I'm using llama.cpp + sillytavern. I'm very much in learning mode with ST however, so I'm confident I could be using it in a more effective manner than I know how to at the moment. It seems like koboldcpp + ST ought to be similar to what I'm doing.

but it’s no fun when the LLM simply says “yeah, sure whatever.” I

I hear ya. LLMs tend to heavily tilt toward what the user wants, which is not ideal for an RPG.

Have you tried any of the specialized RPG models? The one I'm using now has, at least twice so far, put me into a situation where I felt my party (2 chars, me and the AI) were going to die unless we ran away. We just finished a very difficult fight, used everything at our disposal, and sustained several serious injuries in the process. Then an even more powerful foe appeared, and it really felt that was going to be the end unless we ran. Would it really have killed us? I can't say, but I did get a genuine sense of that. It might help that in the system prompt, I had put this:

The story should be genuinely dangerous and frightening, but survivable if we use our wits.

I have the feeling the generalist models are much more tilted in the "yeah, sure, whatever" direction. I tried at least one RPG focused model (Dan's dangerous winds, or something like that) which was downright brutal, and would kill me off right away with no opportunity for me to do anything about it. That wasn't fun for the opposite reason. But like you say, it's also not fun to have no risk and no boundaries to test one's mettle. The sweet spot is can be elusive.

I'm thinking that a non-LLM rules system around an LLM for descriptive purposes could really help here too, to enforce a kind of rigor on the experience.

To add to my lame noob answer, I found this, which has a better rundown of ollama vs llama.cpp. I don't know if it's considered bad form to link to ##ddit on lemmy, so ~~I'll just put the title here and you can search for it on there if you want~~ link added per comment from mutual_ayed below. There are a couple informative posts which are upvoted. "There is a big difference between use LM-Studio, Ollama, LLama.cpp?"

What’s the advantage over Ollama?

I'm very new to this so someone more knowledgeable should probably answer this for real.

My impression was that ollama somehow uses the llama.cpp source internally, but wraps it up to provide features like auto-downloading of models. I didn't care about that, but I liked the very tiny dependency footprint of llama.cpp. I haven't tried ollama for network inference.

There are other backends too which support network inference, and some posts allege they are better for that than llama.cpp is. vllm and ... exllama or something like that? I haven't looked into either of them. I'm running on inertia so far with llama.cpp, since it was so easy to get going and I'm kinda lazy.

I like this project. Very nice!

I haven't tried RAG yet, nor the fancy vector space whatsit which looks like it requires a specialized model(?) to create. I've been wanting to do something similar in spirit to your project here, but for an online RPG, so I dig this.