Empower business users with expert LLM insights – are you in?

I’m looking to harness the collective smarts of our Data Alchemy community. Who is interested in taking on the challenge of evolving a Large Language Model to process queries with the insight and depth of an expert?

## The problem with current LLMs

The more you integrate LLMs in your workflows, the clearer their shortcomings become. Let’s say you request a business plan for “an AI-driven e-commerce site”. It will jump straight into a carbon-copy template business buzzwords – Problem Definition, Unique Value Proposition, Early Adopters – without context or insights.

Why doesn’t the LLM pause and wonder: what does “AI-driven” mean to you? What stage of your business story are you in? It's the nuance, the back-and-forth of understanding that's missing—direct prompts often lead to a one-way street of uninspired affirmations.

In my talks with business people, a recurring theme is their need for an LLM that doesn’t churn out cookie-cutter answers, but crafts them with the considered questions of an expert analyst. Wouldn’t it be great if we were the ones to offer more useful LLMs to our customers?

## A new mental model

In May of this year, academics proposed a mental model where the AI imagines a team of experts discussing the problem, each representing a different point of view. They propose their ideas step by step to the group, up until the point that they realize they are wrong, which is when they leave:

“3 experts are discussing the question with a panel discussion, trying to solve it step by step, and make sure the result is correct and avoid penalty.”

What is it about this prompt that triggers AIs into more productive thinking patterns? This 'panel' construct reshapes the pattern, emphasizing critical engagement over rote response. Instead of locking onto the most expected answer, the AI is coaxed into a more layered and intricate mode of thinking. But does it do so structurally, for the rest of the conversation?

It’s my experience that the AI will stray from the initial prompt. If you don’t remind it from time to time to have three experts discuss the problem, it tends to fall back to pleasing the user with the most expected answer. Isn’t there a way to elevate the LLM’s mental model permanently?

This is where I step into the domain of speculation. An LLM’s mental model emerges from the training it receives. GPT-4 was famously given large datasets of text taken from the internet and trained to predict the next token in those datasets. It was then subjected to reinforcement learning from human feedback in order to display behavior acceptable to OpenAI.

Could we not train a Large Language Model always to ignore surface text patterns? Perhaps the dataset could consist specifically of discussions that question assumptions. Or maybe the reinforcement training could induce the LLM? (No: reinforcement learning aligns a model to a world view - brainwashing, in other words; it does not not nudge it to look at problems from various angles).

## The power of Instruction Tuning

The staggering cost of training foundational AI models is likely to demotivate enthusiasts like ourselves. With figures soaring beyond the $50 million mark as noted by OpenAI's Sam Altman, it's easy to feel sidelined in the race to develop cutting-edge AI. Right. So what else?

One possible way forward is Instruction Tuning. I found this GitHub repository aimed at testing the efficacy of Tree of Thought prompting. They reference a scientific paper that Instruction-Tuned an LLM on an M2 MacBook Air in 50 minutes. I don’t understand Instruction Tuning well enough, but it looks like a promising experiment.

So here’s my pitch to you: let’s work on Instruction Tuning an LLM to get it to give responses beyond the immediately obvious. Are you interested in taking on the challenge with me? Let’s do this project and build something enticing for our customers. Perhaps we learn something on the way as well!

61 comments