LLMs (Large Language Models) are AI models that have been trained on massive datasets comprised mostly of text and code. As such, it’s no surprise that they are able to generate natural sounding text responses, but one of the biggest surprises that no one really expected is that these AI models also have shown incredible abilities to reason their way through problems to come up with answers even though they were not explicitly taught to do so.
Taking this ability into account, and combining it with the general approach we’ve covered before called Prompt Engineering, we can use a technique called Chain of Thought, or CoT, to help the AI agent we’re working with to produce more accurate answers to more complex logical problems than if we simply ask the LLM to provide answer after we supply it with a more complex logical question to solve.
Helping the LLM break down the problem
The key to CoT prompt engineering is to instruct the LLM on how it should break down a more complex problem into smaller parts.
For example, let’s consider the following problem:
A man walks into a bar and orders a drink. He puts a $50 bill on the counter and the bartender gives him back $4 in change. The man then orders another drink and puts a $20 bill on the counter. The bartender gives him back $6 in change. How much did the man pay for the two drinks?As a human, your approach to solving this problem probably goes something like this:
- The man buys two drinks so the total he paid is the cost of the first drink plus the cost of the second drink.
- The cost of the first drink is the difference between what the man paid and the change the man received back.
- The main paid $50 and received back $4 so the cost of the first drink was $46.
- The cost of the second drink was the difference between the $20 he paid and the $6 he got back in change, so the cost of that drink was $14
- The total of the two drinks was $14 + $46
- The main paid a total of $60 for the two drinks
LLM behavior without CoT
Let’s see how ChatGPT using gpt3-turbo performs at figuring this out when we just ask it this question with no additional context.
As you can see, ChatGPT makes a reasonable guess at answering our question by totaling up the amount the man paid, but disregards the amount of change returned. And to be fair, ChatGPT may get the answer right some portion of the time. However, we can improve the reliability of ChatGPT’s performance with this problem by giving the LLM more explicit instructions for how to go about solving it.
Providing Chain of Thought Instructions
To use a chain of thought approach to prompt engineering, we can modify our original prompt with instructions on how we want ChatGPT to break the problem down and how we would like for the system to try to solve the problem. To do this, we follow the process we described earlier by telling ChatGPT how to break the problem down and how to use the solutions to the simpler sub-problems to answer the more complex main question. In this case, we could provide a prompt that looks something like:
Prompt: A man walks into a bar and orders a drink. He puts a $50 bill on the counter and the bartender gives him back $4 in change. The man then orders another drink and puts a $20 bill on the counter. The bartender gives him back $6 in change. How much did the man pay for the two drinks?
Instructions:
1. First consider how much the first drink costs by considering how much the man paid and how much change the man received.
2. Then consider how much the second drink cost by following the same approach.
3. Then add up the costs to the two drinks.
4. The price of the two drinks combined is how much the man paid for the drinks.Now, given this additional information, ChatGPT is able to provide a more clear breakdown of how it constructed its answer and will be able to more reliably produce a correct answer:
Asking for Chain of Thought
Another technique we can use is to ask the LLM to provide the subproblems and overall approach to solving the problem rather than just jumping straight to the answer. In our previous example, our prompt may look something like this:
A man walks into a bar and orders a drink. He puts a $50 bill on the counter and the bartender gives him back $4 in change. The man then orders another drink and puts a $20 bill on the counter. The bartender gives him back $6 in change. How much did the man pay for the two drinks?
Provide a break down the steps that you should follow to solve this problem.Interestingly, without actually providing the instructions to the LLM, just by asking it to explain the steps itself, we’re able to get a much more reliable, accurate response:
When to use Chain of Thought?
Chain of thought can be a useful technique in many situations where you’re using a tool like ChatGPT to assist you with solving more complex problems. CoT is really just a fancy way of expressing to the LLM the steps that you would go through as a human and having it follow the same process. For simple cases like we’ve looked at here, we may be able to provide all the context we need in a single prompt, but for more complex tasks, you may want to use this technique over a series of interactions.
For example, if you are a marketing manager putting together a marketing strategy for your organization, you may want to start by asking the LLMs what steps you should follow to build a marketing strategy. Based on the response, if you want the LLM to take a different approach you can state that up front before moving on to the completing the first step in the process together. For example, the LLM may say that step one is to identify your target market and you may feel that identifying the key value propositions for your product is a better starting point in your eyes. You can then explain your thinking back to the LLM until you agree on the steps that you will take to develop the strategy.
Likewise, when using an LLM to help you write an article, you may want to start with identifying the key points you want to communicate, then move on to an outline of the article, then decide on how you will transition between sections, then write a draft each section. Doing this can help you produce an article that reflects what you are trying to communicate rather than whatever the LLM decides it should produce.
Conclusion
Chain of thought is a useful prompt engineering technique that can enable you to solve more complex problems using an LLM-based AI agent like ChatGPT or Google Bard than you would be able to otherwise. Consider this technique any time you have a problem that can be broken down into a series of subproblems to solve. Depending on the scenario, you can often leverage this technique simply by asking the LLM to focus on doing the step by step breakdown before answering, and in certain cases you can help the LLM to do a better job of responding by telling it how you would break down the problem, and then stepping through the problem solving process. Combined with more basic prompt engineering techniques, CoT can greatly improve your results and productivity when working with LLMs.
Add comment