Prompting for Multi-Step Processes: Stepwise vs. Prompt Chaining

TL;DR: When working with complex, multi-step tasks in language models, 🔗 Prompt Chaining offers precise control by breaking the process into discrete steps, while Stepwise Prompting combines all steps into a single prompt for efficiency. Start with stepwise prompting for simpler tasks, and switch to prompt chaining if quality or consistency declines.

Prompting for multi-step processes

Prompting often involves instructing models to produce responses for complex, multi-step processes. Try to generate prompt for the following scenario.  

This scenario involves processing a document through three main steps:

• Summarizing each paragraph of the document in one line.
• Extracting key points from the entire document.

• Organizing the generated summary lines under their respective key points in the order they appear in the original document.
  This task requires a combination of summarization, key point extraction, and organizational skills. It's a complex process that involves understanding the document's structure, content, and main themes.

The two popular ways to handle such tasks are: Prompt Chaining, and Stepwise Prompting. 

Prompt chaining

Prompt chaining is an approach for refining outputs from large language models (LLMs) that involves using a series of discrete prompts to guide the model through different phases of a task. This method allows for a more structured and controlled approach to refinement, with each step having a specific focus. By breaking down complex tasks into smaller, more manageable prompts, prompt chaining provides greater flexibility and precision in directing the LLM's output. For example, in a text summarization task, one prompt might focus on extracting key points, another on organizing them coherently, and a final prompt on polishing the language.

Stepwise prompting

Stepwise prompting, on the other hand, integrates all phases of the task within a single prompt. This approach attempts to guide the LLM through the entire process in one go, challenging the model to generate a longer and more complex output based on a single set of instructions. While simpler to implement, stepwise prompting may be less effective for complex tasks that benefit from a more granular approach. For instance, when summarizing a lengthy academic paper, a single stepwise prompt might struggle to capture all the nuances of content selection, organization, and style that separate prompts in a chain could address individually.

Further, you can improve the clarity by assigning step names followed by step explanation. For example: 
  1) Analysis: Analyze the given text for key themes.
  2) Summary: Summarize the themes identified in `Analysis`.
  3) Conclusion: Draw conclusions based on the `Summary`.

The use of backticks to reference previous outputs leaves no ambiguity.

When to Use Step Names in Stepwise Prompts?
Use step names when the output of a step will be referenced later in the prompt or in subsequent prompts. Step names help in organization and add to clarity, especially for tasks with multiple interdependent steps. Additionally, it can help the model more effectively associate output variables with the specific steps in which they are extracted.

When Step Names Might Be Optional?
1. Simple, Linear Tasks: For straightforward tasks with clear progression, step names might be unnecessary.
2. Short Prompts: In brief prompts with only 2-3 steps, numbering alone might suffice.
Best Practices
- Be consistent: If you use step names, use them for all steps in the prompt.
- Keep names short and descriptive: Use clear, concise labels that indicate the purpose of each step.

Illustration of Stepwise prompting and Prompt Chaining: Generate document outline

Let's create prompts for the scenario explained earlier using both techniques:

Prompt Chaining:

For prompt chaining, we'll break down the task into three separate prompts, each focusing on a specific subtask.

Prompt 1 (Paragraph Summarization):

You are tasked with summarizing a document. For each paragraph in the given document, create a one-line summary that captures its main idea. Please provide these summary lines in a numbered list, with each number corresponding to the paragraph number in the original document.

Prompt 2 (Key Point Extraction):

Based on the entire document, identify and list the main key points. These should be the overarching themes or crucial ideas that span multiple paragraphs. Present these key points in a bulleted list.

Prompt 3 (Organization):
You will be provided with two lists: one containing one-line summaries of each paragraph, and another containing key points extracted from the document.

Go through the `Document`. Your task is to organize the summary lines from `Summary list` under their most relevant key points from `Key point list`. Maintain the original order of the summary lines within each key point. Present the result as a structured list with key points as main headings and relevant summary lines as sub-points. Output can be in the following format:
**<key point 1>
<summary line 1>
<summary line 2>
**<key point 2>
<summary line 3>
<summary line 4>
<summary line 5>

Ensure that all summary lines are included and that they maintain their original numbering order within each key point category.

Document: It is the document that needs to be converted into topic-wise `summary list`
Summary list: It is a list of one-line summaries of each paragraph in the `document`
Key point list: It is a list of key points covered in the `Document`

Stepwise prompting:

The prompt below demonstrates how we can replicate the three distinct steps previously used in prompt chaining within a single step by using stepwise prompting.

You are tasked with analyzing, summarizing, and organizing the content of a given document. Please follow these steps in order:
1. Read document: Carefully read through the entire document.
2. Summary list: Create a one-line summary for each paragraph, capturing its main idea. Number these summaries according to the paragraph they represent.
3. Key point list: Identify the main key points of the entire document. These should be overarching themes or crucial ideas that span multiple paragraphs.
4. Key point wise Summary list: Organize the `Summary list` under their most relevant key points from `Key point list`. Maintain the original order of the summary lines within each key point. Present the result as a structured list with key points as main headings and relevant summary lines as sub-points. 
   The output should be in the following format:
   **<key point 1>
   <summary line 1>
   <summary line 2>
   **<key point 2>
   <summary line 3>
   <summary line 4>
   <summary line 5>

Ensure that all summary lines are included and that they maintain their original numbering order within each key point category.

Missing paragraphs or lines

Notice the last line: Ensure that all summary lines are included and that they maintain their original numbering order within each key point category.

Despite our repeated efforts to ensure the AI model analyzes every paragraph and line, it continues to overlook some. Models may consider multiple paragraphs as one paragraph, or may skip paragraphs completely. This is much common problem when processing long documents. Similarly, if you have to process each line in a paragraph, model may combine lines or skip lines.

The solution is you should consider breaking the document into smaller chunks and process each chunk separately. Further, I've seen better results if you provide numbered list of paragraphs, and ask it to generate output as the numbered list corresponding to each paragraph. This makes it easier for the AI model to keep track of paragraphs. Same goes for line processing within paragraphs.

Stepwise prompting vs Prompt Chaining

Here's a comparison of prompt chaining and stepwise prompting in table format: