First run-through of langchain + ollama: a few pitfalls notes

Version and Configuration#

Software Version:

langchain                 0.3.25                   pypi_0    pypi
langchain-community       0.3.25                   pypi_0    pypi
langchain-core            0.3.65                   pypi_0    pypi
langchain-ollama          0.3.3                    pypi_0    pypi
langchain-openai          0.3.24                   pypi_0    pypi
langchain-text-splitters  0.3.8                    pypi_0    pypi

python version: 3.10.16

Requirements#

An episodic task needs to be performed. In each step, first, a solution is obtained using traditional methods. Then, a large language model is used to optimize the solution to observe the final effect.

Implementation#

File Structure

src/llm
├── agents.py
├── keys.toml
├── models.py
├── prompts
│   ├── __init__.py
│   ├── [xxxx] // your scenario name
│   │   ├── human.txt
│   │   └── system.txt
│   └── utils.py
└── tools.py

First, use it locally on Ollama, and after it works, consider integrating a more powerful model. For example, I initially used the qwen2:7b model.

# test/test_llm.py
from langchain.agents import create_structured_chat_agent

from src.llm.models import get_model
from src.llm.prompts.utils import load_prompts

def test_llm():
    model = get_model("ollama", "qwen2:7b")
    prompt = load_prompts(scenario="routing")

    agent = create_structured_chat_agent(model, [], prompt)
    result = agent.invoke({"input": "hello", "intermediate_steps": []})
    print(result)

# models.py
from langchain_ollama import ChatOllama
from langchain_openai import ChatOpenAI

def get_model(provider: str, model: str, temperature: float = 0.3):
    if provider == "ollama":
        return ChatOllama(
            model=model, temperature=temperature, base_url="http://localhost:11434" 
        ) # Note: when passing parameters, the llm should be passed to the model parameter, otherwise a Validation Error will occur
    elif provider == "openai":
        return ChatOpenAI(...) # omitted
    else:
        raise ValueError("unsupported provider")

# prompts/utils.py
from pathlib import Path

from langchain.prompts import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)

def load_prompts(scenario: str):
    path = Path(__file__).parent / scenario

    with open(path / "system.txt", "r", encoding="utf-8") as f:
        s_template = f.read()
        s_prompt = SystemMessagePromptTemplate.from_template(s_template)

    with open(path / "human.txt", "r", encoding="utf-8") as f:
        h_template = f.read()
        h_prompt = HumanMessagePromptTemplate.from_template(h_template)

    return ChatPromptTemplate.from_messages([s_prompt, h_prompt])

Prompt File, reference link configuration https://smith.langchain.com/hub/hwchase17/structured-chat-agent

# human.txt

{input}

{agent_scratchpad}
(reminder to respond in a JSON blob no matter what)

# system.txt

Respond to the human as helpfully and accurately as possible. You have access to the following tools:
{tools}

Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).
Valid "action" values: "Final Answer" or {tool_names}
Provide only ONE action per $JSON_BLOB, as shown:


{{
  "action": $TOOL_NAME,
  "action_input": $INPUT
}}

Follow this format:

Question: input question to answer
Thought: consider previous and subsequent steps
Action:

$JSON_BLOB

Observation: action result

... (repeat Thought/Action/Observation N times)

Thought: I know what to respond
Action:

{{
  "action": "Final Answer",
  "action_input": "Final response to human"
}}

Begin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate. Format is Action:```$JSON_BLOB```then Observation

Pitfalls#

1. Version#

Langchain is still evolving, and there are certain interface differences between earlier and current versions that need special attention. Timeliness is crucial when searching for information online.

2. Prompts#

When using the create_structured_chat_agent function to create an agent, it is important to ensure that the prompt must contain the three placeholders: tools, tool_names, agent_scratchpad, wrapped in {}.

Otherwise, it will raise an error ValueError: Prompt missing required variables: {'agent_scratchpad', 'tool_names', 'tools'}

During parsing, this prompt template will be treated as an f-string, so the contents within {} will be automatically replaced with variable values.

Update 25/6/24#

Using traditional .txt as prompts has poor scalability (as it does not support comments). It is recommended to use jinja2 templates to manage prompts.

This way, comments can be supported, and placeholders can be parsed.

The jinja template uses {{...}} for placeholders, which is different from the {...} used in langchain for prompt placeholders. These two cannot be mixed; otherwise, parsing errors will occur.

For the system prompt, it is usually provided when the agent is initialized and will not be changed afterward. If parameters need to be passed, it is recommended to do so through the jinja template.

from langchain_core.prompts.string import jinja2_formatter

s_template = jinja2_formatter(
  Path(path / "system.jinja").read_text(encoding="utf-8"),
  ... # pass in parameters corresponding to placeholders  
)
s_prompt = SystemMessagePromptTemplate.from_template(s_template)

For the human prompt, which is the agent's input, it is usually updated in real-time for each step, so it can be loaded using jinja or passed in through the .invoke() method. However, note that the former requires placeholders to be written in {{...}} form and imported similarly to the system prompt. The latter should use the {...} style as placeholders and be imported directly as a string (as shown below).

h_template = Path(path / "human.jinja").read_text(encoding="utf-8")
h_prompt = HumanMessagePromptTemplate.from_template(h_template)

3. Invocation#

When executing agent.invoke(), it is necessary to provide the content of the input field as well as the value for the intermediate_steps field.

Otherwise, it will raise an error KeyError: 'intermediate_steps'

Update#

You can use AgentExecutor to wrap the agent, allowing for more fine-tuned customization and the ability to return debugging information.

## original code
agent = create_structured_chat_agent(model, [], prompt)
result = agent.invoke({"input": "hello", "intermediate_steps": []})

## updated code
agent = create_structured_chat_agent(model, [], prompt)
executor = AgentExecutor.from_agent_and_tools(
    agent=agent, tools=[], verbose=True, return_intermediate_steps=True
)
result = executor.invoke({"input": "hello"})

This allows for more precise customization without manually passing the intermediate_steps parameter.