Skip to main content

Hugging Face

This notebook shows how to get started using Hugging Face LLM's as chat models.

In particular, we will:

  1. Utilize the HuggingFaceEndpoint integrations to instantiate an LLM.
  2. Utilize the ChatHuggingFace class to enable any of these LLMs to interface with LangChain's Chat Messages abstraction.
  3. Explore tool calling with the ChatHuggingFace.
  4. Demonstrate how to use an open-source LLM to power an ChatAgent pipeline

Note: To get started, you'll need to have a Hugging Face Access Token saved as an environment variable: HUGGINGFACEHUB_API_TOKEN.

%pip install --upgrade --quiet  langchain-huggingface text-generation transformers google-search-results numexpr langchainhub sentencepiece jinja2

1. Instantiate an LLM​

HuggingFaceEndpoint​

from langchain_huggingface import HuggingFaceEndpoint

llm = HuggingFaceEndpoint(
    repo_id="meta-llama/Meta-Llama-3-70B-Instruct",
    task="text-generation",
    max_new_tokens=512,
    do_sample=False,
    repetition_penalty=1.03,
)
API Reference:HuggingFaceEndpoint

HuggingFacePipeline​

from langchain_huggingface import HuggingFacePipeline

llm = HuggingFacePipeline.from_model_id(
    model_id="HuggingFaceH4/zephyr-7b-beta",
    task="text-generation",
    pipeline_kwargs=dict(
        max_new_tokens=512,
        do_sample=False,
        repetition_penalty=1.03,
    ),
)
API Reference:HuggingFacePipeline

To run a quantized version, you might specify a bitsandbytes quantization config as follows:

from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_use_double_quant=True
)

and pass it to the HuggingFacePipeline as a part of its model_kwargs:

pipeline = HuggingFacePipeline(
    ...

    model_kwargs={"quantization_config": quantization_config},
    
    ...
)

2. Instantiate the ChatHuggingFace to apply chat templates​

Instantiate the chat model and some messages to pass.

Note: you need to pass the model_id explicitly if you are using self-hosted text-generation-inference

from langchain_core.messages import (
    HumanMessage,
    SystemMessage,
)
from langchain_huggingface import ChatHuggingFace

messages = [
    SystemMessage(content="You're a helpful assistant"),
    HumanMessage(
        content="What happens when an unstoppable force meets an immovable object?"
    ),
]

chat_model = ChatHuggingFace(llm=llm)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Check the model_id

chat_model.model_id
'meta-llama/Meta-Llama-3-70B-Instruct'

Inspect how the chat messages are formatted for the LLM call.

chat_model._to_chat_prompt(messages)
"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou're a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWhat happens when an unstoppable force meets an immovable object?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

Call the model.

res = chat_model.invoke(messages)
print(res.content)
One of the classic thought experiments in physics!

The concept of an unstoppable force meeting an immovable object is a paradox that has puzzled philosophers and physicists for centuries. It's a mind-bending scenario that challenges our understanding of the fundamental laws of physics.

In essence, an unstoppable force is something that cannot be halted or slowed down, while an immovable object is something that cannot be moved or displaced. If we assume that both entities exist in the same universe, we run into a logical contradiction.

Here

3. Explore the tool calling with ChatHuggingFace​

text-generation-inference supports tool with open source LLMs starting from v2.0.1

Create a basic tool (Calculator):

from langchain_core.pydantic_v1 import BaseModel, Field


class Calculator(BaseModel):
    """Multiply two integers together."""

    a: int = Field(..., description="First integer")
    b: int = Field(..., description="Second integer")

Bind the tool to the chat_model and give it a try:

from langchain_core.output_parsers.openai_tools import PydanticToolsParser

llm_with_multiply = chat_model.bind_tools([Calculator], tool_choice="auto")
parser = PydanticToolsParser(tools=[Calculator])
tool_chain = llm_with_multiply | parser
tool_chain.invoke("How much is 3 multiplied by 12?")
API Reference:PydanticToolsParser
[Calculator(a=3, b=12)]

4. Take it for a spin as an agent!​

Here we'll test out Zephyr-7B-beta as a zero-shot ReAct Agent.

The agent is based on the paper ReAct: Synergizing Reasoning and Acting in Language Models

The example below is taken from here.

Note: To run this section, you'll need to have a SerpAPI Token saved as an environment variable: SERPAPI_API_KEY

from langchain import hub
from langchain.agents import AgentExecutor, load_tools
from langchain.agents.format_scratchpad import format_log_to_str
from langchain.agents.output_parsers import (
    ReActJsonSingleInputOutputParser,
)
from langchain.tools.render import render_text_description
from langchain_community.utilities import SerpAPIWrapper

Configure the agent with a react-json style prompt and access to a search engine and calculator.

# setup tools
tools = load_tools(["serpapi", "llm-math"], llm=llm)

# setup ReAct style prompt
prompt = hub.pull("hwchase17/react-json")
prompt = prompt.partial(
    tools=render_text_description(tools),
    tool_names=", ".join([t.name for t in tools]),
)

# define the agent
chat_model_with_stop = chat_model.bind(stop=["\nObservation"])
agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_log_to_str(x["intermediate_steps"]),
    }
    | prompt
    | chat_model_with_stop
    | ReActJsonSingleInputOutputParser()
)

# instantiate AgentExecutor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke(
    {
        "input": "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?"
    }
)


> Entering new AgentExecutor chain...
Question: Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?

Thought: I need to use the Search tool to find out who Leo DiCaprio's current girlfriend is. Then, I can use the Calculator tool to raise her current age to the power of 0.43.

Action:

{ "action": "Search", "action_input": "leo dicaprio girlfriend" }

Leonardo DiCaprio may have found The One in Vittoria Ceretti. “They are in love,” a source exclusively reveals in the latest issue of Us Weekly. “Leo was clearly very proud to be showing Vittoria off and letting everyone see how happy they are together.”Now that we know Leo DiCaprio's current girlfriend is Vittoria Ceretti, let's find out her current age.

Action:

{ "action": "Search", "action_input": "vittoria ceretti age" }

25 yearsNow that we know Vittoria Ceretti's current age is 25, let's use the Calculator tool to raise it to the power of 0.43.

Action:

{ "action": "Calculator", "action_input": "25^0.43" }

Answer: 3.991298452658078Final Answer: Vittoria Ceretti, Leo DiCaprio's current girlfriend, when raised to the power of 0.43 is approximately 4.0 rounded to two decimal places. Her current age is 25 years old.

> Finished chain.
{'input': "Who is Leo DiCaprio's girlfriend? What is her current age raised to the 0.43 power?",
 'output': "Vittoria Ceretti, Leo DiCaprio's current girlfriend, when raised to the power of 0.43 is approximately 4.0 rounded to two decimal places. Her current age is 25 years old."}

Wahoo! Our open-source 7b parameter Zephyr model was able to:

  1. Plan out a series of actions: I need to use the Search tool to find out who Leo DiCaprio's current girlfriend is. Then, I can use the Calculator tool to raise her current age to the power of 0.43.
  2. Then execute a search using the SerpAPI tool to find who Leo DiCaprio's current girlfriend is
  3. Execute another search to find her age
  4. And finally use a calculator tool to calculate her age raised to the power of 0.43

It's exciting to see how far open-source LLM's can go as general purpose reasoning agents. Give it a try yourself!

Was this page helpful?


You can leave detailed feedback on GitHub.