How to Specify the Generation Configuration when Using LLMs

Download Python Script

Python script/notebook for this guide.

Generation Configuration how-to script

Generation parameters, such as temperature, top-p, and the maximum number of output tokens, are important for achieving the desired performance with Large Language Models (LLMs). In WayFlow, these parameters can be configured with the LlmGenerationConfig class.

This guide will show you how to:

• Configure the generation parameters for an agent.

• Configure the generation parameters for a flow.

• Apply the generation configuration from a dictionary.

• Save a custom generation configuration.

Note

For a deeper understanding of the impact of each generation parameter, refer to the resources at the bottom of this page.

Basic implementation

Configure the generation parameters for an agent

Customizing the generation configuration for an agent requires the use of the following wayflowcore components.

from wayflowcore.agent import Agent
from wayflowcore.models.llmgenerationconfig import LlmGenerationConfig

The generation configuration can be specified when initializing the LLM using the LlmGenerationConfig class. This ensures that all the outputs generated by the agent will have the same generation configuration.

The generation configuration dictionary can have the following arguments:

• max_new_tokens: controls the maximum numbers of tokens to generate, ignoring the number of tokens in the prompt;

• temperature: controls the randomness of the output;

• top_p: controls the randomness of the output;

• stop: defines a list of stop words to indicate the LLM to stop generating;

• frequency_penalty: controls the frequency of tokens generated.

Additionally, the LlmGenerationConfig offers the possibility to set a dictionary of arbitrary parameters, called extra_args, that will be sent as part of the llm generation call. This allows specifying provider-specific parameters that might not be common to all.

Note

The extra parameters should never include sensitive information.

generation_config = LlmGenerationConfig(
    max_tokens=512,
    temperature=1.0,
    top_p=1.0,
    stop=["exit", "end"],
    frequency_penalty=0,
    extra_args={"seed": 1},
)

WayFlow supports several LLM API providers. You can pass the generation_config for each of them. Select an LLM from the options below:

OCI GenAIvLLMOllama

from wayflowcore.models import OCIGenAIModel

if __name__ == "__main__":

    llm = OCIGenAIModel(
        model_id="provider.model-id",
        service_endpoint="https://url-to-service-endpoint.com",
        compartment_id="compartment-id",
        auth_type="API_KEY",
    )

Important

API keys should not be stored anywhere in the code. Use environment variables and/or tools such as python-dotenv

Now, you can build an agent using the LLM as follows:

agent = Agent(llm=llm)
conversation = agent.start_conversation()
conversation.append_user_message("What is the capital of Switzerland?")
conversation.execute()
print(conversation.get_last_message())

Configure the generation parameters for a flow

Customizing the generation configuration for a flow requires the use of the following wayflowcore components.

from wayflowcore.flow import Flow
from wayflowcore.models.llmgenerationconfig import LlmGenerationConfig
from wayflowcore.property import StringProperty
from wayflowcore.steps import PromptExecutionStep, StartStep

Refer to the previous section to learn how to configure the generation parameters when initializing an LLM using the LlmGenerationConfig class.

You can then create a one-step flow using the PromptExecutionStep step.

start_step = StartStep(name="start_step", input_descriptors=[StringProperty("user_question")])
prompt_step = PromptExecutionStep(
    name="PromptExecution",
    prompt_template="{{user_question}}",
    llm=llm,
    generation_config=LlmGenerationConfig(temperature=0.8),
)
flow = Flow(
    begin_step=start_step,
    control_flow_edges=[
        ControlFlowEdge(source_step=start_step, destination_step=prompt_step),
        ControlFlowEdge(source_step=prompt_step, destination_step=None),
    ],
    data_flow_edges=[DataFlowEdge(start_step, "user_question", prompt_step, "user_question")],
)
conversation = flow.start_conversation(
    inputs={"user_question": "What is the capital of Switzerland?"}
)
conversation.execute()

Important

The generation_config parameter passed to the PromptExecutionStep overrides the LLM’s original generation configuration.

Advanced usage

The LlmGenerationConfig class is a serializable object. It can be instantiated from a dictionary or saved to one, as you will see below.

Apply the generation configuration from a dictionary

If you have a generation configuration in a dictionary (for example, from a JSON or YAML file), you can instantiate the LlmGenerationConfig class as follows:

config_dict = {
    "max_tokens": 512,
    "temperature": 0.9,
}

config = LlmGenerationConfig.from_dict(config_dict)

Save a custom generation configuration

If you would like to share your specific generation configuration, you can create a LlmGenerationConfig class instance and store it to a dictionary.

config = LlmGenerationConfig(max_tokens=1024, temperature=0.8, top_p=0.6)
config_dict = config.to_dict()

Agent Spec Exporting/Loading

You can export the assistant configuration to its Agent Spec configuration using the AgentSpecExporter. The following example exports the serialization of the flow defined above.

from wayflowcore.agentspec import AgentSpecExporter

serialized_assistant = AgentSpecExporter().to_yaml(flow)

Here is what the Agent Spec representation will look like ↓

Click here to see the assistant configuration.

JSONYAML

{
  "component_type": "Flow",
  "id": "fc3d10f4-5ee2-40d8-a580-0db6c44b0b39",
  "name": "flow_0e4b989a",
  "description": "",
  "metadata": {
    "__metadata_info__": {}
  },
  "inputs": [
    {
      "type": "string",
      "title": "user_question"
    }
  ],
  "outputs": [
    {
      "description": "the generated text",
      "type": "string",
      "title": "output"
    }
  ],
  "start_node": {
    "$component_ref": "d8870848-f3c1-4a88-a0f3-b6ca20c61bab"
  },
  "nodes": [
    {
      "$component_ref": "d8870848-f3c1-4a88-a0f3-b6ca20c61bab"
    },
    {
      "$component_ref": "25917cac-52d4-4816-8c62-c18d8b70ee33"
    },
    {
      "$component_ref": "158c838b-f8be-41ef-8b66-64348c8d379c"
    }
  ],
  "control_flow_connections": [
    {
      "component_type": "ControlFlowEdge",
      "id": "7b8c0c0b-fcd1-4bf3-96be-dcf726ab1526",
      "name": "start_step_to_PromptExecution_control_flow_edge",
      "description": null,
      "metadata": {
        "__metadata_info__": {}
      },
      "from_node": {
        "$component_ref": "d8870848-f3c1-4a88-a0f3-b6ca20c61bab"
      },
      "from_branch": null,
      "to_node": {
        "$component_ref": "25917cac-52d4-4816-8c62-c18d8b70ee33"
      }
    },
    {
      "component_type": "ControlFlowEdge",
      "id": "6b2c2840-126c-43fe-a8e1-f3cb08e8ae88",
      "name": "PromptExecution_to_None End node_control_flow_edge",
      "description": null,
      "metadata": {},
      "from_node": {
        "$component_ref": "25917cac-52d4-4816-8c62-c18d8b70ee33"
      },
      "from_branch": null,
      "to_node": {
        "$component_ref": "158c838b-f8be-41ef-8b66-64348c8d379c"
      }
    }
  ],
  "data_flow_connections": [
    {
      "component_type": "DataFlowEdge",
      "id": "86ba0435-be9b-46b0-97ae-64e145045e19",
      "name": "start_step_user_question_to_PromptExecution_user_question_data_flow_edge",
      "description": null,
      "metadata": {
        "__metadata_info__": {}
      },
      "source_node": {
        "$component_ref": "d8870848-f3c1-4a88-a0f3-b6ca20c61bab"
      },
      "source_output": "user_question",
      "destination_node": {
        "$component_ref": "25917cac-52d4-4816-8c62-c18d8b70ee33"
      },
      "destination_input": "user_question"
    },
    {
      "component_type": "DataFlowEdge",
      "id": "8638e234-9a23-45c5-89d4-296fc5a8c5ac",
      "name": "PromptExecution_output_to_None End node_output_data_flow_edge",
      "description": null,
      "metadata": {},
      "source_node": {
        "$component_ref": "25917cac-52d4-4816-8c62-c18d8b70ee33"
      },
      "source_output": "output",
      "destination_node": {
        "$component_ref": "158c838b-f8be-41ef-8b66-64348c8d379c"
      },
      "destination_input": "output"
    }
  ],
  "$referenced_components": {
    "25917cac-52d4-4816-8c62-c18d8b70ee33": {
      "component_type": "LlmNode",
      "id": "25917cac-52d4-4816-8c62-c18d8b70ee33",
      "name": "PromptExecution",
      "description": "",
      "metadata": {
        "__metadata_info__": {}
      },
      "inputs": [
        {
          "description": "\"user_question\" input variable for the template",
          "type": "string",
          "title": "user_question"
        }
      ],
      "outputs": [
        {
          "description": "the generated text",
          "type": "string",
          "title": "output"
        }
      ],
      "branches": [
        "next"
      ],
      "llm_config": {
        "component_type": "VllmConfig",
        "id": "93d098ef-9643-4d38-a012-8903bacbb784",
        "name": "LLAMA_MODEL_ID",
        "description": null,
        "metadata": {
          "__metadata_info__": {}
        },
        "default_generation_parameters": null,
        "url": "LLAMA_API_URL",
        "model_id": "LLAMA_MODEL_ID"
      },
      "prompt_template": "{{user_question}}"
    },
    "d8870848-f3c1-4a88-a0f3-b6ca20c61bab": {
      "component_type": "StartNode",
      "id": "d8870848-f3c1-4a88-a0f3-b6ca20c61bab",
      "name": "start_step",
      "description": "",
      "metadata": {
        "__metadata_info__": {}
      },
      "inputs": [
        {
          "type": "string",
          "title": "user_question"
        }
      ],
      "outputs": [
        {
          "type": "string",
          "title": "user_question"
        }
      ],
      "branches": [
        "next"
      ]
    },
    "158c838b-f8be-41ef-8b66-64348c8d379c": {
      "component_type": "EndNode",
      "id": "158c838b-f8be-41ef-8b66-64348c8d379c",
      "name": "None End node",
      "description": "End node representing all transitions to None in the WayFlow flow",
      "metadata": {},
      "inputs": [
        {
          "description": "the generated text",
          "type": "string",
          "title": "output"
        }
      ],
      "outputs": [
        {
          "description": "the generated text",
          "type": "string",
          "title": "output"
        }
      ],
      "branches": [],
      "branch_name": "next"
    }
  },
  "agentspec_version": "25.4.1"
}

You can then load the configuration back to an assistant using the AgentSpecLoader.

from wayflowcore.agentspec import AgentSpecLoader

assistant = AgentSpecLoader().load_yaml(serialized_assistant)

Next steps

Having learned how to specify the generation configuration, you may now proceed to:

• API Reference on LLM Models

• Build a Simple Conversational Assistant with Agents

• How to Create Conditional Transitions in Flows

• How to Build Assistants with Tools

Some additional resources we recommend:

• HuggingFace - Generation with LLMs

• HuggingFace - Text generation strategies

Full code

Click on the card at the top of this page to download the full code for this guide or copy the code below.

# Copyright © 2025 Oracle and/or its affiliates.
#
# This software is under the Universal Permissive License
# %%[markdown]
# Code Example - How to Specify the Generation Configuration when Using LLMs
# --------------------------------------------------------------------------

# How to use:
# Create a new Python virtual environment and install the latest WayFlow version.
# ```bash
# python -m venv venv-wayflowcore
# source venv-wayflowcore/bin/activate
# pip install --upgrade pip
# pip install "wayflowcore==26.1" 
# ```

# You can now run the script
# 1. As a Python file:
# ```bash
# python example_generationconfig.py
# ```
# 2. As a Notebook (in VSCode):
# When viewing the file,
#  - press the keys Ctrl + Enter to run the selected cell
#  - or Shift + Enter to run the selected cell and move to the cell below# (UPL) 1.0 (LICENSE-UPL or https://oss.oracle.com/licenses/upl) or Apache License
# 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0), at your option.




# %%[markdown]
## Imports

# %%
from wayflowcore.agent import Agent
from wayflowcore.models.llmgenerationconfig import LlmGenerationConfig


# %%[markdown]
## Define the llm generation configuration

# %%
generation_config = LlmGenerationConfig(
    max_tokens=512,
    temperature=1.0,
    top_p=1.0,
    stop=["exit", "end"],
    frequency_penalty=0,
    extra_args={"seed": 1},
)


# %%[markdown]
## Define the vLLM

# %%
from wayflowcore.models import VllmModel

llm = VllmModel(
    model_id="LLAMA_MODEL_ID",
    host_port="LLAMA_API_URL",
    generation_config=generation_config,
)
# NOTE: host_port should be a string with the IP address/domain name and the port. An example string: "192.168.1.1:8000"
# NOTE: model_id usually indicates the HuggingFace model id,
# e.g. meta-llama/Llama-3.1-8B-Instruct from https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

# %%[markdown]
## Build the agent and run it

# %%
agent = Agent(llm=llm)
conversation = agent.start_conversation()
conversation.append_user_message("What is the capital of Switzerland?")
conversation.execute()
print(conversation.get_last_message())


from wayflowcore.controlconnection import ControlFlowEdge
from wayflowcore.dataconnection import DataFlowEdge


# %%[markdown]
## Import what is needed to build a flow

# %%
from wayflowcore.flow import Flow
from wayflowcore.models.llmgenerationconfig import LlmGenerationConfig
from wayflowcore.property import StringProperty
from wayflowcore.steps import PromptExecutionStep, StartStep


# %%[markdown]
## Build the flow using custom generation parameters

# %%
start_step = StartStep(name="start_step", input_descriptors=[StringProperty("user_question")])
prompt_step = PromptExecutionStep(
    name="PromptExecution",
    prompt_template="{{user_question}}",
    llm=llm,
    generation_config=LlmGenerationConfig(temperature=0.8),
)
flow = Flow(
    begin_step=start_step,
    control_flow_edges=[
        ControlFlowEdge(source_step=start_step, destination_step=prompt_step),
        ControlFlowEdge(source_step=prompt_step, destination_step=None),
    ],
    data_flow_edges=[DataFlowEdge(start_step, "user_question", prompt_step, "user_question")],
)
conversation = flow.start_conversation(
    inputs={"user_question": "What is the capital of Switzerland?"}
)
conversation.execute()


# %%[markdown]
## Export config to Agent Spec

# %%
from wayflowcore.agentspec import AgentSpecExporter

serialized_assistant = AgentSpecExporter().to_yaml(flow)


# %%[markdown]
## Load Agent Spec config

# %%
from wayflowcore.agentspec import AgentSpecLoader

assistant = AgentSpecLoader().load_yaml(serialized_assistant)


# %%[markdown]
## Build the generation configuration from dictionary

# %%
config_dict = {
    "max_tokens": 512,
    "temperature": 0.9,
}

config = LlmGenerationConfig.from_dict(config_dict)


# %%[markdown]
## Export a generation configuration to dictionary

# %%

config = LlmGenerationConfig(max_tokens=1024, temperature=0.8, top_p=0.6)
config_dict = config.to_dict()