Over the last few days, I’ve had a few discussions where I had to explain what an AI agent is and how it works.

Each time, the same questions came up: what exactly are AI agents, how do they work, how do you build and tune one to do the work you want, the way you want it done?

So I thought I’ll talk a bit about it to make agents at-least conceptually understandable. This is not an advanced discussion on the topic.

AI Agent

An AI agent, in its simplest form is an actor that uses interactions with Large Language Models to attain one or a set of tasks as directed by the operator/user. At its heart, most agents run on very simple logic:

- Ask user for input

    - Send input to LLM and wait for response

    - Process any requests for additional information etc from LLM

    - Show output from the LLM to the user

- Ask user for next input

In the following sections we delve deeper into the different important aspects of an AI agent.

Structure of an Agent

The agent will have 3 major parts:

1. Ways to communicate with the user and the model
2. A system prompt defining its core function and behaviour
3. A set of functions/tools it can use to interact with other systems

We delve a little deeper into each section to get a better understanding of the system.

Communication

For an agent to work, it needs to be able to receive input from a user or operator. Note, I have mentioned an operator here, because it is very possible that the input comes from another system or another agent. Similarly once, the processing is done, or during processing, the agent might want to send out some information to the user/operator and publish the final result. The communication channel is very dependent on the environment in which the agent is running.

• If the agent is part of a CLI tool that runs in a terminal it might take input from the standard input and print its output on standard output or standard error channels as the case may be. Many agents would run the responses through prettifiers to render the markdown generated by the LLM in a nice format to the user. For example, a session with a simple agent could look like this:

  $ weatherbot
  > What's the weather in Mumbai today?
  
  Getting weather data for Mumbai...
  
  Weather for Mumbai, 2026-09-27:
  ─────────────────────────────
  Temperature : 31°C (feels like 34°C)
  Condition   : cloudy
  Humidity    : 74%
  Wind        : 12 km/h
  

• If the agent is part of a web server it might communicate using HTTP APIs for input and return the output via a variety of outputs, including, but not limited to:

  • A simple response to the API if the processing is fast enough
  • Server Side Events/Web-Sockets to stream the response from the server to the client
  • A mailbox type thing, where the agent leaves a response and the client picks it up In the simplest case, the client sends a request with the user’s question and the server replies with the answer:

  {
    "question": "What's the weather in Mumbai today?"
  }
  

  And the response could look something like this:

  {
    "answer": "It is 31°C and cloudy in Mumbai today."
  }
  

System Prompt

A system prompt is a set of instructions given to an agent to lay down a basic set of rules and purpose for itself. Here is a small example, the system prompt used by our WeatherBot agent:

You are WeatherBot, a helpful assistant that answers questions about the weather.

Rules:
1. Always use the get_weather tool to get current weather data.
2. If the user asks about something other than weather, politely say you can only help with weather questions.
3. Keep your answers short and friendly.

This is the core set of instructions that set the guidelines for user behaviour, outlining what to do, how to talk to the user and so on. This might contain some examples as well. Every chat turn (see later) will contain the system prompt once and must remain immutable to get any kind of decent performance from the model (explanation in the Important Notes section). It is better not to get carried away and write an essay here. Most new models can work with smaller system prompts. Try to avoid negative instructions here.

Tools

A tool is a function that an LLM can see and invoke to get something done through the agent. Some very simple examples of tools can be:

• The web for latest information.
• Information from a knowledge store that the agent has access to.
• Run a SQL query on a database that the agent has access to.
• Create/update/remove items from a TODO list somewhere.
• Make API calls to interact with another service.
• Run arbitrary bash commands on the system hosting the agent. Hopefully you get the drift now. Basically the LLM does not do any of these things. The LLM is aware of only the following information about a tool:
• The name of the tool (eg: web_search)
• A description for what the tool does (eg: Search the internet for information)
• Arguments that need to be passed to the tool along with the type information for the argument (eg: query: string, num_results: int)

Based on this information and the work at hand it will tell the agent to execute one or multiple tools by passing in the required arguments. For example, when you ask our WeatherBot “What’s the weather in Mumbai today?”, the model does not go out and check the weather itself. Instead, it replies to the agent with a tool call, like this:

{
  "id": "call_001",       //Tool call id
  "name": "get_weather",  //The name of the tool to be invoked
  "arguments": {          //Arguments for the tool
    "city": "Mumbai"
  }
}

This should trigger the following function in the Agent:

struct CityWeather {
    value: int,
    unit: TemperatureUnit { C|F }
    condition: Cloudy|Sunny
    humidity: float { 0.0-1.0 }
    // Other fields
}

func get_weather(city: String) -> CityWeather {
    // Call remote weather service API
    return CityWeather {
        ...
    }
}

The model already knows the type of the response to get (CityWeather in this case) and can use that information to generate a user-friendly textual response.

Interactions between agent and LLM

Most of the agent builder libraries will allow you to thread these things together easily and abstract out the complexities to certain extent. However, getting a basic understanding of the interaction will help you build agents effectively and tend to issues that crop up when something breaks. Let’s see the basic protocol and inner workings of how an agent interacts with an LLM.

On a side-note, there are 2-3 popular protocols/formats that can be used to talk to LLM providers (OpenAI chat completions, OpenAI Responses, Anthropic Messages etc), the following sections are vaguely modelled on the most popular one, the OpenAI Chat Completions format.

We treat the models as stateless for simplicity. There are some nuances we ignore here for simplicity.

Messages

The information exchange between the Agent and the LLM is encoded as messages. Typically messages are of the following types:

• Prompts

  • System Prompt - System behaviour guidance (Agent to model). Most models support only one system prompt.
  • User Prompt - Whatever input the user/operator that is sent to the model (Agent to model). Multiple of these can be sent. Can contain text, audio, video etc.
  • Developer Messages - Optional messages that can be added by the system developer to instruct nuances to the model (Agent to model).

• Tool calling

  • Tool call - The LLM sends this to the agent to request the agent to execute a particular function. This will contain a unique id, the name of the tool and the argument json string (Model to agent).
  • Tool call response - The Agent executes the function/tool and the response/error is encoded in a pre-defined format and sent back to the model in json encoded format. Also contains the tool call id to help the model identify which tool call this response was for (Agent to model).

• Response

  • This is the final response generated by the model and sent back to the user.

Chat Turn

A chat turn encapsulates whatever happens between a user asking a question to the Agent all the way to a final response being returned back to the user. Depending on the nature of the query from the user, there might be an arbitrary number of messages being passed between the agent and the model.

Request sent by Agent to Model

The general structure of a request from the Agent to the Model would consist of:

• A list of messages
• A list of available tools
• Settings to tune certain behaviour of the model
• Optionally, the shape of the output the model should generate

Putting it all together, here is what a request from our WeatherBot agent to the model might look like (this is not an actual request):

{
  "model": "model", //Model name
  "messages": [ // A list of all messages till now (see below)
    { // System prompt
      "role": "system", 
      "content": "You are WeatherBot, a helpful assistant that answers questions about the weather. Always use the get_weather tool to get current weather data. Never guess the weather. Keep your answers short and friendly."
    },
    { // User prompt
      "role": "user",
      "content": "What's the weather in Mumbai today?"
    },
    { // Response from the LLM. In this case we have received a tool call
      "role": "assistant",
      "content": null,
      "tool_calls": [
        {
          "id": "call_001",
          "name": "get_weather",
          "arguments": {
            "city": "Mumbai"
          }
        }
      ]
    },
    { // Response from agent with the results of all the tool calls
      // one message each
      "role": "tool",
      "tool_call_id": "call_001",
      "content": {
        "value": 31,
        "unit": "C",
        "condition": "Cloudy",
        "humidity": 0.74,
        // Other fields
      }
    }
  ], // This is all the messages we are sending for now
  "tools": [ // List of all tools we want the model to see for now
    {
      "type": "function",
      "function": {
        "name": "get_weather", // Name of the tool
        "description": "Get the current weather for a city", // Description for the tool
        "parameters": { // List of parameters it expects. This is a JSON Schema.
          "type": "object",
          "properties": {
            "city": {
              "type": "string",
              "description": "The name of the city, e.g. Mumbai"
            }
          },
          "required": ["city"]
        }
      }
    }
  ],
  "temperature": 0.2, // Settings for the model
  "tool_choice": "auto"
}

Note: The messages section will contain all messages till now from all parties.

Response from Model to Agent

This will typically consist of:

• A set of new messages from the model
  • A list of tool calls the model wants the agent to execute
  • A response if it is done
• A marker indicating the content of the messages it is sending:
  • tool_calls indicating that the messages are actually model asking the agent to run a few tools
  • stop indicating it is done and the message is actually the final response
  • Error codes indicating a reason why the call has actually failed
• Usage statistics
• System metadata

{
  "id": "chatcmpl_001", // Unique id for this response
  "model": "my-model", // The model that generated this response
  "messages": [ // Messages from model
    {
      "role": "assistant",
      "content": "It is 31°C and cloudy in Mumbai today, with a humidity of 74%." // The final answer, null if tool_calls
    }
  ],
  "finish_reason" : "stop", // Reason for responding (processing completed in this case)
  "usage": { // Token usage statistics
    "prompt_tokens": 510,
    "completion_tokens": 28,
    "total_tokens": 538
  }
}

Sequence of calls

Now that we understand the basic anatomy of requests and responses, here is a general heuristic of how a chat turn happens:

Messages[] chat_turn(
    tool_metadata: List{ tool_name, description, arguments schema }, // Metadata for all tools we want model to see
    user_input: List{ String | Image | Audio ... }, // Input(s) from the user
    old_messages: List{ Messages } // List of messages from previous turns (optional)
) {
    messages = [] // Create array of messages
                + system_prompt // Add the system prompt
                + old_messages // Add existing messages we've received
                + user_input // Add input from the user

    // Setup loop
    stop_received = false

    // Run in a loop till we get final response back from the model
    while !stop_received {
        //Send messages and tool metadata to model
        response = send_message(messages, tool_metadata)

        // Save the message in messages list
        messages = messages + response.messages

        // Check why we got a response
        if response.finish_reason == "stop" {

            // Processing complete
            print_response(response.messages)

            // We are done
            stop_received = true
        }
        else if response.finish_reason == "tool_calls" {
            // Model is asking agent to run a tool

            // Get the required information we need to run tools and respond
            tool_calls{tool_call_id, tool_name, tool_argument} = response.messages

            // Actually execute the tools and get responses or errors
            tool_call_results = execute(tool_calls)

            // Add responses to list of messages.
            messages = messages + tool_call_results
        }
        else {
            // We must have received an error
            // Show error to the user
            print_error(response.finish_reason, response.messages)
            
            // We are done
            stop_received = true
        }
    }

    messages = messages - system_prompt // Remove system prompt anyways as it wouldn't change

    return messages // For use in the next turn
}

Once the above runs through, our chat turn is done.

Note - If we are designing a multi-turn system where the user can ask follow-up questions we need to save the list of messages at the end of a turn and feed it back to the model in the next turn.

We can visually represent this in the following Sequence diagram for the WeatherBot agent:

That’s how we send a request and get a response from the Agent.

Conversations and Sessions

Now that we have the basic building block in place, we need to make this a continuous conversation like we usually do. This is where a Session comes in. A session is a collection of chat turns and generally has a unique ID called the Session ID. This id helps in various ways like for example:

• Saving and retrieving conversation history from an in-memory or more permanent local or remote storage
• Some (most?) providers accept this ID to optimize the processing on their end to provide you with a faster experience (A bit more on this later).

As seen from the above example, to get conversational experience from your agent, you need to keep persisting the list of messages after each chat turn and send it back in along with the new input for the next turn. Taking this into account, the life-cycle of a simple agent would look something like this:

Messages[] converse_with_agent(session_id: String) {
    // Discover all the tools defined in the system
    tool_metadata = discover_tools()

    // Intialize session messages
    messages = load_messages(session_id) || []

    while !stop_received() {
        // Get input from user
        input = read_user_input()
        
        // Process
        messages = chat_turn(tool_metadata, input, messages)
    }

    //Save messages for later
    save_messages(session_id, messages)

    return messages
}

Now our WeatherBot can help us answer more detailed questions based on previous answers. The call sequence will look something like this:

That about does explain what agents are for a majority of use case. But before we wrap up, a few more things to understand in case you are planning to get into building your own agents.

Important Notes

Here are some of the nuances we need to keep in mind when building agents.

Context Window

This is the space the LLM considers when processing the information.

All the messages we are sending to the model end up in the context window. This is precious real-estate, and depending on size of the information being processed and that the model can handle, it might be a significant amount of iterative improvement work to ensure we cram in as much useful information into the window without filling it up.

The time it takes for models is generally proportional to the size of the context it is working with. Which brings us to the next points.

Compaction

It is a process of extracting relevant set of information from the existing messages list in such a way as to capture the essence of the work being done in this session and what needs to be done going forward.

Once compaction is being done, the older messages are not sent to the model and the compaction message becomes a user input to usher in the start of the next set of messages.

This is a bit of an over simplification and is handled differently by different agents. Some agents send in some of the older messages back in along with the compacted message, some keep trimming the old message list to a fixed size in real-time to ensure only some messages are sent back, thereby trying to defer the context getting filled up as much as possible.

Cache

As mentioned before, models are state-less, i.e. they do not save any information. This means that during a chat-turn every time we send a messages list, the model needs to process the whole thing.

As you can probably guess, this is really expensive and will significantly increase the time between when we send the first request and when we get back the final response. To reduce this time, most engines will store the information already processed in a cache. This cache is based on the match with the message list.

Therefore it is critical to keep the messages being sent back to the model stable/unchanged. So keep the messages same and any transient information should only come at the end of the messages list. Good agents can go above 95-96% consistently. Great ones even higher. This, however, is also somewhat dependent on the solution used by the inferencing provider.

Sub-Agents, Hand-off, Orchestration

Sometimes agents need to do some work which can be handled out of context with some information being sent in and the agent completing the work.

In such cases the main agent might spawn out other agents to get the work done independently and send the final output back to the parent agent through some pre-agreed-upon mechanism.

If the agent(s) being spun off don’t need to come back, the work submission to other agents is called as a hand-off.

If they do however come back to the parent that will then consume the responses they produce, and the number of such spun up sub-agents is high, an orchestration mechanism might be needed to manage the life-cycle of such agents.

If the life-cycle of the work done is fixed, people tend to implement fixed workflows. In other cases workflows might depend on context or be more complicated enough to be determined and constructed at the run-time by the parent agent itself.

Evals/Evaluation

A set of tests that ensure agents are generating the intended outputs for a given input. Such verification of the output can be done using rules or by passing the outputs through another agent/LLM call to ensure adherence. It is critical to have a good set of evals for agents you are planning to deploy for critical tasks and production use.

Conclusion

There are newer discoveries and mechanisms people keep coming up with everyday as this in itself is a nascent field as of the time of writing this. However, hopefully, this article will help some people understand agents better and try creating some of their own to alleviate and automate some boring parts of their lives.

A few examples of agents you can try to build:

• You can have tools like bash, read, write_file and create a coding agent.
• You can have tools that can run SQL queries to have an agent that helps you make sense of some data
• You can have tools like spin_up_subagent that starts another sub-agent to do some stuff

The possibilities are endless.