Cover Page

Backend Page

TypeScript with Express

We assume that your chatterd code base has accumulated code from at least the llmPrompt, chatter backend tutorials. If you’ve, in addition, also accumulated code from latter tutorials, that’s fine.

toolbox

Let us start by creating a toolbox to hold our tools. Change to your chatterd folder and create a new TypeScript file, name it toolbox.ts:

server$ cd ~/reactive/chatterd
server$ vi toolbox.ts

Put the following import at the top of the file:

import HttpStatus from "http-status-codes";

The contents of this file can be categorized into three purposes: tool/function definition, the toolbox itself, and tool use (or function calling).

Tool/function definition

Ollama tool schema: at the top of Ollama’s JSON tool definition is a JSON Object respresenting a tool schema. The tool schema is defined using nested JSON Objects and JSON Arrays. Add the full nested definitions of Ollama’s tool schema to your file:

export type OllamaToolSchema = {
    type: string
    function: OllamaToolFunction
}

type OllamaToolFunction = {
    name:        string
    description: string
    parameters?: OllamaFunctionParams
}

type OllamaFunctionParams = {
    type:        string
    properties?: Record<string, OllamaParamProp>
    required?:   string[]
}

type OllamaParamProp = {
    type:        string
    description: string
    enum?:       string[]
}

Weather tool schema: in this tutorial, we have only one tool resident in the backend. Add the following tool definition to your file:

const WEATHER_TOOL: OllamaToolSchema = {
    type: "function",
    function: {
        name:        "get_weather",
        description: "Get current temperature",
        parameters: {
            type: "object",
            properties: {
                "latitude": {
                    type:        "string",
                    description: "latitude of location of interest",
                },
                "longitude": {
                    type:        "string",
                    description: "longitude of location of interest",
                },
            },
            required: ["latitude", "longitude"],
        },
    },
}

Weather tool function: we implement the get_weather tool as a getWeather() function that makes an API call to the free Open Meteo weather service. Add the following nested struct definition to hold Open Meteo’s return result. For this tutorial, we’re only interested in the latitude, longitude, and temperature returned by Open Meteo:

type Current = {
    temperature_2m: number
}

type OMeteoResponse = {
    latitude: number
    longitude: number
    current: Current
}

Here’s the definition of the getWeather() function:

export async function getWeather(argv: string[]): Promise<[string?, string?]> {
    // Open-Meteo API doc: https://open-meteo.com/en/docs#api_documentation
    let response: Response
    try {
        response = await fetch(`https://api.open-meteo.com/v1/forecast?latitude=${argv[0]}&longitude=${argv[1]}&current=temperature_2m&temperature_unit=fahrenheit`, {
            method: "GET",
        })
        if (response.status != HttpStatus.OK) {
            return [null, `Open-meteo: ${response.status}: ${response.statusText}`]
        }
    } catch (error) { 
        return [null, "Cannot connect to Open Meteo"]
    }
    const ometeoResponse: OMeteoResponse  = await response.json()
    return [`Weather at latitude ${ometeoResponse.latitude} and longitude ${ometeoResponse.longitude} is ${ometeoResponse.current.temperature_2m}ºF`, null]
}

The toolbox

Even though we have only one resident tool in this tutorial, we want a generalized architecture that can hold multiple tools and invoke the right tool dynamically. To that end, we’ve chosen to use a switch table (or jump table or, more fancily, service locator registry) as the data structure for our tool box. We implement the switch table as a dictionary. The “keys” in the dictionary are the names of the tools/functions. Each “value” is a record containing the tool’s definition/schema and a pointer to the function implementing the tool. To send a tool as part of a request to Ollama, we look up its schema in the switch table and copy it to the request. To invoke a tool called by Ollama in its response, we look up the tool’s function in the switch table and invoke the function.

Add the following type for an async tool function and the record type containing a tool definition and the async tool function:

type ToolFunction = (args: string[]) => Promise<[string?, string?]>

type Tool = {
    schema:   OllamaToolSchema
    function: ToolFunction
}

Now create a switch-table toolbox and put the WEATHER_TOOL in it:

export const TOOLBOX: Record<string, Tool> = {
    "get_weather": { schema: WEATHER_TOOL, function: getWeather},
}

Tool use or function calling

Ollama tool call: Ollama’s JSON tool call comprises a JSON Object containing a nested JSON Object carrying the name of the function and the arguments to pass to it. Add these nested struct definitions representing Ollama’s tool call JSON to your file:

export type OllamaToolCall = {
    function: OllamaFunctionCall
}

type OllamaFunctionCall = {
    name:      string
    arguments: Map<string, string>
}

Tool invocation: finally, here’s the tool invocation function. We call this function to execute any tool call we receive from Ollama response. It looks up the toolbox for the tool name. If the tool is resident, it runs it and returns the result, otherwise it returns a null.

export async function toolInvoke(func: OllamaFunctionCall): Promise<[string?, string?]> {
    if (func.name in TOOLBOX) {
        return TOOLBOX[func.name]?.function(Object.values(func.arguments))
    }
    return [null, null]
}

That concludes our toolbox definition. Save and exit the file.

handlers

Edit handlers.ts:

server$ vi handlers.ts

imports

Under import { chatterDB } from './main.js'} add:

import type { OllamaToolCall, OllamaToolSchema } from "./toolbox.js"
import { getWeather, TOOLBOX, toolInvoke } from "./toolbox.js"

interfaces

Next add or update the following interfaces:

• if update, add a tool-calls field to your OllamaMessage interface:

  interface OllamaMessage  {
    role: string;
    content: string;
    tool_calls?: OllamaToolCall[]
  }
  

• if update, add a tools field to your OllamaRequest: ```ts interface OllamaRequest { appID: string; model: string; messages: OllamaMessage[]; stream: boolean; tools?: OllamaToolSchema[] }

The OllamaResponse interface (unchanged if exists):

interface OllamaResponse {
    model: string;
    created_at: string;
    message: OllamaMessage;
}

For the /weather testing API, add also the following struct:

type Location = {
    lat: string
    lon: string
}

weather

Let’s implement the handler for the /weather API that we can use to test our getWeather() function later:

export async function weather(req: Request, res: Response) {
    let loc: Location
    try {
        loc = req.body
    } catch (error) {
        logClientErr(res, HttpStatus.UNPROCESSABLE_ENTITY, error.toString())
        return
    }

    const [temp, error] = await getWeather([loc.lat, loc.lon])
    if (error) {
        logServerErr(res, error)
    }
    res.json(temp)
}

llmtools

The underlying request/response handling of llmtools() is basically that of llmchat(), however with all the mods needed to support tool calling, it’s simpler to just start the llmtools() handler from scratch. We will name variables according to this scheme:

• camelCase for language-level data objects,
• snake_case for string version of data objects to be used with PostgreSQL or JSON, and, earlier,
• ALL_CAPS for immutable global toolbox and tool definitions.

To store the client’s conversation context/history with Ollama in the PostgreSQL database, llmtools() first confirms that the client has sent an appID that can be used to tag its entries in the database. Here’s the signature of llmtools(). The Json deserialization will check for the existence of appID and return an HTTP error if it is absent:

export async function llmtools(req: Request, res: Response) {
    let ollamaRequest: OllamaRequest;

    try {
        ollamaRequest = req.body
    } catch (err) {
        logClientErr(res, HttpStatus.UNPROCESSABLE_ENTITY, err.toString())
        return
    }

    if (ollamaRequest.appID.length == 0) {
        logClientErr(res, HttpStatus.UNPROCESSABLE_ENTITY, "Invalid appID: ${ollama_request.appID}")
        return
    }

    // retrieve client's tool(s)

}

Our goal here is to prepend all prior conversations between the client and Ollama as context to the current prompt. The client’s appID allows us to identify its conversation with Ollama stored in the PostgreSQL database—similar to how MCP tags JSON-RPC 2.0 messages with a session ID. Once we confirm that the client has an appID, we retrieve any tool definitions attached to the ollamaRequest carrying the prompt. We will assemble these tools along with any tools the client may have previously sent to Ollama, attached to an earlier prompt, and any tools resident on chatterd and attach them all to the contextualized prompt request we will POST to Ollama. Replace // retrieve client's tool(s) with:

    // convert tools from client as JSON string (client_tools) and save to db;
    // prepare ollama_request for re-use to be sent to Ollama:
    // clear tools in request, to be populated later
    let client_tools: string | null = null
    try {
        if (ollamaRequest.tools != null) {
            client_tools = JSON.stringify(ollamaRequest.tools)
            ollamaRequest.tools = null
        }

        // insert into DB

    } catch (err) {
        logServerErr(res, err.toString())
        return
    }

    // assemble resident tools

Then we insert the current prompt into the database, adding to the client’s conversation history with Ollama. As shown in the example in Tool definition JSON section, the client’s current prompt could comprise of multiple elements in the messages array of the ollamaRequest, but the tools will reside in a single tools array next to the messages array. When there are multiple elements in an ollamaRequest, we want to insert the tools only once. Below we have chosen to insert the tools only with the first element of the messages array. Replace the comment // insert into DB with the following code:

        // insert each message into the database
        // insert client_tools only with the first message:
        // reset it to empty after first message.
        for (const msg of ollamaRequest.messages) {
            try {
                await chatterDB`INSERT INTO chatts (username, message, id, appid, toolschemas) VALUES (${msg.role}, ${msg.content.replace('\n', ' ').replaceAll("  ", " ").trim()}, gen_random_uuid(), ${ollamaRequest.appID}, ${client_tools})`;
            } catch (err) {
                logServerErr(res, `${err as PostgresError}`)
                return
            }
            
            // store device's tools only once
            client_tools = null
        }

To prepare the full assemblage of tools to send to Ollama, we first attach all the tools resident on chatterd. Replace // assemble resident tools with:

    // append all of chatterd's resident tools to ollamaRequest
    ollamaRequest.tools = []
    for (const tool of Object.values(TOOLBOX)) {
        ollamaRequest.tools.push(tool.schema)
    }

    // reconstruct ollamaRequest

Then we retrieve the client’s conversation history, including the recently inserted, current prompt, as the last entry, and put each as a separate element in the ollamaRequest.messages array, taking care to accumulate any tool(s) present into ollamaRequest.tools array instead. Replace // reconstruct ollamaRequest with:

    // reconstruct ollamaRequest to be sent to Ollama:
    // - add context: retrieve all past messages by appID,
    //   incl. the one just received, and attach them to
    //   ollamaRequest
    // - convert each back to OllamaMessage and
    // - insert it into ollamaRequest
    // - add each message's clientTools to chatterd's resident tools
    //   already copied to ollamaRequest.tools.
    try {
        ollamaRequest.messages = (await chatterDB`SELECT username, message, toolcalls, toolschemas FROM chatts WHERE appid = ${ollamaRequest.appID} ORDER BY time ASC`)
            .map(row => {
                if (row.toolschemas != null) {
                    ollamaRequest.tools.push(...JSON.parse(row.toolschemas))
                }
                return {
                    role: row.username,
                    content: row.message,
                    tool_calls: JSON.parse(row.toolcalls),
                }
            })
    } catch (err) {
        logServerErr(res, `${err as PostgresError}`)
        return
    }

    // NDJSON to SSE stream transformation

NDJSON to SSE stream transformation

As we know, Ollama response is in the form of an NDJSON stream, which we transform into a stream of SSE events to be sent to the client. We first prepare the response header with which to send the SSE stream. Then we declare an accumulator variable, full_response, to assemble the reply tokens Ollama streams to us. To accommodate resident-tool call, we use a flag, sendNewPrompt, to indicate to our stream generator whether:

• to start a resident-tool call connection to Ollama and continue yielding results to the client or
• to conclude streaming to the connection. While sendNewPrompt is true—it is initialized to true, we open a new POST connection to Ollama and send it the ollamaRequest message. Place the following below the // NDJSON to SSE stream transformation comment:

    res.writeHead(HttpStatus.OK, {
        'Content-Type': 'text/event-stream',
        'Cache-Control': 'no-cache',
    }).flushHeaders()
  
    let sendNewPrompt = true
    while (sendNewPrompt) {
        sendNewPrompt = false  // assume no resident-tool call
        let response
        let full_response = '' // if set to null, causes "null" to be preprended!
  
        try {
            response = await fetch(OLLAMA_BASE_URL + "/chat", {
                method: req.method,
                body: JSON.stringify(ollamaRequest), // convert the request to JSON
            })                                       // send request to Ollama
        } catch (err) {
            res.write(`event: error\ndata: { "error": ${err} }\n\n`)
            res.end()
            return
        }
  
        // handle Ollama response
  
    }
    res.end()   
  

We convert each NDJSON line to a language-level type, OllamaResponse in this case, with semantically meaningful structure and fields that we can more easily manipulate than a linear byte stream or string. If the conversion is unsuccessful and the model property of the type is empty, we return an SSE error event and move on to the next NDJSON line. Otherwise, we append the content of this OllamaResponse.message to the full_response accumulator. Replace // handle Ollama response with:

        let tool_calls = null // doesn't cause "null" to be entered in Postgres DB when null
        let tool_result = ''  // if set to null, causes "null" to be preprended!

        try {
            for await (const chunk of response.body) {
                const line = Buffer.from(chunk).toString().replace(/[\n]/, '')

                try {
                    // deserialize each line into OllamaResponse
                    const ollamaResponse: OllamaResponse = JSON.parse(line);
                    if (ollamaResponse.model == "") {
                        // didn't receive an ollamaresponse, report to client as error
                        res.write(`event: error\ndata: { "error": ${line.replaceAll("\\\"", "'")} }\n\n`)
                        continue
                    }

                    // append response token to full assistant message
                    full_response += ollamaResponse.message.content


                    // check for tool call

                } catch (err) {
                    res.write(`event: error\ndata: { "error": ${err} }\n\n`)
                }
            }
        } catch (err) { // loop through response
            res.write(`event: error\ndata: { "error": ${err} }\n\n`)
            res.end()
            return
        }    
                    
        // insert full response into db

The tool call field in OllamaResponse is an array, even though it looks like Qwen3 on Ollama is presently limited to making only one tool call per HTTP round. We loop through the array and for each tool call, we try to call its function by calling toolInvoke() from our toolbox. If there is no tool call, we simply encode the full NDJSON line into an SSE Message event and yield it as an element of the SSE stream and move on to the next NDJSON line, as we do in llmchat. Replace // check for tool call with:

                    // is there a tool call?
                    if (ollamaResponse.message.tool_calls != null) {
                        // convert toolCalls to JSON string (tool_calls) and save to db
                        tool_calls = JSON.stringify(ollamaResponse.message.tool_calls)

                        for (const toolCall of ollamaResponse.message.tool_calls) {
                            if (toolCall.function.name == "") {
                                continue // LLM miscalled
                            }
                            const [toolResult, err] = await toolInvoke(toolCall.function)

                            // handle tool result

                        } // for toolCall
                    } else {
                        // no tool call, send NDJSON line as SSE data line
                        res.write(`data: ${line}\n\n`); // SSE
                    }

If the tool is resident, toolInvoke() returns the result of the tool call. There are three possible outcomes from the call to toolInvoke():

1. the tool is resident but the call was unsuccesfull and returns an error,
2. the tool is resident and the call was successful, or
3. the tool is non-resident.

If the result indicates that an error has occured, we are dealing with the first outcoe above. We simply report the error to the client and move on to the next NDJSON line. If there’s no error but toolInvoke() returns null result, this indicates that the tool is non resident. We forward the tool call to the client as a tool_calls SSE event. Otherwise, we prepare the result to be saved to PostgreSQL and return the result to Ollama. Replace // handle tool result with:

                            if (toolResult) {
                                // outcome 2: tool call is resident

                                // convert toolResult to JSON string (tool_result)
                                // to be saved to db
                                tool_result += toolResult

                                // create new OllamaMessage with tool result
                                // to be sent back to Ollama
                                const toolresultMsg: OllamaMessage = {
                                    role: 'tool',
                                    content: toolResult,
                                }
                                ollamaRequest.messages.push(toolresultMsg)

                                // send result back to Ollama
                                sendNewPrompt = true
                            } else if (err) {
                                // outcome 1: tool resident but had error
                                res.write(`event: error\ndata: { "error": ${err} }\n\n`)
                            } else {
                                // outcome 3: tool non resident, forward 
                                // to device as 'tool_calls' SSE event
                                res.write(`event: tool_calls\ndata: ${line}\n\n`)
                            }

When we reach the end of the NDJSON stream, we insert the full Ollama response and any resident tool calls and their results into PostgreSQL database as the assistant’s reply. Any error in the insertion yields an SSE error event sent to the client. Replace // insert full response into db with:

        try {
            await chatterDB`INSERT INTO chatts (username, message, id, appid, toolcalls) \
            VALUES ('assistant', ${full_response.replace(/\s+/g, ' ')}, gen_random_uuid(), ${ollamaRequest.appID}, ${tool_calls})`
            if (sendNewPrompt) {
                await chatterDB`INSERT INTO chatts (username, message, id, appid) \
                VALUES ('tool', ${tool_result}, gen_random_uuid(), ${ollamaRequest.appID})`;
            }
        } catch (err) {
            res.write(`event: error\ndata: { "error": ${JSON.stringify((err as PostgresError).toString())} }\n\n`)
        }

We’re done with handlers.ts! Save and exit the file.

main.ts package

Edit main.ts:

server$ vi main.ts

Find the initialization of app and add these routes right after the route for /llmprompt:

      .post('/llmtools/', handlers.llmtools)
      .get('/weather/', handlers.weather)

We’re done with main.ts. Save and exit the file.

Build and test run

TypeScript is a compiled language, like C/C++ and unlike JavaScript and Python, which are an interpreted languages. This means you must run npx tsc each and every time you made changes to your code, for the changes to show up when you run node.

To build your server, transpile TypeScript into JavaScript:

server$ npx tsc

To run your server:

server$ sudo node main.js
# Hit ^C to end the test

Return to the Testing your /llmtools API section.