TypeScript with Fastify

Cover Page

Back-end Page

Install Node.js

We will first use the apt package manager to install Node.js version 18.19.1 or later (up to version 26.3.1 tested), the pnpm package manager, and the tsgo TypeScript transpiler:

server$ sudo apt update
server$ sudo apt install libmimalloc2.0 undici npm # will automatically install node.js also
server$ sudo npm install -g @typescript/native-preview
# install pnpm
server$ curl -fsSL https://pnpm.io | sh -

Confirm that you’ve installed Node.js version 18.19.1 or later:

server$ node --version
# output:
v18.19.1
# or later
Installing other versions of Node.js

To install Node.js v26:

server$ curl -sL https://deb.nodesource.com/setup_26.x | sudo bash -
server$ sudo apt update
server$ sudo apt install nodejs # automatically install npm also
# or sudo apt upgrade

For other versions of Node.js, replace 26 with the version number you want.

To remove Node.js installed from nodesource:

server$ sudo apt purge nodejs
server$ sudo rm /etc/apt/sources.list.d/nodesource.list

See documentations on How to Install Node.js on Ubuntu and How to remove nodejs from nodesource.com


package.json and tsconfig.json

First create and change into a directory where you want to keep your harnessd package:

server$ mkdir ~/agentic/harnessd
server$ cd ~/agentic/harnessd

Create the file package.json with the following content:

{
  "type": "module",
  "scripts": {
    "build": "pnpm exec tsgo && esbuild main.ts --bundle --platform=node --format=esm --external:readline/promises --external:better-sqlite3 --banner:js=\"import { createRequire } from 'module'; import { fileURLToPath } from 'url'; import { dirname } from 'path'; const require = createRequire(import.meta.url); const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename);\" --outfile=harnessd.js"
  },
  "devDependencies": {
    "@typescript/native-preview": "7.0.0-dev.20260622.1"
  }  
}

This allows use of import statements instead of require() of CommonJS and install tsgo.

Install the following packages:

server$ pnpm install fastify http-status-codes
# later, for images: @fastify/static @fastify/multipart
server$ pnpm install -D esbuild @types/node
server$ pnpm approve-builds # hit space, then enter 'y' to approve building esbuild

Your ~/agentic/harnessd directory should now contain the following:

node_modules/  package.json  pnpm-lock.yaml  pnpm-workspace.yaml

The package.json file should now contain:

{
  "type": "module",
  "scripts": {
    "build": "pnpm exec tsgo && esbuild main.ts --bundle --platform=node --format=esm --external:readline/promises --external:better-sqlite3 --banner:js=\"import { createRequire } from 'module'; import { fileURLToPath } from 'url'; import { dirname } from 'path'; const require = createRequire(import.meta.url); const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename);\" --outfile=harnessd.js"
  },
  "devDependencies": {
    "@types/node": "^26.0.0",
    "@typescript/native-preview": "7.0.0-dev.20260622.1",
    "esbuild": "^0.28.1"
  },
  "dependencies": {
    "fastify": "^5.8.5",
    "http-status-codes": "^2.3.0",
    "undici": "^8.5.0"
  }
}

The devDependencies are used in development, not deployed in production. In our case, all the devDependencies are for static type checking. The dependencies with @types prefix define the types used in their corresponding native JavaScript modules that have no static typing information. The file package-lock.json locks the dependencies to specific versions, to prevent automatically incorporating later, breaking versions. The node_modules directory contains the module files of both direct dependencies in package.json and indirect dependencies used by the direct dependencies.

updates

Updating packages downloaded from the registry:

server$ pnpm self-update
server$ pnpm outdated
server$ pnpm update # -g # to update globally


Create a tsconfig.json configuration file for use by TypeScript:

server$ vi tsconfig.json

and put the following lines in it:

{
  "compilerOptions": {
    // https://aka.ms/tsconfig for definitions

    // to use esbundler
    "noEmit": true,
    // save project graph for incremental build in tsconfig.tsbuildinfo
    "incremental": true,
    
    /* Language and Environment */
    "target": "esnext",                   // version of emitted JavaScript and compatible library
    "module": "esnext",                   // version of generated module code
    "moduleResolution": "bundler",        // how TypeScript looks up a file from module

    /* Interop Constraints */
    "verbatimModuleSyntax": true,         // transform and elide only imports/exports marked
                                          // "type"; the rest are emitted per the "module" setting
    "esModuleInterop": true,              // can import CommonJS modules; enables 'allowSyntheticDefaultImports'

    /* Lazy Iterator */
    "lib": ["ESNext"],
    
    /* Type Checking */
    "strict": true,                       // enable strict type-checking options    
    "alwaysStrict": true,                 // always enable all strict type-checking options

    /* Null Check */
    "strictNullChecks": true,
    
    /* Type safe array access */
    "noUncheckedIndexedAccess": true,
    
    /* Completeness */
    "skipLibCheck": true,                  // skip type checking all .d.ts files
    
    "noImplicitAny": true,
    "strictFunctionTypes": true,
    "strictBindCallApply": true,
    "noImplicitThis": true,
    "useUnknownInCatchVariables": true,
    "noImplicitOverride": true
  }
}

Save and exit tsconfig.json.

You have installed all packages and set up all configurations needed by the harnessd back end.

main.ts

Create a file called main.ts:

server$ vi main.ts

Edit the file to add the following import lines:

import Fastify from "fastify"
import type { FastifyRequest, FastifyReply, RouteGenericInterface } from "fastify"
import type { Http2SecureServer, Http2ServerRequest, Http2ServerResponse } from "node:http2"
import { readFileSync } from 'node:fs'       // import function
import type { AddressInfo } from 'node:net'  // import type

import * as handlers from './handlers.js'    // import namespace

We import our module handlers, which we will define later.

For logging purposes, we save the client’s IP as an extended property of the Fastify type definition. We also specialize the FastifyRequest and FasitfyReply generic types to HTTP/2 traffic. Add the following lines:

declare module "fastify" {
    interface FastifyRequest {
        clientIp: string
    }
}
export type FastifyHttp2Request = FastifyRequest<
    RouteGenericInterface,
    Http2SecureServer,
    Http2ServerRequest
>
export type FastifyHttp2Reply = FastifyReply<
    RouteGenericInterface,
    Http2SecureServer,
    Http2ServerRequest,
    Http2ServerResponse
>

Next we instantiate a Fastify instance and configure it to:

} catch (error) { console.error(error) process.exit(1) }

The line `.post("/llmprompt", handlers.llmprompt)` populates `Fastify`'s *routing table* to serve
the HTTP POST request with URL endpoint `/llmprompt`. We route the endpoint to the `llmprompt()`
function. Since we only allow HTTP POST request for the `/llmprompt` endpoint, we pass the
`llmprompt()` handler to `post()` 

The function `llmprompt()` will be implemented in `handlers.ts` later.

We now register a callback function with the process: (1) to shut the process down gracefully when
terminated or interrupted by user, and (2) to print out a stack trace and exit with an error code in
the case of any uncaught exception. Replace the comment `// clean up on exit` with:
```ts
    const cleanUp = (signal: string | Error, exitCode: number = 0) => {
        app.close(async () => {
            if (signal instanceof Error) {
                console.error(`Uncaught exception: ${signal}\n` + `Stack trace: ${signal.stack}`)
            }
            process.exit(exitCode)
        })
    }
    process.on('SIGINT', cleanUp)
    process.on('SIGTERM', cleanUp)
    process.on('uncaughtException', (err) => cleanUp(err, 2))

With process termination events taken care of, staying inside the try block, after setting up the Fastify app, we launch up the node.js HTTP/2 server, binding it to the wildcard IP address (0.0.0.0, equivalent to any) and the default HTTPS port (443). Replace // launch the server with:

    app.listen({ host: "0.0.0.0", port: 443 }, (err, addr) => {
        if (err) {
            console.error(err) 
            process.exit(1) 
        }
        const address = app.server.address() as AddressInfo 
        console.log(`harnessd on https://${address.address}:${address.port}`) 
    })

We’re done with main.ts. Save and exit the file.

handlers.ts

We implement URL path API handlers in handlers.ts:

server$ vi handlers.ts

Start the file with the following imports:

import { type FastifyHttp2Request, type FastifyHttp2Reply } from './main.js'
import HttpStatus from 'http-status-codes'
import { Agent } from 'undici'
import { LlmRunners } from './runners.js'

We add a couple of logging functions that, in case of error, prepare a JSON response to be returned to the client:

const logOk = (request: FastifyHttp2Request, runner: string, model: string) => {
    console.info(`runner: ${runner}:${model}`)
}

const logInfo = (request: FastifyHttp2Request, runner: string, model: string, errcode: number, msg: string) => {
    console.info(`${errcode} |${request.ip} | ${runner}:${model} | ${msg}`)
}

const logClientErr = (request: FastifyHttp2Request, reply: FastifyHttp2Reply, runner: string, errcode: number, errmsg: string): FastifyHttp2Reply => {
    console.warn(`${errcode} | ${request.ip} | ${runner} |`)
    return reply.status(errcode).send(errmsg)
}

We instantiate a single undici.Agent dispatcher for use by fetch to connect to LLM runners. We set the dispatcher’stimeout to 5 secs so that a non-responding runner will not stall our LLM runner selector. We reuse keep a dispatcher alive after a connection is terminated so that we can reuse it for the subsequent connection to the same destination, to skip connection setup time. We also disable per-byte read deadline so that idle but incomplete SSE stream is not automatically ended.

// share a fetch dispatcher to the same destination to avoid
// subsequent connection establishment handshake
const dispatcher = new Agent({
    connect: { timeout: 5_000 }, // dead destination
    headersTimeout: 10_000,      // stalled upstream
    bodyTimeout: 0,              // keep SSE streams open indefinitely
    keepAliveTimeout: 60_000,    // reuse socket for the same destination
})

We next define the handler llmprompt(), which first pick an LLM runner from a ranked list of available runners. Since different runner provides different models, we patch the "model" field of the incoming prompt to the model available at the runner. We use Node’s fetch() function to connect to the LLM runner.

const runners = new LlmRunners()

export async function llmprompt(request: FastifyHttp2Request, reply: FastifyHttp2Reply) {
    const reqBody = request.body as Record<string, any>
    const body = { ...reqBody }
    const model = body.model

    for (const runner of runners) {
        // patch the "model" field with the model available for this runner, 
        // if specified, else restore original model that came with request.
        body.model = runner.model.length == 0 ? model: runner.model

        // send prompt and return response
        
    }

    // no runner found
}

Then we assemble together a request addressed to the runner’s OpenAI-compatible API end point, along with the preloaded API key for that runner, and send it to the LLM runner. If we fail to connect to the LLM runner, we log the error and try the next runner on our list. If connection is successful but the runner returns any kind of error, including being rate limited, we again log the error and try the next runner on our list. If the runner completes the prompt, we log the success and forward the completion to the user. Replace //send prompt and return response with:

        const headers: Record<string, string> = {
            "Content-Type": "application/json"
        }
        if (runner.apiKey) {
            headers["Authorization"] = `Bearer ${runner.apiKey}`
        }

        try {
            const response = await fetch(`${runner.baseUrl}/v1/chat/completions`, {
                method: request.method,
                headers,
                body: JSON.stringify(body),
                dispatcher,
            } as RequestInit & { dispatcher: typeof dispatcher })
            
            // If rate limited or server error, 'continue' naturally advances the iterator
            if (!response.ok) {
                logInfo(request, runner.baseUrl, runner.model, response.status, "Server error. Trying another one.")
                if (response.body) {
                    //  cleanup: drop the stream directly to prevent Node target connection leaks
                    await response.body.cancel()
                }
                continue
            }

            // Success! Break the loop and return the streaming response
            logOk(request, runner.baseUrl, runner.model)
            return reply
                .status(response.status)
                .header('Content-Type', response.headers.get('content-type') ?? 'text/event-stream')
                .send(response.body) // Fastify natively streams standard ReadableStreams at max speed

        } catch (error: any) {
            // catch network/connection errors and failover
            logInfo(request, runner.baseUrl, runner.model, HttpStatus.PROCESSING, `Connection error: ${error.message}. Trying another one.`)
            continue
        }

If we’ve exhausted all the runners on our list but couldn’t find any to complete the prompt, we log the error and inform the client. Replace // no runner found with:

    // If the for-loop naturally finishes without returning, we are exhausted.
    return logClientErr(request, reply, "None", HttpStatus.TOO_MANY_REQUESTS, "All available LLM providers are rate limited or offline.")

We’re done with handlers.ts. Save and exit the file.

runners.ts

Your harness needs to connect to an LLM to perform prompt completion. This part of the tutorial connects the harness to an LLM runner of your choice. If you have a list of available LLM runners, we do a weighted-random round-robin selection to pick a runner to complete each incoming HTTP-request. You can also provide a locally provisioned LLM runner to fall back to in case no cloud-based runners are available. Once you set this feature up, we will be using it for the rest of the term but will not be spending more time on it.

Create runners.ts:

server$ vi runners.ts

First we define a struct, LlmRunner, to hold the base URL and API key of an LLM runner. LLM runners each hosts a different set of models. Put the string that identifies the model you want to use to the LLM runner in the model property. We will patch the "model" field of user’s prompt with this string before sending it to this runner. If you want the option to run multiple models from the same runner, create a separate LlmRunner for each model. The weight field records the priority you want to give to this runner. The value of the weight should be in the range [0.0, 1.0], with 1.0 being the highest priority. If a model hosted by Runner A has a weight of 0.8 and two models hosted by Runner B each has a weight of 0.5, you may see Runner B selected more frequently in your trace. The weights don’t have to add up to 1.0. Add the following code to your file:

interface LlmRunner {
    baseUrl: string
    model: string
    weight: number
    apiKey: string | null
}

The weighted-random round-robin selection is implemented as the [Symbol.iterator]() method of an Iterable interface. If the selected runner cannot complete the prompt, we move on to the next one on the list. Failover runners are also selected in a weighted-random round-robin manner. Add the following code to create an iterable list of LlmRunners:

export class LlmRunners implements Iterable<LlmRunner> {
    private runners: LlmRunner[] = []
    private local: LlmRunner | null

    // The iterator that powers the `for (const runner of runners)` loop:
    // to iterate through the runner array randomized per HTTP-request to load balance    
    // Implemented using the Schwartzian Transform (Decorate-Sort-Undecorate).
    [Symbol.iterator](): Iterator<LlmRunner> {
        // Calculate the Efraimidis and Spirakis A-Res scores based on weights
        // "Weighted Random Sampling with a Reservoir", Information Processing Letters, 97:5, March 2006.
        // Each runner gets a randomized score that, when sorted, determines
        // the fallback order for each HTTP-request.
        // Each HTTP-request gets a different ordered list, effecting the
        // round-robin selection of runners across HTTP-requests.
        const scored = this.runners.map(runner => {
            const score = -Math.log(Math.random()) / (runner.weight > 0.0 ? runner.weight : 1e-9)
            return { score, runner } // decorate
        })

        // Sort ascending by score.        
        scored.sort((a, b) => a.score - b.score)

        // undecorate: extract just the ordered references and dynamically append the local fallback.
        const sorted = scored.map(item => item.runner)
        if (this.local) {
            sorted.push(this.local)
        }

        return sorted.values()
    }

    // constructor()
}

Finally, constructor() initializes the list of runners. The code here lists LLM runners with free tiers. You have to provide your own API key for each runner in an environment variable. If you have subscription to other LLM runners (such as Claude or ChatGPT), you can add them to the list. If you have provisioned for a local LLM as shown in llmprompt-backend, you can assign it to the local variable as your ultimate fallback server. Replace // constructor() comment with:

    constructor() {
        this.local = null
        // If you have provisioned a local llama-server, replace null above with:
        //{ baseUrl: "http://localhost:11433", model: "", weight: 0.0, apiKey: null }

        const apiKeys = {
            cerebras: process.env.CEREBRAS_API_KEY || null,
            groq: process.env.GROQ_API_KEY || null,
            mistral: process.env.MISTRAL_API_KEY || null,
            gemini: process.env.GEMINI_API_KEY || null,
            google: process.env.GOOGLE_API_KEY || null,
            nvidia: process.env.NVIDIA_API_KEY || null,
            //openrouter: process.env.OPENROUTER_API_KEY || null,
        }

        const llmRunners: LlmRunner[] = [
            {
                baseUrl: "https://api.cerebras.ai",
                apiKey: apiKeys.cerebras,
                model: "zai-glm-4.7",
                weight: 0.3
            },
            {
                baseUrl: "https://api.groq.com/openai",
                apiKey: apiKeys.groq,
                model: "llama-3.1-8b-instant",
                weight: 0.4
            },
            {
                baseUrl: "https://api.groq.com/openai",
                apiKey: apiKeys.groq,
                model: "meta-llama/llama-4-scout-17b-16e-instruct",
                weight: 0.4
            },
            {
                baseUrl: "https://api.mistral.ai",
                apiKey: apiKeys.mistral,
                model: "ministral-8b-latest",
                weight: 0.2
            },
            {
                baseUrl: "https://api.mistral.ai",
                apiKey: apiKeys.mistral,
                model: "open-mistral-nemo",
                weight: 0.2
            },
            {
                baseUrl: "https://integrate.api.nvidia.com",
                apiKey: apiKeys.nvidia,
                model: "meta/llama-3.1-8b-instruct",
                weight: 0.2
            },
            {
                baseUrl: "https://integrate.api.nvidia.com",
                apiKey: apiKeys.nvidia,
                model: "deepseek-ai/deepseek-v4-flash",
                weight: 0.2
            },
            {
                baseUrl: "https://integrate.api.nvidia.com",
                apiKey: apiKeys.nvidia,
                model: "nvidia/nemotron-nano-12b-v2-vl",
                weight: 0.3
            },
            /*
            * Don't include Open Router in swap_remove() reshuffling.
            * If added, should add it as LlmRunners.fallback to try
            * as last resort before local llm, but after all the others,
            * due to its low rate limit and variable model availability
            { 
                baseUrl: "https://openrouter.ai/api", 
                apiKey: apiKeys.openrouter, 
                model: "openrouter/free", // whichever best model is available and least loaded
                weight: 0.1
            },
            */
        ]

        // loop to filter out missing API Key and push
        for (const runner of llmRunners) {
            if (runner.apiKey) {
                console.error(`Runner ${runner.baseUrl} ACTIVE.`)
                this.runners.push(runner)
            } else {
                console.error(`Runner ${runner.baseUrl} INACTIVE. No API KEY.`)
            }
        }

        if (this.runners.length === 0 && !this.local) {
            console.error("CRITICAL: No API keys found and no local LLM configured. Terminating process.")
            process.exit(1)
        }
    }

We’re done with runners.ts. Save and exit the file.

Build and Test run

:point_right:To run, TypeScript must first be transpiled into JavaScript. This means you must run pnpm build each and every time you made changes to your code, for the changes to show up when you run node.

To build your server, transpile TypeScript into JavaScript:

server$ pnpm build

You should now see a harnessd.js file in the directory.

Storing API keys

When we have only one or two environment variables to set, we can do it on the command-line interface (CLI) right before running harnessd, e.g.,

server$ sudo LLM_RUNNER_API_KEY=YOUR_API_KEY node harnessd.js

When you have multiple API keys, however, this becomes impractical. A not very secure option is to store your API keys in /root/.bashrc or /root/.zshrc (:warning: make sure you don’t push this file to your git repo!):

server$ sudo su
# You are now root, note the command-line prompt changed from '$' to '#'.
# You can do a lot of harm with root privileges, so be very careful what you do here.
server# vi /root/.bashrc

Add these to the end of the file and provide your available API keys:

export CEREBRAS_API_KEY=""
export GROQ_API_KEY=""
export GEMINI_API_KEY=""
export MISTRAL_API_KEY=""
export NVIDIA_API_KEY=""
export OPENROUTER_API_KEY=""

export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmimalloc.so
export MIMALLOC_ARENA_RESERVE=320MiB
export NODE_ENV=production

Note that we have also added an environment variable to point to the mimalloc memory allocator we’re using with harnessd. Mimalloc uses memory more efficiently and supports higher concurrency than either the V8 or Linux malloc or jemalloc, especially for harness architecture workload. After you add these environment variables to your /root/.bashrc, you’d have to do:

server# source /root/.bashrc

to load them into your current shell. Subsequent sudo su will load them automatically.

Then you can just run your harnessd:

server# node harnessd.js     # NOTE `.js` not `.ts`
# Hit ^C to end the test
server# exit
# So that you're no longer root.
server$
`dotenv`

The dotenv library is often used to read .env file. However, common container, serverless, and managed runtime platforms (such as AWS ECS/Fargate and Lambda, Cloudflare Workers, Heroku, Kubernetes, Vercel, etc.), inject configurations directly into process memory, bypassing physical files. We thus import environment variables directly from the parent (shell) process when you run harnessd from the CLI or access them indirectly through systemd service configuration file.

:warning:Now return to the back-end cover page to complete a TODO item and test your implementation following the instructions in the subsequent “Testing llmPrompt APIs” section. You must complete ithe TODO to get full credit for this tutorial.

References

Prepared by Chenglin Li, Xin Jie ‘Joyce’ Liu, Sugih Jamin Last updated June 23rd, 2026