`Go` with `Echo`

Install `Go`

ssh to your server and download the latest version of Go: Check Go’s Downloads page for the latest version. Replace “LATEST_VERSION” below with the latest version:

server$ cd /tmp
server$ wget https://go.dev/dl/goLATEST_VERSION.linux-amd64.tar.gz
server$ sudo rm -rf /usr/local/go
server$ sudo tar -C /usr/local -xzf goLATEST_VERSION.linux-amd64.tar.gz
server$ sudo chmod -R go+rX /usr/local/go

use Go version 1.26 or later for better CGo scheduling.

update

To update to a later version of Go, repeat the instructions above—which would have you manually delete the existing Go folder (usually /usr/local/go/), so don’t put any custom files there.

To upgrade all dependent packages to their latest versions:

  server$ go get -u

First create and change into a directory where you want to keep your harnessd module:

server$ mkdir ~/agentic/harnessd
server$ cd ~/agentic/harnessd

Create a Go module called harnessd:

server$ go mod init harnessd
# output:
go: creating new go.mod: module harnessd

`main.go`

Create a file called main.go:

server$ vi main.go

We will put the server and URL routing code in main.go, starting with the following import lines:

package main

import (
	"log"
  
	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

We next create a routing table to hold the URL routing information needed by Echo and assign it to a global variable routes. We define the route to serve the HTTP POST request with URL endpoint /llmprompt. We route the endpoint to the llmprompt() function. With each route, we specify which HTTP method is allowed for the URL endpoint, in this case the endpoint accepts only an HTTP POST request:

type Route struct {
    httpMethod  string
    urlPath    string
    urlHandler echo.HandlerFunc
}

var routes = []Route {
		{"POST", "/llmprompt/", llmprompt},
}

The function llmprompt() will be implemented in handlers.go later.

For now, staying in main.go, in the main() function, set up the Echo server:

disable printing the Echo banner on the console,
load up the routes array above,
set it to automatically reroute any URL specified without a trailing ‘/’ to one with trailing ‘/’; for example https://YOUR_SERVER_IP/llmprompt will be rerouted to https://YOUR_SERVER_IP/llmprompt/,
bind it to the default HTTPS port (443),
assign it the harnessd certificate and key we created earlier, and . . .

launch the server:

func main() {
	server := echo.New()
	server.HideBanner = true
	for _, route := range routes {
		server.Match([]string{route.httpMethod}, route.urlPath, route.urlHandler)
	}
	server.Pre(middleware.AddTrailingSlash())

	log.Fatal(server.StartTLS(":443",
		"/home/ubuntu/agentic/harnessd.crt",
		"/home/ubuntu/agentic/harnessd.key"))
}

We’re done with main.go. Save and exit the file.

`handlers.go`

We implement URL path API handlers in handlers.go:

server$ vi handlers.go

Start the file with the following imports:

package main

import (
    "bytes"
    "fmt"
    "io"
    "log"
    "net/http"

    "github.com/goccy/go-json"
    "github.com/labstack/echo/v4"
)

We add logging functions to print to console results of handling each HTTP request:

func logOk(c echo.Context, runner string, model string) {
    log.Println("[Echo] |", http.StatusOK, `|`, c.RealIP(), `|`, runner, `:`, model, `|`, c.Request().Method, c.Request().RequestURI)
}
func logInfo(c echo.Context, runner string, model string, errcode int, msg string) {
    log.Println("[Echo] |", errcode, `|`, c.RealIP(), `|`, runner, `:`, model, `|`, msg)
}

func logServerErr(c echo.Context, runner string, model string, err error) error {
    log.Println("[Echo] |", http.StatusInternalServerError, `|`, c.RealIP(), `|`, runner, `:`, model, `|`, c.Request().Method, c.Request().RequestURI, err.Error())
	return c.JSON(http.StatusInternalServerError, err.Error())
}

func logClientErr(c echo.Context, runner string, model string, errcode int, err error) error {
    log.Println("[Echo] |", errcode, `|`, c.RealIP(), `|`, runner, `:`, model, `|`, c.Request().Method, c.Request().RequestURI, err.Error())
	return c.JSON(errcode, err.Error())
}

We add a flushWriter wrapper for Echo’s writer. Echo buffers writes into ~4 KB chunks before sending them, which would make streaming LLM response stutter. Wrapping its writer in flushWriter forces Echo to Flush() each Write() as it occurs, restoring the typewriter-style streaming of LLM completion.

// flushWriter forwards Write to the Echo response and flushes after
// each write so SSE events are forwarded to the client as they arrive.
type flushWriter struct{ response *echo.Response }

func (fw flushWriter) Write(p []byte) (int, error) {
    n, err := fw.response.Writer.Write(p)
    fw.response.Flush()
    return n, err
}

We next define the handler llmprompt(), which first pick an LLM runner from a ranked list of available runners. Since different runner provides different models, we patch the "model" field of the incoming prompt to the model available at the runner. We use http.DefaultClient to connect to the LLM runner.

var runners = NewLlmRunners()

func llmprompt(c echo.Context) error {
	req := c.Request()

	var body map[string]any
	if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
		return logClientErr(c, "All", "", http.StatusBadRequest, err)
	}
	model, _ := body["model"].(string)

	// if Iter alternate:	for runner := range runners.Iter() {
	for _, runner := range runners.RoundRanked() {
	    // patch the "model" field with the model available for this runner, 
        // if specified, else restore original model that came with request.
		if runner.Model == "" { body["model"] = model } else { body["model"] = runner.Model }

        // send prompt and return response
        
    }

    // no runner found
}

Then we assemble together a request addressed to the runner’s OpenAI-compatible API end point, along with the preloaded API key for that runner, and send it to the LLM runner. If we fail to connect to the LLM runner, we log the error and try the next runner on our list. If connection is successful but the runner returns any kind of error, including being rate limited, we again log the error and try the next runner on our list. If the runner completes the prompt, we log the success and forward the completion to the user. Replace //send prompt and return response with:

		requestBody, err := json.Marshal(body)
		if err != nil {
			continue
		}

		request, err := http.NewRequestWithContext(
			req.Context(), // propagate client disconnects!
			req.Method,
			runner.BaseUrl+"/v1/chat/completions",
			bytes.NewReader(requestBody),
		)
		if err != nil {
			continue
		}

		request.Header.Set("Content-Type", "application/json")
		if runner.ApiKey != "" {
			request.Header.Set("Authorization", "Bearer "+runner.ApiKey)
		}

		response, err := http.DefaultClient.Do(request)
		if err != nil {
		    // catch network/connection errors and failover
            logInfo(c, runner.BaseUrl, runner.Model, http.StatusProcessing, 
                "Connection error: "+err.Error()+" Trying another one.")
			continue
		}

		// If rate limited or server error, 'continue' naturally advances the iterator
		if response.StatusCode != http.StatusOK {
		    logInfo(c, runner.BaseUrl, runner.Model, response.StatusCode,
                "Server error. Trying another one.")
			response.Body.Close() // cleanup: prevent goroutine leaks!
			continue
		}
		defer response.Body.Close()

		// Success! Break the loop and stream the response to the client
		// Prepare SSE stream header to be sent to client
		res := c.Response()
		res.Header().Set(echo.HeaderContentType, response.Header.Get("Content-Type"))
		res.Header().Set(echo.HeaderCacheControl, "no-cache")
		res.WriteHeader(response.StatusCode)

		// io.Copy streams chunks efficiently without loading the whole response in RAM
		_, err = io.Copy(flushWriter{res}, response.Body)
		if err != nil {
		    return logServerErr(c, runner.BaseUrl, model, err)
		}

		logOk(c, runner.BaseUrl, runner.Model)
		return nil
	}
	

If we’ve exhausted all the runners on our list but couldn’t find any to complete the prompt, we log the error and inform the client. Replace // no runner found with:

	// If the for-loop naturally finishes without returning, we are exhausted.
	return logClientErr(c, "None", model, http.StatusTooManyRequests, fmt.Errorf("All available LLM providers are rate limited or offline."))
}

We’re done with handlers.go. Save and exit the file.

`runners.go`

Your harness needs to connect to an LLM to perform prompt completion. This part of the tutorial connects the harness to an LLM runner of your choice. If you have a list of available LLM runners, we do a weighted-random round-robin selection to pick a runner to complete each incoming HTTP-request. You can also provide a locally provisioned LLM runner to fall back to in case no cloud-based runners are available. Once you set this feature up, we will be using it for the rest of the term but will not be spending more time on it.

Create runners.go:

server$ vi runners.go

Start the file with the following imports:

package main

import (
	"cmp"
	"log"
	"math"
	"math/rand/v2"
	"os"
	"slices"
)

Next we define a struct, LlmRunner, to hold the base URL and API key of an LLM runner. LLM runners each hosts a different set of models. Put the string that identifies the model you want to use to the LLM runner in the Model property. We will patch the "model" field of user’s prompt with this string before sending it to this runner. If you want the option to run multiple models from the same runner, create a separate LlmRunner for each model. The weight field records the priority you want to give to this runner. The value of the weight should be in the range [0.0, 1.0], with 1.0 being the highest priority. If a model hosted by Runner A has a weight of 0.8 and two models hosted by Runner B each has a weight of 0.5, you may see Runner B selected more frequently in your trace. The weights don’t have to add up to 1.0. Add the following code to your file:

type LlmRunner struct {
	BaseUrl string
	ApiKey  string
	Model   string
	weight  float64
}

The weighted-random round-robin selection is implemented as returning a simple array of pointers to LlmRunner. If the selected runner cannot complete the prompt, we move on to the next one on the list. Failover runners are also selected in a weighted-random round-robin manner. Add the following code to create an iterable list of LlmRunners:

type LlmRunners struct {
	runners []LlmRunner
	local   *LlmRunner
}
// The Schwartzian Transform (Decorate-Sort-Undecorate)
// returns a runners array weighted-randomized per HTTP-request to load balance
func (llmRunners *LlmRunners) RoundRanked() []*LlmRunner {
    // Calculate the Efraimidis and Spirakis A-Res scores based on weights
	// "Weighted Random Sampling with a Reservoir", Information Processing Letters, 97:5, March 2006.
	// Each runner gets a randomized score that, when sorted, determines
	// the fallback order for each HTTP-request.
	// Each HTTP-request gets a different ordered list, effecting the
	// round-robin selection of runners across HTTP-requests.
	//
	// decorate: bind scores to pointers (zero-copy sorting)
	type decorated struct {
		score float64
		runner *LlmRunner
	}

	scored := make([]decorated, len(llmRunners.runners))
	for i := range llmRunners.runners {
		weight := llmRunners.runners[i].weight
		if weight <= 0.0 {
			weight = 1e-9
		}
		
		scored[i] = decorated{
		    // rand/v2 is globally concurrent and ultra-fast
			score: -math.Log(rand.Float64()) / weight,
			runner: &llmRunners.runners[i],
		}
	}

	// Sort ascending by score using Go 1.21+ slices package
	slices.SortFunc(scored, func(a, b decorated) int {
		return cmp.Compare(a.score, b.score) // Blazing fast native float comparison
	})

	// undecorate: extract and yield just the ordered references
	numrunners := len(llmRunners.runners)
	if llmRunners.local != nil {
		numrunners++
	}

	runners := make([]*LlmRunner, 0, numrunners)
	for _, item := range scored {
		runners = append(runners, item.runner)
	}

	// append the local fallback if it exists
	if llmRunners.local != nil {
		runners = append(runners, llmRunners.local)
	}

	return runners
}

Finally, NewLlmRunners() initializes the list of runners. The code here lists LLM runners with free tiers. You have to provide your own API key for each runner in an environment variable. If you have subscription to other LLM runners (such as Claude or ChatGPT), you can add them to the list. If you have provisioned for a local LLM as shown in llmprompt-backend, you can assign it to the local variable as your ultimate fallback server. Add the following code:

func NewLlmRunners() *LlmRunners {
	// 1. Fetch from environment
	apiKeys := map[string]string{
		"cerebras": os.Getenv("CEREBRAS_API_KEY"),
		"groq":     os.Getenv("GROQ_API_KEY"),
		"mistral":  os.Getenv("MISTRAL_API_KEY"),
		"gemini":   os.Getenv("GEMINI_API_KEY"),
		"nvidia":   os.Getenv("NVIDIA_API_KEY"),
		//"openrouter":  os.Getenv("OPENROUTER_API_KEY"),
	}

	// 2. Define the full list of potential runners
	llmRunners := []LlmRunner{
	{
	    BaseUrl: "https://api.cerebras.ai", 
		ApiKey: apiKeys["cerebras"], 
		Model: "zai-glm-4.7", 
		weight: 0.3,
	},
	{
	    BaseUrl: "https://api.groq.com/openai", 
		ApiKey: apiKeys["groq"], 
		Model: "llama-3.1-8b-instant", 
		weight: 0.4,
	},
	{
	    BaseUrl: "https://api.groq.com/openai", 
		ApiKey: apiKeys["groq"], 
		Model: "meta-llama/llama-4-scout-17b-16e-instruct", 
		weight: 0.4,
	},
	{
	    BaseUrl: "https://api.mistral.ai", 
		ApiKey: apiKeys["mistral"], 
		Model: "ministral-8b-latest", 
		weight: 0.3,
	},
	{
	    BaseUrl: "https://api.mistral.ai", 
		ApiKey: apiKeys["mistral"], 
		Model: "open-mistral-nemo", 
		weight: 0.2,
	},
	{
	    BaseUrl: "https://integrate.api.nvidia.com", 
		ApiKey: apiKeys["nvidia"], 
		Model: "meta/llama-3.1-8b-instruct", 
		weight: 0.2,
	},	
	{
	    BaseUrl: "https://integrate.api.nvidia.com", 
		ApiKey: apiKeys["nvidia"], 
		Model: "deepseek-ai/deepseek-v4-flash", 
		weight: 0.2,
	},
	{
	    BaseUrl: "https://integrate.api.nvidia.com", 
		ApiKey: apiKeys["nvidia"], 
		Model: "nvidia/nemotron-nano-12b-v2-vl", 
		weight: 0.3,
	},			
	/*
	* Don't include Open Router in swap_remove() reshuffling.
	* If added, should add it as LlmRunners.fallback to try
	* as last resort before local llm, but after all the others,
	* due to its low rate limit and variable model availability
	{ 
	    BaseUrl: "https://openrouter.ai/api", 
    		ApiKey: apiKeys["openrouter"], 
    		Model: "openrouter/free", // whichever best model is available and least loaded
    		weight: 0.1 
	},
	*/
}

	runners := &LlmRunners{
		runners: make([]LlmRunner, 0), // initialize empty slice
		local: nil,
		// If you have provisioned a local llama-server, replace nil above with:
		//&LlmRunner{ BaseUrl: "http://localhost:11433", ApiKey: "", Model: "", weight: 0.0 },
	}

	// loop and filter out runners missing API keys
	for _, runner := range llmRunners {
		if runner.ApiKey != "" {
			log.Printf("Runner %s ACTIVE.", runner.BaseUrl)
			runners.runners = append(runners.runners, runner)
		} else {
			log.Printf("Runner %s INACTIVE. No API KEY.", runner.BaseUrl)
		}
	}

	if len(runners.runners) == 0 && runners.local == nil {
		log.Fatal("CRITICAL: No API keys found and no local LLM configured. Terminating process.") 
		// log.Fatal calls os.Exit(1)
	}

	return runners
}

We’re done with runners.go. Save and exit the file.

Build and test run

To build your server:

server$ go get   # -u  # to upgrade all packages to the latest version
server$ go build

Go is a compiled language, like C/C++ and unlike Python, which is an interpreted language. This means you must run go build each and every time you made changes to your code, for the changes to show up in your executable.

Storing API keys

When we have only one or two environment variables to set, we can do it on the command-line interface (CLI) right before running harnessd, e.g.,

server$ sudo LLM_RUNNER_API_KEY=YOUR_API_KEY node harnessd.js

When you have multiple API keys, however, this becomes impractical. A not very secure option is to store your API keys in /root/.bashrc or /root/.zshrc ( make sure you don’t push this file to your git repo!):

server$ sudo su
# You are now root, note the command-line prompt changed from '$' to '#'.
# You can do a lot of harm with root privileges, so be very careful what you do here.
server# vi /root/.bashrc

Add these to the end of the file and provide your available API keys:

export CEREBRAS_API_KEY=""
export GROQ_API_KEY=""
export GEMINI_API_KEY=""
export MISTRAL_API_KEY=""
export NVIDIA_API_KEY=""
export OPENROUTER_API_KEY=""

export GOMEMLIMIT=320MiB

After you add these environment variables to your /root/.bashrc, you’d have to do:

server# source /root/.bashrc

to load them into your current shell. Subsequent sudo su will load them automatically.

Then you can just run your harnessd:

server# ./harnessd
# Hit ^C to end the test
server# exit
# So that you're no longer root.
server$

`dotenv`

The dotenv library is often used to read .env file. However, common container, serverless, and managed runtime platforms (such as AWS ECS/Fargate and Lambda, Cloudflare Workers, Heroku, Kubernetes, Vercel, etc.), inject configurations directly into process memory, bypassing physical files. We thus import environment variables directly from the parent (shell) process when you run harnessd from the CLI or access them indirectly through systemd service configuration file.

Now return to the back-end cover page to complete a TODO item and test your implementation following the instructions in the subsequent “Testing llmPrompt APIs” section. You must complete ithe TODO to get full credit for this tutorial.

References

Working with Go
- Golang tutorial series gentle and clear, though a bit out of date in parts now.
Running multiple HTTP servers in Go
Making a RESTful JSON API in Go
Golang Json Marshal Example
HTTP Status Code
Effective Error Handling in Golang
URL
Single Host Reverse Proxy

Prepared by Chenglin Li, Xin Jie ‘Joyce’ Liu, Sugih Jamin

Last updated June 23rd, 2026

`Go` with `Echo`

Cover Page

Back-end Page

Install `Go`

`main.go`

`handlers.go`

`runners.go`

Build and test run

Storing API keys

References

Go with Echo

Cover Page

Back-end Page

Install Go

main.go

handlers.go

runners.go

Build and test run

Storing API keys

References

`Go` with `Echo`

Install `Go`

`main.go`

`handlers.go`

`runners.go`