Tutorial: llmPrompt Back End

Cover Page

All the tutorials and projects in this course require access to a back-end server to host your harness. You may want to keep the server you set up here running for the duration of the term.

harnessd serving as HTTP/2 proxy for an LLM runner

Harness hosting

You need an Internet-accessible server with a public IP address and is online at all times (i.e., it cannot be running on your laptop) to host your harness. And your server must be running the Ubuntu 24.04 operating system. You can use a real physical host or a virtual machine on Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, AlibabaCloud, etc. as long as you can ssh to an Ubuntu shell with root privileges. Apparently GCP offers a permanent Always Free tier that includes a free micro virtual machine. It’s worth checking out, especially since AWS’s free tier is limited to only 6 months. The setup presented here has been verified to work on Ubuntu 24.04 hosted on AWS, GCP, a local Linux KVM and on a physical host.

Please click the relevant link below to view the instructions to set up your host and return here to resume the server setup once you have your instance running.

GCP | AWS

Updating packages

Login/ssh to your newly instantiated server and run the following:

server$ sudo apt update
server$ sudo apt upgrade

If you see *** System restart required *** when you ssh to your server, immediately run:

server$ sync
server$ sudo reboot

Your ssh session will be ended at the server. Wait a few minutes for the system to reboot before you ssh to your server again.

Clone your course tutorial repo

Clone your course tutorial GitHub repo so that you can push your back-end files for submission:

• First, on your browser, navigate to your course tutorial GitHub repo
• Click the green Code button and copy the URL to your clipboard by clicking the clipboard icon next to the URL
• Then on your back-end server:

    server$ cd ~
    server$ git clone <paste the URL you copied above> agentic
  

  If you haven’t, you would need to create a personal access token to use HTTPS Git.

If all goes well, your assignment repo should be cloned to ~/agentic. Check that:

server$ ls ~/agentic

shows the content of your agentic git repo, including your Agent front end.

Preparing certificates for HTTPS

Apple required apps to use HTTPS, the secure version of HTTP. Android has followed suite and defaulted to blocking all cleartext (HTTP) traffic.

To support HTTPS, we first need a public key signed by a Certification Authority (CA). Obtaining a certificate from a legitimate CA requires that your server have a fully qualified domain name (FQDN), such as www.eecs.umich.edu, which would require extra set up. Instead, we opt to generate our own certificate, which can only be used during development–good enough for the course.

Start by installing wget so that we can grab the configuration file, selfsigned.cnf, which we will use to generate a certificate and its corresponding private key:

  server$ sudo apt install wget
  server$ hash -r
  server$ wget https://reactive.eecs.umich.edu/asns/selfsigned.cnf

Open selfsigned.cnf:

  server$ vi selfsigned.cnf

When asked to open and/or edit a file on your back-end server, use your favorite editor. In this and all subsequent tutorials, we will assume vi (or vim or nvim) because it has the shortest name 😊. You can replace vi with your favorite editor; for example, nano has on-screen help and may be easier to pick up. See cheatsheet for a list of vi’s commands.

search for the string YOUR_SERVER_IP and replace it with yours:

  extendedKeyUsage = serverAuth
  #                  👇👇👇👇👇👇👇
  subjectAltName = IP:YOUR_SERVER_IP   # replace YOUR_SERVER_IP with yours

Now create a self-signed key-certificate pair with the following command; here also, replace YOUR_SERVER_IP in the command line with yours:

  server$ openssl req -x509 -newkey rsa:4096 -sha256 -keyout /home/ubuntu/agentic/harnessd.key -out /home/ubuntu/agentic/harnessd.crt -days 100 -subj "/C=US/ST=MI/L=AA/O=UM/OU=CSE/CN=YOUR_SERVER_IP" -config selfsigned.cnf -nodes
  # replace YOUR_SERVER_IP with yours

We will use the generated key and certicate files later.

Web server framework and harnessd

We provide instructions to set up the harness using different back-end stacks.

Please click the relevant link to setup the harnessd server with llmprompt API using the web framework of your choice:

Go | Python | Rust | TypeScript

and return here to resume the server setup once you have your web framework running.

server$ sudo systemctl stop harnessd
server$ sudo systemctl daemon-reload
server$ sudo systemctl start harnessd

TODO (graded!)

Add a route for an HTTP GET / API end point and route it to a top() handler. The top() handler simply returns a JSON containing the string “UM EECS agentic harnessd” and, by default, HTTP status code OK (200).

Testing llmprompt APIs

There are several ways to test HTTP APIs. You can use a REST API cient with a graphical interface or you can use a command-line tool.

Your harnessd must be running on your back-end host. If it’s not running, there is no server to test.

with a REST API client

To test HTTP APIs graphically, you could use a REST API client such as Postman, Apidog, or, if you use VSCode, the EchoAPI plugin. On Android or iOS device and emulator/simulator, you can use Teste - API, downloadable from both Apple AppStore and Google PlayStore.

The new REST API client Apidog can auto-merge the return stream and display it as scrolling text. It is a bit more opinionated and thus harder to learn than Postman, but the Auto-Merge feature for viewing LLM completion is indispensable.

Postman

The instructions for Postman in this and subsequent tutorials are for the desktop version. If you’re proficient with the web version, you can use the web version.

• First for Settings > General > Request > HTTP version choose HTTP/2 on the drop down menu. Then three items down, disable Settings > General > SSL certificate verification. We use self-signed certificate that cannot be verified with a trusted authority.

• Then in the main Postman screen, next to the Overview tab, click the + tab. You get a new Untitled Request screen. You should see GET listed under Untitled Request.

• Click on GET to show the drop down menu and select POST.

• Enter https://YOUR_SERVER_IP/llmprompt/ in the field next to POST.

• Under the server URL you entered, there’s a menu with Body as the fourth element. Click on Body.

• You should see a submenu under Body. Click on raw.

• At the end of the submenu, click on TEXT and replace it with JSON in the drop down menu.

• You can now enter the following in the box under the submenu:

{
    "model": "gemma3:270m",
    "messages": [
      { "role": "user", "content": "howdy?"}
    ],
    "stream": true
}

and click the big blue Send button.

If everything works as expected, the bottom pane of Postman should say, to the right of its menu line, 200 OK, and the pane should display OpenAI’s SSE stream. If you’re using the provided llama-server, the first JSON looks something like:

{
    "choices": [
        {
            "finish_reason": null,
            "index": 0,
            "delta": {
                "role": "assistant",
                "content": null
            }
        }
    ],
    "created": 1781690778,
    "id": "chatcmpl-LeLlnEvnjbZAveeg9QgABzJmAfWoR8Cw",
    "model": "gemma3:270m",
    "system_fingerprint": "b9656-581e8eca8",
    "object": "chat.completion.chunk"
}

with a command-line tool

For all of the below, if you’re running a server without HTTPS, replace https with http.

laptop$ curl -i -X POST -H "Content-Type: application/json" -d '{ "model": "gemma3:270m", "messages": [ { "role": "user", "content": "howdy?"} ], "stream": true }' -kL https://YOUR_SERVER_IP/llmprompt/
# output:
HTTP/2 200
server: llama.cpp
content-type: text/event-stream
access-control-allow-origin:
date: Wed, 17 Jun 2026 10:33:12 GMT

data: {"choices":[{"finish_reason":null,"index":0,"delta":{"role":"assistant","content":null}}],"created":1781692392,"id":"chatcmpl-2nXU2XB3p2MW21VHomxZOSmdMv2HD846","model":"gemma3:270m","system_fingerprint":"b9656-581e8eca8","object":"chat.completion.chunk"}
# and a few more lines . . . .

laptop$ echo '{ "model": "gemma3:270m", "messages": [ { "role": "user", "content": "howdy?"} ], "stream": true }' | http --verify=no POST https://YOUR_SERVER_IP/llmprompt/
# output:
HTTP/1.1 200 OK
access-control-allow-origin:
connection: close
content-type: text/event-stream
date: Wed, 17 Jun 2026 10:20:01 GMT
keep-alive: timeout=5, max=100
server: llama.cpp
transfer-encoding: chunked

data: {
    "choices": [
        {
            "delta": {
                "content": null,
                "role": "assistant"
            },
            "finish_reason": null,
            "index": 0
        }
    ],
    "created": 1781691601,
    "id": "chatcmpl-vAxze9iMVWniY7MoLjdMCgU6VBosW12I",
    "model": "gemma3:270m",
    "object": "chat.completion.chunk",
    "system_fingerprint": "b9656-581e8eca8"
}
# and a few more lines . . . .

Automatic harnessd restart

Once you have your server tested, to run it automatically on system reboot or on failure, first copy the /home/ubuntu/.env file you created in the “Storing API keys” section of your respective web framework tutorial:

server$ cp /home/ubuntu/.env /home/ubuntu/.env.sys

Edit .env.sys to remove all occurrences of export ’s.

Next create the service configuration file:

server$ sudo vi /etc/systemd/system/harnessd.service

and put the following lines in it:

[Unit]
Description=UM EECS Agentic harnessd
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=on-failure
RestartSec=1
User=ubuntu
Group=www-data
WorkingDirectory=/home/ubuntu/agentic/harnessd

# Allow non-root user to bind to port 443 and read
SecureBits=keep-caps
Capabilities=cap_net_bind_service+ep
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

# Copy of `/home/ubuntu/.env` with all `export `'s deleted:
EnvironmentFile=/home/ubuntu/.env.sys

# >>>>>> Uncomment one of the following choices for your web framework of choice <<<<<<

# >>>>>> Go or Rust <<<<<<
#ExecStart=/home/ubuntu/agentic/harnessd/harnessd

# >>>>>> Python <<<<<<
#ExecStart=/home/ubuntu/agentic/harnessd/.venv/bin/python3 -u harnessd.py

# >>>>>> TypeScript <<<<<<
#ExecStart=/usr/bin/node harnessd.js

[Install]
WantedBy=multi-user.target

To test the service, run:

server$ sudo systemctl daemon-reload
server$ sudo systemctl start harnessd
server$ systemctl status harnessd
# first 3 lines of output:
● harnessd.service - UM EECS Agentic harnessd
     Loaded: loaded (/etc/systemd/system/harnessd.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2026-08-15 01:28:56 EDT; 2min 30s ago
     . . .   👆👆👆👆👆👆👆👆

The last line should say, Active: active (running).

To have the system restart automatically upon reboot, run:

server$ sudo systemctl enable harnessd
server$ systemctl status harnessd
# first 2 lines of output:
● harnessd.service - UM EECS Agentic harnessd
     Loaded: loaded (/etc/systemd/system/harnessd.service; enabled; vendor preset: enabled)
     . . .                                                 👆👆👆👆

The second field inside the parentheses in the second line should now say “enabled.”

For each tutorial, leave your harnessd running until you have received your tutorial grade!

To view harnessd’s console output, run:

server$ sudo systemctl status harnessd

To view continuous status updates from harnessd, run:

server$ sudo journalctl -fu harnessd

server$ sudo systemctl status harnessd

Everytime you rebuild your Go or Rust server or make changes to either of your TypeScript or Python files, you need to restart harnessd:

server$ sudo systemctl restart harnessd

If you change the /etc/systemd/system/harnessd.service configuration file, run:

server$ sudo systemctl daemon-reload

before restarting your harnessd.

To turn off auto restart:

server$ sudo systemctl disable harnessd

WARNING: You will not get full credit if your front end is not set up to work with your back end!

That’s all we need to do to prepare the back end. Before you return to work on your front end, wrap up your work here by submitting your files to GitHub.

Submitting your back end

We will only grade files committed to the main branch. If you use multiple branches, please merge them all to the main branch for submission.

Navigate to your agentic folder:

server$ cd ~/agentic/

Run:

server$ cp /etc/systemd/system/harnessd.service harnessd/

Then run:

server$ git add harnessd

You can check git status by running:

server$ git status

You should see the newly added files.

Commit changes to the local repo:

server$ git commit -am "harnessd"

and push your harnessd folder to the remote GitHub repo:

server$ git push

If git push failed due to changes made to the remote repo by your tutorial partner, you must run git pull first. Then you may have to resolve any conflicts before you can git push again.

Go to the GitHub website to confirm that your back-end files have been uploaded to your GitHub repo.

You can now return to complete the front end: Android | iOS.

References

Setup

• Ubuntu setup
• GCP Instructions to set up ssh access with public key pair.
• chmod in WSL
• Creating a Linux service with systemd
  • How to add your own services to systemd for easier management
  • How to create a systemd service in Linux

Security

• What is HTTPS
• Everything about HTTPS and SSL
• New self-signed SSL Certificate for iOS 13
  • Requirements for trusted certificates in iOS 13 and macOS 10.15
  • Missing X.509 extensions with an openssl-generated certificate
  • iOS SSL-related error code
• Self-Signed SSL Certificate for Nginx
• Connecting mobile apps (iOS and Android) to backends for development with SSL
  • Android Network Security Configuration
  • Network security config for range of ip addresses?
• How to do the CHMOD 400 Equivalent Command on Windows
  • icacls

LLMs

• A Very Gentle Introduction to Large Language Models without the Hype
• Large language models, explained with a minimum of math and jargon
• How computers got shockingly good at recognizing images
• Few-Shot Prompting
• A Deep Dive into Building Enterprise grade Generative AI Solutions

Appendix A

Optional: self hosting llama-server

Should you choose to self host an LLM, we provide a llama-server LLM runner you can run on a 1GB *-micro instance. On such a resource-constraint host, however, the LLM models you can run are limited to gemma3:270m (~240 MB file) and qwen2.5:0.5b (~490 MB file). You likely won’t be satisfied interacting with these models, but they are good enough to show that your solutions are working. Furthermore, you will need to run two such instances in the last third of the term, so you may want to team up with a classmate then if not now. Incidentally, the reference harnessd running on mada.eecs.umich.edu connects to an oMLX LLM runner hosting models gemma3:270m, gemma4:e4b, gemma4:12b, qwen2.5:0.5b, qwen3.5:4b, and qwen3.5:9b.

You can download the llama-server LLM runner we have pre-built for Ubuntu hosts following the instructions below:

server$ cd ~/agentic 
server$ wget http://web.eecs.umich.edu/~sugih/courses/reactive/llama-server-bundle.tgz
server$ tar xzf llama-server-bundle.tgz
server$ cd llama-server 
server$ ls                  # and you should see:
README.txt   args   bin/   llama-server.service   models/   models.ini

Our llama-server bundle comes with four models in the models subfolder that are suitable to run on GCP/AWS free-tier micro instances: gemma3:270m (for quick testing), qwen2.5:0.5b (for tool use in latter tutorials and projects), nomic1.5 (for quick testing of embedding generation), and egemma:300m (for embedding generation in latter tutorials and projects). The file README.txt tells you more about the content of this bundle.

You run the llama-server on the CLI by:

server$ xargs -P 1 -a args bin/llama-server-static
# Hit ^C to end process

To run your llama-server everytime your instance reboots:

server$ sudo cp llama-server.service /etc/systemd/system
server$ systemctl daemon-reload
server$ systemctl start llama-server.service