Cover Page

Back-end Page

Rust with axum

Install Rust

ssh to your server and install Rust:

server$ sudo apt install gcc          # cargo depends on gcc's linker
server$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
server$ hash -r
server$ rustup update

If you see:

rustup: Command not found.

and you need help updating your PATH shell environment variable, please see a teaching staff.

The command rustup update is also how you can subsequently update your installation of Rust to a new version.

Cargo.toml: add dependencies

First create and change into a directory where you want to keep your chatterd package:

server$ cd ~/reactive
server$ cargo new chatterd
# output:
     Created binary (application) `chatterd` package

This will create the ~/reactive/chatterd/ directory for you. Change to this directory and edit the file Cargo.toml to list all the 3rd-party libraries (crates in Rust-speak) we will be using.

server$ cd chatterd
server$ vi Cargo.toml

In Cargo.toml, add the following below the [dependencies] tag:

axum = { version = "0.8.7", features = ["macros"] }
axum-server = { version = "0.7.3", features = ["tls-rustls"] }
reqwest = { version = "0.12.24", features = ["json"] }
serde_json = "1.0.145"
tokio = { version = "1.48.0", features = ["full"] }
tower-http = { version = "0.6.7", features = ["normalize-path", "trace"] }
tower-layer = "0.3.3"
tracing = "0.1.43"
tracing-subscriber = { version = "0.3.22", features = ["env-filter"] }

chatterd package

In ~/reactive/chatterd/src/ the file main.rs has also been created for you. Edit the file:

server$ vi src/main.rs

and replace the existing lines in main.rs with the server and URL routing code, starting with the following use lines::

#![allow(non_snake_case)]
use axum::{
    extract::{ FromRef, Request },
    routing::{ get, post },
    Router, ServiceExt,
};
use axum_server::tls_rustls::RustlsConfig;
use reqwest::Client;
use std::{ net::SocketAddr, process };
use tower_http::{
    normalize_path::NormalizePathLayer,
    trace::{ DefaultMakeSpan, DefaultOnFailure, TraceLayer },
};
use tower_layer::Layer;
use tracing::Level;

mod handlers;

After listing all our imports, we export our module handlers, which we will define later.

We create a struct to hold state(s) that we want to pass to each API handler. For llmprompt, the only state we pass to its handler is an instantiation of the reqwest client, with which we will initiate a connection to the Ollama server:

#[derive(FromRef, Clone)]
pub struct AppState {
    client: Client,
}

The FromRef macro creates a “getter” for each of the structure’s property that allows the property to be passed as axum’s State.

Create the following main() function and enable logging (tracing):

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_level(false)
        .with_target(false)
        .compact()
        .init();

And instantiate an AppState that we will pass to each API handler:

    // Create HTTP client for proxying
    let appState = AppState {
        client: Client::new(),
    };

Continuing inside the main() function, we next create a routing table, in the form of a Router structure, to hold the URL routing information needed by axum_server. We define the routes to serve llmprompt’s two APIs: HTTP GET request with URL endpoint ‘/’ and HTTP POST request with URL endpoint /llmprompt. We route the first endpoint to the top() function and the second endpoint to the llmprompt() function. With each route, we specify which HTTP method is allowed for the URL endpoint by calling get() or post(), according to whether the endpoint accepts an HTTP GET or POST request respectively. We enable the tracing middleware and pass the appState instantiated above to each API handler:

    let router = Router::new()
        .route("/", get(handlers::top))
        .route("/llmprompt", post(handlers::llmprompt))
        .layer(
            // must be after all handlers to be traced
            TraceLayer::new_for_http()
                .make_span_with(DefaultMakeSpan::new().level(Level::INFO))
                .on_failure(DefaultOnFailure::new().level(Level::INFO)),
        )
        .with_state(appState);

The functions top() and llmprompt() will be implemented in handlers.rs later.

For now, staying in main.rs, in the main() function, set up the axum_server:

    // certificate and private key used with HTTPS
    let certkey = RustlsConfig::from_pem_file(
        "/home/ubuntu/reactive/chatterd.crt",
        "/home/ubuntu/reactive/chatterd.key",
    )
    .await
    .map_err(|err| { eprintln!("{:?}", err); process::exit(1) })
    .unwrap();

    // bind HTTPS server to wildcard IP address and default port number:
    let addr = SocketAddr::from(([0, 0, 0, 0], 443));
    tracing::info!("chatterd on https://{}", addr);
    // run the HTTPS server
    axum_server::bind_rustls(addr, certkey)
        .serve(
            ServiceExt::<Request>::into_make_service_with_connect_info::<SocketAddr>(
                NormalizePathLayer::trim_trailing_slash().layer(router),
            ),
        )
        .await
        .map_err(|err| { eprintln!("{:?}", err); process::exit(1) })
        .unwrap();
}

We’re done with main.rs. Save and exit the file.

handlers module

We implement the URL path API handlers in src/handlers.rs:

server$ vi src/handlers.rs

Start the file with the following use imports:

#![allow(non_snake_case)]
use axum::{
    extract::{ ConnectInfo, Json, State },
    http::{ Response, StatusCode },
};
use serde_json::{ json, Value };
use std::{ net::SocketAddr, };
use crate::AppState;

We add a couple of logging functions to print to console results of handling each HTTP request and, in the case of error, return a tuple of HTTP status code and error message to the client:

fn logOk(clientIP: &SocketAddr) {
    tracing::info!("{:?} | {:?} |", StatusCode::OK, clientIP);
}

fn logServerErr(clientIP: &SocketAddr, errmsg: String) -> (StatusCode, String) {
    tracing::info!("{:?} | {:?} |", StatusCode::INTERNAL_SERVER_ERROR, clientIP);
    (StatusCode::INTERNAL_SERVER_ERROR, errmsg)
}

The top() handler for the server’s root ‘/’ API simply returns a JSON containing the string “EECS Reactive chatterd” and, by default, HTTP status code 200:

pub async fn top(
    ConnectInfo(clientIP): ConnectInfo<SocketAddr>,
) -> Result<Json<Value>, (StatusCode, String)> {
    logOk(&clientIP);
    Ok(Json(json!("EECS Reactive chatterd")))
}

We next set the Ollama base URL and specify the handler llmprompt(), which simply forwards user prompt from the client to Ollama’s generate API using reqwest::CLient and passes through Ollama’s NDJSON reply stream to the client:

pub const OLLAMA_BASE_URL: &str = "http://localhost:11434/api";

pub async fn llmprompt(
    State(appState): State<AppState>,
    ConnectInfo(clientIP): ConnectInfo<SocketAddr>,
    Json(body): Json<Value>,
) -> Result<Response<reqwest::Body>, (StatusCode, String)> {
    logOk(&clientIP);
    Ok(appState.client
        .post(format!("{OLLAMA_BASE_URL}/generate"))
        .json(&body)
        .send()
        .await
        .map_err(|err| logServerErr(&clientIP, err.to_string()))?
        .into())
}

We’re done with handlers.rs. Save and exit the file.

Build and test run

To build your server:

server$ cargo build --release
server$ ln -s target/release/chatterd chatterd

The first time around cargo build runs, it will take some time to download and build all the 3rd-party crates. Be patient.

Build release version?

We would normally build for development without the --release flag, but due to the limited disk space on AWS virtual hosts, cargo build for debug version often runs out of space. The release version at least doesn’t keep debug symbols around.

Linking error with cargo build?

When running cargo build --release, if you see:

  error: linking with cc failed: exit status: 1
  note: collect2: fatal error: ld terminated with signal 9 [Killed]

below a long list of object files, try running cargo build --release again. It usually works the second time around, when it will have less remaining linking to do. If the error persisted, please talk to the teaching staff.

:point_right:Rust is a compiled language, like C/C++ and unlike Python, which is an interpreted language. This means you must run cargo build each and every time you made changes to your code, for the changes to show up in your executable.

To run your server:

server$ sudo ./chatterd
# Hit ^C to end the test

You can test your implementation following the instructions in the Testing llmPrompt APIs section.

References

Prepared by Chenglin Li, Xin Jie ‘Joyce’ Liu, Sugih Jamin Last updated January 7th, 2026