Cover Page

Backend Page

Rust with axum

Install Rust

ssh to your server and install Rust:

server$ sudo apt install gcc          # cargo depends on gcc's linker
server$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
server$ rustup update

If you see:

rustup: Command not found.

try to logout from your server and ssh back to it again. If the problem persisted and you need help updating your PATH shell environment variable, please see a teaching staff.

The command rustup update is also how you can subsequently update your installation of Rust to a new version.

Cargo.toml: add dependencies

First create and change into a directory where you want to keep your chatterd package:

server$ cd ~/reactive
server$ cargo new chatterd
# output:
     Created binary (application) `chatterd` package

This will create the ~/reactive/chatterd/ directory for you. Change to this directory and edit the file Cargo.toml to list all the 3rd-party libraries (crates in Rust-speak) we will be using.

server$ cd chatterd
server$ vi Cargo.toml

In Cargo.toml, add the following below the [dependencies] tag:

axum = { version = "0.8.4", features = ["macros"] }
axum-server = { version = "0.7.2", features = ["tls-rustls"] }
reqwest = { version = "0.12.20", features = ["json"] }
serde_json = "1.0.140"
tokio = { version = "1.45.1", features = ["full"] }
tower-http = { version = "0.6.2", features = ["normalize-path", "trace"] }
tower-layer = "0.3.3"
tracing = "0.1.41"
tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }

chatterd package

In ~/reactive/chatterd/src/ the file main.rs has also been created for you. Edit the file:

server$ vi src/main.rs

and replace the existing lines in main.rs with the server and URL routing code, starting with the following use lines::

#![allow(non_snake_case)]
use axum::{
    extract::{ FromRef, Request },
    routing::post,
    Router, ServiceExt,
};
use axum_server::tls_rustls::RustlsConfig;
use reqwest::Client;
use std::{ net::SocketAddr, process };
use tower_http::{
    normalize_path::NormalizePathLayer,
    trace::{ DefaultMakeSpan, DefaultOnFailure, TraceLayer },
};
use tower_layer::Layer;
use tracing::Level;

mod handlers;

After listing all our imports, we export our module handlers, which we will define later.

We create a struct to hold state(s) that we want to pass to each API handler. For llmprompt, the only state we pass to its handler is an instantiation of the reqwest client, with which we will initiate a connection to the Ollama server:

#[derive(FromRef, Clone)]
pub struct AppState {
    client: Client,
}

The FromRef macro creates a “getter” for each of the struct’s property that allows the property to be passed as axum’s State.

Create the following main() function and enable logging (tracing):

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt()
        .with_level(false)
        .with_target(false)
        .compact()
        .init();

And instantiate an AppState that we will pass to each API handler:

    // Create HTTP client for proxying
    let appState = AppState {
        client: Client::new(),
    };

Continuing inside the main() function, we next create a Router struct to hold the URL routing information needed by axum_server. We define a route to serve llmprompt’s single API: HTTP POST request with URL endpoint llmprompt. We route this endpoint to the llmprompt() function. With each route, we implicit specify which HTTP method is allowed for the URL endpoint by calling get(), post(), or other specific MethodRouter. We enable the tracing middleware and pass the appState instantiated above to each API handler:

    let router = Router::new()
        .route("/llmprompt", post(handlers::llmprompt))
        .layer(
            // must be after all handlers to be traced
            TraceLayer::new_for_http()
                .make_span_with(DefaultMakeSpan::new().level(Level::INFO))
                .on_failure(DefaultOnFailure::new().level(Level::INFO)),
        )
        .with_state(appState);

The functions llmprompt() will be implemented in handlers.rs later.

For now, staying in main.rs, in our main() function, we set up the axum_server:

    // certificate and private key used with HTTPS
    let certkey = RustlsConfig::from_pem_file(
        "/home/ubuntu/reactive/chatterd.crt",
        "/home/ubuntu/reactive/chatterd.key",
    )
    .await
    .map_err(|err| { eprintln!("{:?}", err); process::exit(1) })
    .unwrap();

    // bind HTTPS server to wildcard IP address and default port number:
    let addr = SocketAddr::from(([0, 0, 0, 0], 443));
    tracing::info!("chatterd on https://{}", addr);
    // run the HTTPS server
    axum_server::bind_rustls(addr, certkey)
        .serve(
            ServiceExt::<Request>::into_make_service_with_connect_info::<SocketAddr>(
                NormalizePathLayer::trim_trailing_slash().layer(router),
            ),
        )
        .await
        .map_err(|err| { eprintln!("{:?}", err); process::exit(1) })
        .unwrap();
}

We’re done with main.rs. Save and exit the file.

handlers module

We implement the URL path API handler module in src/handlers.rs:

server$ vi src/handlers.rs

Start the file with the following use imports:

#![allow(non_snake_case)]
use axum::{
    extract::{ ConnectInfo, Json, State },
    http::{ Response, StatusCode },
};
use serde_json::Value;
use std::{ net::SocketAddr, };

use crate::AppState;

We add a couple of logging functions to print to console results of handling each HTTP request and, in case of error, return a tuple of HTTP status code and error message to the client:

fn logOk(clientIP: SocketAddr) {
    tracing::info!("{:?} | {:?} |", StatusCode::OK, clientIP);
}

fn logServerErr(clientIP: SocketAddr, errmsg: String) -> (StatusCode, String) {
    tracing::info!("{:?} | {:?} |", StatusCode::INTERNAL_SERVER_ERROR, clientIP);
    (StatusCode::INTERNAL_SERVER_ERROR, errmsg)
}

We next set the Ollama base URL string and specify the handler llmprompt(), which simply forwards user prompt from the client to Ollama’s generate API using reqwest::CLient and passes through Ollama’s reply NDJSON stream to the client:

pub const OLLAMA_BASE_URL: &str = "http://localhost:11434/api";

pub async fn llmprompt(
    State(appState): State<AppState>,
    ConnectInfo(clientIP): ConnectInfo<SocketAddr>,
    Json(body): Json<Value>,
) -> Result<Response<reqwest::Body>, (StatusCode, String)> {
    logOk(clientIP);
    Ok(appState.client
        .post(format!("{OLLAMA_BASE_URL}/generate"))
        .json(&body)
        .send()
        .await
        .map_err(|err| logServerErr(clientIP, err.to_string()))?
        .into())
}

We’re done with handlers.rs. Save and exit the file.

Build and test run

To build your server:

server$ cargo build --release
server$ ln -s target/release/chatterd chatterd

The first time around, it will take some time to download and build all the 3rd-party crates. Be patient.

Build release version?

We would normally build for development without the --release flag, but due to the limited disk space on AWS virtual hosts, cargo build for debug version often runs out of space. The release version at least doesn’t keep debug symbols around.

Linking error with cargo build?

When running cargo build --release, if you see:

  error: linking with cc failed: exit status: 1
  note: collect2: fatal error: ld terminated with signal 9 [Killed]

below a long list of object files, try running cargo build --release again. It usually works the second time around, when it will have less remaining linking to do. If the error persisted, please talk to the teaching staff.

:point_right:Rust is a compiled language, like C/C++ and unlike Python, which is an interpreted language. This means you must run cargo build each and every time you made changes to your code, for the changes to show up in your executable.

To run your server:

server$ sudo ./chatterd
# Hit ^C to end the test

You can test your implementation following the instructions in the Testing Chatter APIs section.

References

Prepared by Chenglin Li, Xin Jie ‘Joyce’ Liu, Sugih Jamin Last updated August 24th, 2025