Cover Page

Backend Page

Python with Starlette

Install uv

First confirm that you have Python version 3.12 or 3.13 installed on your system:

server$ python3 --version
# output:
Python 3.12 # or 3.13

At the time of writing, the http.client, psycopg, and maybe other packages are not compatible with Python 3.14.

server$ sudo add-apt-repository ppa:deadsnakes/ppa
server$ sudo apt update
server$ sudo apt install python3.13 # for example

server$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3<TAB> 0 # TAB for autocompletion
server$ sudo update-alternatives --config python
# then type a selection number corresponding to the Python of choice.

We’ll be using uv to manage Python package and project. It is reputably faster than other Python package management tools you may have used, such as pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and others.

server$ curl -LsSf https://astral.sh/uv/install.sh | sh

Confirm that uv is installed:

server$ uv --version

Create project and install dependencies

Create and change into a directory where you want to keep your chatterd project and initialize the project:

server$ mkdir ~/reactive/chatterd
server$ cd ~/reactive/chatterd
server$ uv init --bare

Install the following packages, including the PostgreSQL adaptor:

server$ uv add granian http.client 'httpx[http2]' starlette

uv sync updates all packages to the latest version.

chatterd app

Create and open a file called main.py to put the server and URL routing code.

server$ vi main.py

Edit the file to add the following import lines:

import handlers
from starlette.applications import Starlette
from starlette.routing import Route

We declare the routes global array to hold the URL routing information needed by Starlette server and define a route to serve llmprompt’s single API: HTTP POST request with URL endpoint llmprompt. We route this endpoint to the llmprompt() function. With each route, we also specify which HTTP method is allowed for that URL endpoint:

# must include the trailing '/'
routes = [
    Route('/llmprompt', handlers.llmprompt, methods=['POST']),
]

Finally, construct the web server, server, providing it with the routes array:

# must come after route definitions
server = Starlette(routes=routes)

We’re done with main.py. Save and exit the file.

handlers.py

We implement the URL path API handler in handlers.py:

server$ vi handlers.py

Start the file with the following imports:

from http.client import HTTPException
import httpx
from starlette.background import BackgroundTask
from starlette.exceptions import HTTPException
from starlette.responses import JSONResponse, Response, StreamingResponse

We next set the Ollama base URL string and specify the handler llmprompt(), which forwards user prompt from the client to Ollama’s generate API using httpx.AsyncClient(), which we instantiate as a global variable, and returns Ollama’s reply to the client as a NDJSON stream.

OLLAMA_BASE_URL = "http://localhost:11434/api"
asyncClient = httpx.AsyncClient(timeout=None, http2=True)

async def llmprompt(request):
    # https://www.python-httpx.org/async/
    response = await asyncClient.send(
        asyncClient.build_request(
            method = request.method,
            url = f"{OLLAMA_BASE_URL}/generate",
            data=await request.body()
        ), stream = True)

    if response.status_code != 200:
        return Response(headers=response.headers, content=await response.aread())

    return StreamingResponse(response.aiter_raw(),
                             media_type="application/x-ndjson",
                             background=BackgroundTask(response.aclose))

We’re done with handlers.py. Save and exit the file.

Test run

To test run your server, launch it from the command line:

server$ sudo su
# You are now root, note the command-line prompt changed from '$' or '%' to '#'.
# You can do a lot of harm with all of root's privileges, so be very careful what you do.
server# source .venv/bin/activate
(chatterd) ubuntu@server:/home/ubuntu/reactive/chatterd# granian --host 0.0.0.0 --port 443 --interface asgi --ssl-certificate /home/ubuntu/reactive/chatterd.crt --ssl-keyfile /home/ubuntu/reactive/chatterd.key --access-log --workers-kill-timeout 1 main:server
# Hit ^C to end the test
(chatterd) ubuntu@server:/home/ubuntu/reactive/chatterd# exit
# So that you're no longer root.
server$

As of time of writing, uvicorn supports only HTTP/1.1. To support HTTP/2, we run Starlette on Granian, thanks to ASGI.

You can test your implementation following the instructions in the Testing Chatter APIs section.

References

• Python Modules can import each other cyclically
• Granian
• Starlette
• Exceptions
• HTTPX Async Support