Python with Starlette

Cover Page

Back-end Page

Install uv

First confirm that you have Python version 3.12 or 3.13 installed on your system:

server$ python3 --version
# output:
Python 3.12 # or 3.13

At the time of writing, the http.client, psycopg, and maybe other packages are incompatible with Python 3.14.

server$ sudo add-apt-repository ppa:deadsnakes/ppa
server$ sudo apt update
server$ sudo apt install python3.13 # for example

server$ sudo update-alternatives --install /usr/bin/python python /usr/bin/python3<TAB> 0 # TAB for autocompletion
server$ sudo update-alternatives --config python
# then type a selection number corresponding to the Python of choice.

We’ll be using uv to manage Python package and project. It is reputably faster than other Python package management tools you may have used, such as pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and others.

server$ curl -LsSf https://astral.sh/uv/install.sh | sh

Confirm that uv is installed:

server$ uv --version

Create project and install dependencies

Create and change into a directory where you want to keep your chatterd project and initialize the project:

server$ mkdir ~/reactive/chatterd
server$ cd ~/reactive/chatterd
server$ uv init --bare

Install the following packages:

server$ uv add granian http.client 'httpx[http2]' starlette pydantic-core 'psycopg[binary,pool]'

  server$ uv self update
  server$ uv sync --upgrade

chatterd app

Create and open a file called main.py to put the server and URL routing code.

server$ vi main.py

Edit the file to add the following import lines:

import handlers
from starlette.applications import Starlette
from starlette.routing import Route

We next create a routing table to hold the URL routing information needed by Starlette and assign it to a global variable routes. We define the routes to serve llmprompt’s two APIs: HTTP GET request with URL endpoint ‘/’ and HTTP POST request with URL endpoint /llmprompt. We route the first endpoint to the top() function and the second endpoint to the llmprompt() function. With each route, we specify which HTTP method is allowed for the URL endpoint, according to whether the endpoint accepts an HTTP GET or POST request:

routes = [
    Route('/', handlers.top, methods=['GET']),    
    Route('/llmprompt', handlers.llmprompt, methods=['POST']),
]

The functions top() and llmprompt() will be implemented in handlers.py later.

For now, staying in main.py, construct the web server, server, providing it the routes array:

# must come after route definitions
server = Starlette(routes=routes)

We’re done with main.py. Save and exit the file.

handlers.py

We implement the URL path API handlers in handlers.py:

server$ vi handlers.py

Start the file with the following imports:

from http.client import HTTPException
import httpx
from starlette.background import BackgroundTask
from starlette.exceptions import HTTPException
from starlette.responses import JSONResponse, Response, StreamingResponse

The top() handler for the server’s root ‘/’ API simply returns a JSON containing the string “EECS Reactive chatterd” and HTTP status code 200:

async def top(request):
    return JSONResponse('EECS Reactive chatterd', status_code=200)

We next set the Ollama base URL and specify the handler llmprompt(), which simply forwards user prompt from the client to Ollama’s generate API using httpx.AsyncClient(), which we instantiate as a global variable, and returns Ollama’s reply to the client as a NDJSON stream.

OLLAMA_BASE_URL = "http://localhost:11434/api"
asyncClient = httpx.AsyncClient(timeout=None, http2=True)

async def llmprompt(request):
    # Cannot use `async with`, instead we will close the response manually in
    # a background task after streaming is done (https://stackoverflow.com/a/73736138)
    response = await asyncClient.send(
        asyncClient.build_request(
            method = request.method,
            url = f"{OLLAMA_BASE_URL}/generate",
            data=await request.body()
        ), stream = True) #as response:
    
    if response.status_code != 200:
        return Response(headers=response.headers, content=await response.aread())
    
    return StreamingResponse(response.aiter_raw(),
                            media_type="application/x-ndjson",
                            background=BackgroundTask(response.aclose))

We’re done with handlers.py. Save and exit the file.

Test run

To test run your server, launch it from the command line:

server$ sudo su
# You are now root, note the command-line prompt changed from '$' to '#'.
# You can do a lot of harm with root privileges, so be very careful what you do here.
server# source .venv/bin/activate

When you run source .venv/bin/activate, your prompt should change to indicate that you are now operating within a Python virtual environment. It will look something like this: (chatterd) ubuntu@YOUR_SERVER_IP:/home/ubuntu/reactive/chatterd#.

(chatterd) ubuntu@server:/home/ubuntu/reactive/chatterd# granian --host 0.0.0.0 --port 443 --interface asgi --ssl-certificate /home/ubuntu/reactive/chatterd.crt --ssl-keyfile /home/ubuntu/reactive/chatterd.key --access-log --workers-kill-timeout 1 main:server
# Hit ^C to end the test
(chatterd) ubuntu@server:/home/ubuntu/reactive/chatterd# exit
# So that you're no longer root.
server$

As of time of writing, uvicorn supports only HTTP/1.1. To support HTTP/2, we run Starlette over Granian, thanks to ASGI.

You can test your implementation following the instructions in the Testing llmPrompt APIs section.

References

• uv: building Modern Python Projects
  • UV Cheatsheet for Python Developers
• Python Modules can import each other cyclically
• Granian
• Starlette
• Exceptions
• HTTPX Async Support