Project 3: llmAction Front End

Cover Page

DUE Mon, 12/08, 6 pm

Preparing your GitHub repo

If you have not completed the llmTools and Signin tutorials, please complete them first. We will base this project on the llmTools code base, but will be copying over the bulk of the Signin code also.

• On your laptop, navigate to YOUR*TUTORIALS*/
• Create a zip of your llmtools folder
• Rename your llmtools folder pa3
• Prepare your repo as you have done in previous tutorials and projects
• Push your local YOUR*TUTORIALS*/ repo to GitHub and make sure there’re no git issues:
  git push

  • Open GitHub Desktop and click on Current Repository on the top left of the interface
  • Click on your reactive GitHub repo
  • Add Summary to your changes and click Commit to main
  • If you have pushed other changes to your Git repo, click Pull Origin to synch up the clone on your laptop
  • Finally click on Push Origin to push changes to GitHub

Go to the GitHub website to confirm that your folders follow this structure outline:

  reactive
    |-- chatterd
    |-- chatterd.crt
    |-- llmtools
    |-- pa3
        # under here should be your swiftUIChatter or composeChatter folder
    # and other files or folders

Obtaining chatterID

The main purpose of the get_auth tool is for the LLM to obtain an authorization token. We require LLMs to acquire an authorization token to make operative tool calls that can take action with real-world side-effects. The authorization token requirement forms our human-in-the-loop guardrail. In this project, the chatterID first used in the Signin tutorial is our authorization token. To create a chatterID requires user to sign in using Google Signin. To store/load the chatterID for reuse within its limited lifetime, we further require biometric authentication from the user. Both Google Signin and biometric check requirements mean that the authorization token can only be obtained from user-facing front-end devices.

The front-end UI/UX and the network streaming and tool call infrastrucure are those of llmTools. To these, we add code from the Signin tutorial to use Google Signin and biometric authentication.

Signin migration

Installing dependencies

• Follow the instructions in the Signin tutorial to install Google Signin SDK for iOS or for Android.

• Copy over the OAuth client ID you created in the Signin tutorial.
  • For Android: copy over the clientID string from /app/res/values/strings.xml on the Signin tutorial to the file of the same name in this project.
  • For iOS: copy over your CLIENT_ID to your Info property GIDClientID and your REVERSED_CLIENT_ID to your URL Types. You can review the instructions, especially steps 3-8 of the second set of steps in the linked section. You wil also need to provide justification for performing biometric check by following step 1 of the biometric check section.
• Copy over code from the Signin tutorial that handles signing in to Google and then using the idToken from Google Signin obtain a chatterID from our chatterd backend. Copy over also the code to save and load chatterID from local storage with biometric authentication.

UI adaptation

Next we need to make some minor adjustments to your llmTools UI to accommodate the sign-in process:

• Add observable variable showOk to your ChattViewModel and make the username variable in your ChattViewModel observable.
• From the Signin tutorial code, copy over the code to show alert when showOk is true. (You can search for all occurrences of showOk and copy over relevant code.)
• In ChattScrollView, when calling ChattView(), set isSender by comparing the username in a chatt against vm.model instead of against vm.username.

ChatterID access to ChattViewModel

To initiate GoogleSignin from the get_auth tool, we need to access ChattViewModel. ChattViewModel is associated with UI code running on the main thread, yet we now want to access it from a piece of networking code that is most likely running on a background thread. Since we need to access ChattViewModel from networking code only to obtain chatterID, we put network-level availability of ChattViewModel in the ChatterID singleton, instead of making it visible ChattStore-wide:

• add an optional/nullable vm property to ChatterID, intialized to nil/null
• at app launch, assign ChattViewModel to ChatterID’s new vm property and load chatterID from previous run of the app:
  • For Swift:
    • in swiftUIChatterApp.init(), assign the existing viewModel variable to ChatterID.shared.vm,
    • then load previously obtained chatterID from the key chain by launching ChatterID.shared.open() in a .task of ContentView, as we did in the Signin tutorial, without calling getChatts().
  • For Android, in MainActivity:
    • first, copy over some code from the MainActivity of Signin tutorial:
      • MainActivity must extend/inherit from FragmentActivity instead of ComponentActivity,
      • add viewModel property as ChattViewModel by viewModels() as we did in the Signin tutorial,
      • also as in the Signin tutorial, load previously obtained chatterID from SharedPreferences by launching keystore.open() within lifecycleScope,
    • finally, store the viewModel property in ChatterID.vm.

    All of the above before calling setContent {}.

Toolbox

To add get_auth as a tool, first add a global variable AUTH_TOOL, whose definition follows its counterpart LOC_TOOL from the llmTools tutorial.

We will add the getAuth() function in the next subsection. Assuming we already have getAuth() defined, add an entry to the TOOLBOX dictionary/map to register the get_auth tool, again along the line of the get_location entry for TOOLBOX in the llmTools tutorial.

getAuth() function

The main purpose of the getAuth() function is to return a valid authorization token to the LLM. A “valid” authorization token in this project is a chatterID that has not expired. The function getAuth() first checks whether there’s a valid chatterID present. If so, it simply returns the chatterID. Otherwise, it launches SigninView to let user sign in with Google Signin. Upon successful signin, SigninView calls addUser() to obtain a chatterID from the chatterd backend (see the Signin tutorial for further details).

To deliver the above, we need to tackle two issues:

1. How to launch SigninView on the main/UI thread from getAuth(), a piece of networking code most likelly running as a background task?
2. How would SigninView communicate completion of signing in and obtaining chatterID back to getAuth()?

Practically, the Google Signin SDK for iOS is a SwiftUI package. On the Android side, the Google Signin SDK requires access to the Activity context to present the Google Signin panel. Similarly, biometric authentication on both platforms require access to the UI front end. Unfortunately, a networking function that is not part of the UI framework cannot launch a View/composable. A View/composable further cannot be an asynchronous/suspending function. These two constraints together mean that getAuth() cannot launch SigninView as a suspending function and then await its completion and send the resulting chatterID back to the calling LLM.

The first constraint by itself also means that getAuth() cannot launch SigninView directly and pass it a completion function without going through the UI framework. In any case, our tool call infrastructure returns tool result to the backend immediately after the tool call returns. Thus getAuth() cannot pass along a completion function to SigninView and return to the tool-call infrastructure before completion of the sign-in process and the resolution of chatterID.

Neither of the two ways we are used to in initiating asynchronous execution and returning result to the caller is applicable here. Our solution depends on the use of a rendezvous channel and two variables we place in ChattViewModel: a. an observable boolean property getSignedin, initialized to false, and b. property signinCompletion to hold the completion function. This property is of asynchronous/suspending function type with no parameter and returns Void/Unit. It does not need to be observable.

In Kotlin, we use the Channel API from its standard library to implement the rendezvous channel. In Swift, we use the AsyncChannel API, for which you need to add the AsyncAlgorithms package as a dependency to your project and add import AsyncAlgorithms to the top of your Toolbox.swift.

The getAuth() function below first creates a rendezvous channel. Then it grabs ChattViewModel from ChatterID and populates vm.signinCompletion with a λ-expression that closes the channel. With signinCompletion() prepared, it triggers the launch of SigninView by setting vm.getSignedin to true, upon which the reactive UI framework launches SigninView (see below). Meanwhile, back in getAuth(), we wait for the rendezvous channel to close and, depending on the outcome of SigninView, return either a valid chatterID or an error message informing the LLM that authorization is denied.

  func getAuth(_ argv: [String]) async -> String? {
      let rendezvous = AsyncChannel<Void>()
      
      repeat {
          if let chatterID = ChatterID.shared.id {
              return "Authorization token is: \(chatterID)"
          }
          
          if let vm = ChatterID.shared.vm {
              vm.signinCompletion = { () async -> Void in rendezvous.finish() }
              vm.getSignedin = true
              for await _ in rendezvous {}
          }
          if ChatterID.shared.id == nil { // failed to sign in, or vm is nil
              return "401 Unauthorized. Inform user that authentication token has expired and end session."
          }
      } while true
  }

  suspend fun getAuth(argv: List<String>): String? { //= "Authentication token is: chatterID"
      val rendezvous = Channel<Unit>()

      while (true) {
          if (id != null) {
              return "Authorization token is: $id"
          }

          ChatterID.vm?.let { vm ->
              vm.signinCompletion = suspend { rendezvous.close() }
              vm.getSignedin.value = true
              for (elt in rendezvous) {}
          }
          if (id == null) { // failed to sign in, or vm is null
              return "401 Unauthorized. Inform user that authentication token has expired and end session."
          }
      }
  }

To set up the UI framework to launch SigninView when getSignedin in ChattViewModel is true:

• on iOS:
  • As in the Signin tutorial, we launch SigninView in a .sheet in ContentView.
  • The SigninView from the Signin tutorial is unmodified.
  • When launching SigninView, we pass vm.signinCompletion as its completion: argument.

• on Android: the Signin tutorial for Android does not create a SigninView. Instead, the CredentialManager.getCredential() function called by the signin() function handles the UI for Google Signin. In the tutorial, we called signin() from a composable and passed it the Activity context of the composable. The signin() function, in turn, passes this Activity context to the CredentialManager for its use.

  Our getAuth() has no access to an Activity context. Therefore we now create a SigninView composable that can obtain a LocalContext.current available to any composable and use it to call the signin() function. The signin() function is the same one from the Signin tutorial, unmodified.

  @Composable
  fun SigninView() {
      val vm: ChattViewModel = viewModel()
      val context = LocalContext.current
  
      vm.viewModelScope.launch (Dispatchers.Default) {
          vm.getSignedin.value = false
          signin(context, vm)
          vm.signinCompletion?.invoke()
      }
  }
  

  We then add code to MainView to launch SigninView if vm.getSignedin is true.

When SigninView calls signinCompletion(), it closes the rendezvous channel that getAuth() is awaiting on, signalling getAuth() that sign in is complete and a new chatterID is obtained or has been denied.

Run and test to verify and debug

Please see the End-to-end testing section of the spec to test your frontend implementation.

Once you finished testing, change your serverUrl back to YOUR_SERVER_IP so that we know what your server IP is. You will not get full credit if your front end is not set up to work with your backend!

Front-end submission guidelines

Unlike in previous labs, there is one CRUCIAL extra step to do before you push your lab to GitHub:

• For iOS: ensure that the Bundle identifier under the Signing & Capabilities tab of your Project pane is the one you used to create your OAuth client ID.
• For Android:
  • Copy debug.keystore in (~/.android/ on Mac Terminal and Windows PowerShell) to your pa3 lab folder.
  • Put a copy of the SHA1 certificate (in the format of xx:xx:xx:...) you used to obtain your Client ID in the README.md file at your repo’s top level folder.

Without these we won’t be able to run your app.

Be sure you have submitted your modified backend in addition to submitting your updated frontend. As usual, we will only grade files committed to the main branch. If you use multiple branches, please merge them all to the main branch for submission.

Push your front-end code to the same GitHub repo you’ve submitted your back-end code:

• Open GitHub Desktop and click on Current Repository on the top left of the interface
• Click on the GitHub repo you created at the start of this tutorial
• Add Summary to your changes and click Commit to main at the bottom of the left pane
• If you have pushed code to your repo, click Pull Origin to synch up the repo on your laptop
• Finally click Push Origin to push all changes to GitHub

Go to the GitHub website to confirm that your front-end files have been uploaded to your GitHub repo under the folder pa3. Confirm that your repo has a folder structure outline similar to the following. If your folder structure is not as outlined, our script will not pick up your submission and, further, you may have problems getting started on latter tutorials. There could be other files or folders in your local folder not listed below, don’t delete them. As long as you have installed the course .gitignore as per the instructions in Preparing GitHub for Reactive Tutorials, only files needed for grading will be pushed to GitHub.

  reactive
    |-- chatterd
    |-- chatterd.crt
    |-- llmchat
    |-- pa2
        # under here should be your swiftUIChatter or composeChatter folder
    # and other files or folders