Colang v2 Interaction Loop with Query Rewriting #866

niels-garve · 2024-11-20T13:48:03Z

niels-garve
Nov 20, 2024

Hi all,

I am creating an AI solution based on the interaction loop of Colang v2. To improve the intent detection for user messages like confirmations ("yes", "no", etc.) I'd like to add Query Rewriting; basically an LLM call prompting to rephrase the user message with context from the conversation.

Example

User: Show me some bikes
Bot: Sure! Are you looking for some mountain bikes
User: yes

Now, query rewriting would transform the "yes" into "Show me some mountain bikes".

I don't find a solution; I tried with a custom action, but from where do I get the conversation. I could use the default prompt for intent detection, but I also would like to reduce cognitive load on LLM calls. Intent detection is one thing, rephrasing another.

Here's what I have so far:

main.co

import core
import llm

flow main
    activate llm continuation

    global $last_user_message
    while True
        await user said something as $something
        await RewriteQueryAction(user_message=$something.transcript)
        bot say "{$last_user_message}"
        loop

flow loop
    when user asked about bikes
        answering bike questions
    or when unhandled user intent # For any user utterance that does not match
        bot say "My apologies, I can't answer your question"

flow user asked about bikes
    user said "Show me bikes"
        or user said "I want to see bikes"
        or user said "Do you have any bicycles?"
        or user said "Can you show me some bikes?"
        or user said "I'm looking for bikes"
        or user said "Bicycles available?"
        or user said "What bicycles do you have?"
        or user said "Can you display bikes for me?"
        or user said "I'd like to see your bikes"
        or user said "Do you sell bicycles?"
        or user said "Bike options?"
        or user said "What kinds of bikes do you have?"
        or user said "Show me your bicyles collection"
        or user said "Can I browse your bikes?"
        or user said "What bikes do you offer?"
        or user said "I'm interested in buying a bike"
        or user said "Any bicycles for sale?"
        or user said "What's in your bicycle inventory?"
        or user said "Do you have a bike catalog?"
        or user said "I'd like to look at bikes"

flow answering bike questions
    """You are an assistant that should talk to the user about bikes.
    Politely decline to talk about anything else.

    Last user question is: "{{ last_user_message }}"
    Generate the output in the following format:

    bot say '''<<the response>>'''
    """
    ...

config.py

@action()
async def RewriteQueryAction(user_message: str, context: dict, llm: BaseLLM) -> ActionResult:
    context_updates = {}

    # FIXME: no LLM prompt yet, just a static rewrite
    rewrote_query = "Test"
    context_updates["last_user_message"] = rewrote_query

    return ActionResult(context_updates=context_updates)

def init(app: LLMRails):
    app.register_action(RewriteQueryAction, "RewriteQueryAction")

I use a default config.yml

I'd appreciate your help!

Thanks and cheers,
Niels

niels-garve · 2024-11-20T19:06:50Z

niels-garve
Nov 20, 2024
Author

Update: I'm currently trying input rails for user message rephrasing, with no luck. It feels like the correct place to do that, though:

import core
import llm
import guardrails

flow input rails $input_text
    rephrase user message

flow rephrase user message
    global $user_message
    $user_message = await RephraseUserMessageAction()

flow main
    activate llm continuation

    while True
        when user asked about bikes
            answering bike questions
        or when unhandled user intent # For any user utterance that does not match
            bot say "My apologies, I can't answer your question"

flow user asked about bikes
    user said "Show me bikes"
        or user said "I want to see bikes"
        or user said "Do you have any bicycles?"
        or user said "Can you show me some bikes?"
        or user said "I'm looking for bikes"
        or user said "Bicycles available?"
        or user said "What bicycles do you have?"
        or user said "Can you display bikes for me?"
        or user said "I'd like to see your bikes"
        or user said "Do you sell bicycles?"
        or user said "Bike options?"
        or user said "What kinds of bikes do you have?"
        or user said "Show me your bicyles collection"
        or user said "Can I browse your bikes?"
        or user said "What bikes do you offer?"
        or user said "I'm interested in buying a bike"
        or user said "Any bicycles for sale?"
        or user said "What's in your bicycle inventory?"
        or user said "Do you have a bike catalog?"
        or user said "I'd like to look at bikes"

flow answering bike questions
    """You are an assistant that should talk to the user about bikes.
    Politely decline to talk about anything else.

    Last user question is: "{{ last_user_message }}"
    Generate the output in the following format:

    bot say '''<<the response>>'''
    """
    ...

0 replies

niels-garve · 2024-11-22T13:40:30Z

niels-garve
Nov 22, 2024
Author

Opened a PR: #869

4 replies

schuellc-nvidia Nov 25, 2024
Collaborator

Hi @niels-garve

From what I understand, you are trying to improve the intent detection by rephrasing the user input. What LLM model do you currently use? With the newer models like gpt-4-turbo context sensitive user intent detection already works out of the box. Just add another user intent flow about mountain bikes and it will correctly detect it based on the conversation context. Weaker models might have issues sometimes.

flow main
    activate llm continuation

    while True
        when user asked about bikes
            answering bike questions
        or when user asked about mountain bikes
            bot say "Sure, let me show you more about our mountain bikes!"
        or when unhandled user intent # For any user utterance that does not match
            bot say "My apologies, I can't answer your question"

flow user asked about mountain bikes
    """User asked anything about mountain bikes."""
    user said "Tell me more about mountain bikes"

flow user asked about bikes
    """User asked anything about bikes."""
    user said "I want to see bikes"

Also note that you don't need all the examples but you could use a descriptive approach to make the user intent definition more convenient.

niels-garve Nov 28, 2024
Author

Thanks, @schuellc-nvidia , for your answer!

Oh wow, I didn't know that. So, does this mean that by adding a description to a user intent flow, an LLM is called with the description as a prompt? If so, I'm definitely gonna try that!

I'm currently using GPT 4o mini in my AI solution and still see potential in using query rewriting. As far as I understood the Colang library correctly, the following (super abstracted) images show the current and suggested flow of my pull request:

So, for example, if my user enters a phone number and the flow engine detects this as user said something unexpected, generate_user_intent is called with the transcript (the phone number) and not with the user message, where the transcript has been transformed to not contain any phone numbers. Am I getting the image right here?

Thanks,
Niels

schuellc-nvidia Mar 18, 2025
Collaborator

Hi Niels, yes this is correct. The input rails currently cannot modify the user message unfortunately! It can only decide to block (abort) or not block the user request. Sorry for the late response!

niels-garve Jun 6, 2025
Author

Thanks for clarifying @schuellc-nvidia!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Colang v2 Interaction Loop with Query Rewriting #866

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh! 8000

{{title}}

Uh oh!

Select a reply

Uh oh!

Colang v2 Interaction Loop with Query Rewriting #866

Uh oh!

Uh oh!

niels-garve Nov 20, 2024

Replies: 2 comments · 4 replies

Uh oh!

Uh oh!

niels-garve Nov 20, 2024 Author

Uh oh!

niels-garve Nov 22, 2024 Author

Uh oh!

schuellc-nvidia Nov 25, 2024 Collaborator

Uh oh!

niels-garve Nov 28, 2024 Author

Uh oh!

schuellc-nvidia Mar 18, 2025 Collaborator

Uh oh! 8000

niels-garve Jun 6, 2025 Author

niels-garve
Nov 20, 2024

Replies: 2 comments 4 replies

niels-garve
Nov 20, 2024
Author

niels-garve
Nov 22, 2024
Author

schuellc-nvidia Nov 25, 2024
Collaborator

niels-garve Nov 28, 2024
Author

schuellc-nvidia Mar 18, 2025
Collaborator

niels-garve Jun 6, 2025
Author