Customised prompt not present when prompt flow deployed

Question

Customised prompt not present when prompt flow deployed

Emmanuel Falola 0

When I deploy my prompt flow, the LLM doesn't answer the queries the same, its like the customised prompts are not transferred when deployed, when I create the deployment and test it in the endpoints section of Machine learning studio it works as expected but when I test the endpoint via Postman or via VScode by consuming the endpoint I get totally different answers to my queries.

romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-12-07T08:07:36.46+00:00

@Emmanuel Falola Did you try to enable developer mode on your browser and check if the request from the studio matches that of postman or VS code? I have used this before to confirm if I am calling the API with the correct details, like the headers, body, etc.

Emmanuel Falola 0

I have checked the API calls specifically the POST requests, I managed to find this json

{
  "flowId": "f2e93237-e80f-411c-b438-0ebb7ca08c05",
  "flow": {
    "nodeVariants": {
      "modify_query_with_history": {
        "defaultVariantId": "variant_0",
        "variants": {
          "variant_0": {
            "node": {
              "type": "llm",
              "tool": "modify_query_with_history.jinja2",
              "name": "modify_query_with_history",
              "inputs": {
                "deployment_name": "gpt-35-turbo-16k",
                "temperature": "1",
                "top_p": "1",
                "stop": "",
                "max_tokens": "0",
                "presence_penalty": "0",
                "frequency_penalty": "0",
                "logit_bias": "",
                "chat_history": "${inputs.chat_history}",
                "question": "${flow.question}"
              },
              "source": {
                "type": "code",
                "path": "modify_query_with_history.jinja2"
              },
              "api": "chat",
              "provider": "AzureOpenAI",
              "connection": "trialconnectiontest",
              "module": "promptflow.tools.aoai"
            }
          }
        }
      },
      "embed_the_question": {
        "defaultVariantId": "variant_0",
        "variants": {
          "variant_0": {
            "node": {
              "type": "python",
              "tool": "promptflow.tools.embedding.embedding",
              "name": "embed_the_question",
              "inputs": {
                "connection": "trialconnectiontest",
                "deployment_name": "text-embedding-ada-002",
                "input": "${modify_query_with_history.output}"
              },
              "source": {
                "type": "package",
                "tool": "promptflow.tools.embedding.embedding"
              },
              "reduce": false
            }
          }
        }
      },
      "search_question_from_indexed_docs": {
        "defaultVariantId": "variant_0",
        "variants": {
          "variant_0": {
            "node": {
              "type": "python",
              "tool": "promptflow_vectordb.tool.vector_index_lookup.VectorIndexLookup.search",
              "name": "search_question_from_indexed_docs",
              "inputs": {
                "path": "azureml://subscriptions/2ee1d5e1-8abf-4d8c-b61d-5eb397d9eb46/resourcegroups/trial/providers/Microsoft.MachineLearningServices/workspaces/Machinelearningairline/data/cognitivesearchlutontomanchesterfulldestinationdata/versions/1",
                "query": "${embed_the_question.output}",
                "top_k": "2"
              },
              "source": {
                "type": "package",
                "tool": "promptflow_vectordb.tool.vector_index_lookup.VectorIndexLookup.search"
              },
              "reduce": false
            }
          }
        }
      },
      "generate_prompt_context": {
        "defaultVariantId": "variant_0",
        "variants": {
          "variant_0": {
            "node": {
              "type": "python",
              "tool": "generate_prompt_context.py",
              "name": "generate_prompt_context",
              "inputs": {
                "search_result": "${search_question_from_indexed_docs.output}"
              },
              "source": {
                "type": "code",
                "path": "generate_prompt_context.py"
              },
              "reduce": false
            }
          }
        }
      },
      "Prompt_variants": {
        "defaultVariantId": "Variant_0",
        "variants": {
          "Variant_0": {
            "node": {
              "type": "prompt",
              "tool": "Prompt_variants__Variant_0.jinja2",
              "name": "Prompt_variants",
              "inputs": {
                "chat_history": "${flow.chat_history}",
                "contexts": "${generate_prompt_context.output}",
                "question": "${flow.question}"
              },
              "source": {
                "type": "code",
                "path": "Prompt_variants__Variant_0.jinja2"
              },
              "reduce": false
            }
          },
          "Variant_1": {
            "node": {
              "type": "prompt",
              "tool": "Prompt_variants__Variant_1.jinja2",
              "name": "Prompt_variants",
              "inputs": {
                "chat_history": "${flow.chat_history}",
                "contexts": "${generate_prompt_context.output}",
                "question": "${flow.question}"
              },
              "source": {
                "type": "code",
                "path": "Prompt_variants__Variant_1.jinja2"
              },
              "reduce": false
            }
          },
          "Variant_2": {
            "node": {
              "type": "prompt",
              "tool": "Prompt_variants__Variant_2.jinja2",
              "name": "Prompt_variants",
              "inputs": {
                "chat_history": "${flow.chat_history}",
                "contexts": "${generate_prompt_context.output}",
                "question": "${flow.question}"
              },
              "source": {
                "type": "code",
                "path": "Prompt_variants__Variant_2.jinja2"
              },
              "reduce": false
            }
          }
        }
      },
      "answer_the_question_with_context": {
        "defaultVariantId": "variant_0",
        "variants": {
          "variant_0": {
            "node": {
              "type": "llm",
              "tool": "answer_the_question_with_context.jinja2",
              "name": "answer_the_question_with_context",
              "inputs": {
                "deployment_name": "gpt-35-turbo-16k",
                "temperature": "0",
                "top_p": "1",
                "stop": "",
                "max_tokens": "1000",
                "presence_penalty": "0",
                "frequency_penalty": "0",
                "logit_bias": "",
                "prompt_text": "${Prompt_variants.output}"
              },
              "source": {
                "type": "code",
                "path": "answer_the_question_with_context.jinja2"
              },
              "api": "chat",
              "provider": "AzureOpenAI",
              "connection": "trialconnectiontest",
              "module": "promptflow.tools.aoai",
              "reduce": false
            }
          }
        }
      }
    },
    "flowGraphLayout": {
      "nodeLayouts": {
        "inputs": { "x": 67, "y": 62, "index": -1 },
        "outputs": { "x": 67, "y": 832, "index": -1 },
        "modify_query_with_history": {
          "x": 12,
          "y": 172,
          "height": 75,
          "index": 0
        },
        "embed_the_question": { "x": 12, "y": 282, "height": 75, "index": 1 },
        "search_question_from_indexed_docs": {
          "x": 12,
          "y": 392,
          "height": 75,
          "index": 2
        },
        "generate_prompt_context": {
          "x": 12,
          "y": 502,
          "height": 75,
          "index": 3
        },
        "Prompt_variants": { "x": 67, "y": 612, "height": 75, "index": 4 },
        "answer_the_question_with_context": {
          "x": 67,
          "y": 722,
          "height": 75,
          "index": 5
        }
      },
      "orientation": "Vertical"
    }
  },
  "flowRunId": "7507dd31-2b41-4d88-a748-fa5cbb86ae5e",
  "flowSubmitRunSettings": {
    "runtimeName": "MLEuropeRuntime",
    "runMode": "Flow",
    "batch_inputs": [
      {
        "chat_history": [
          {
            "inputs": { "question": "hello" },
            "outputs": { "output": "Hello! How can I assist you today?" }
          },
          {
            "inputs": { "question": "I would like to fly from" },
            "outputs": {
              "output": "Possible departure airports based on the context provided:\n- Manchester (\"MAN\")\n- Luton (\"LTN\")"
            }
          }
        ],
        "question": "I would like to fly from Man to "
      }
    ]
  },
  "useWorkspaceConnection": true,
  "useFlowSnapshotToSubmit": true
}

I placed the entire Json inside of the data variable but the prompt didn't change

User's image

So instead of what's in the data variable now we paste the entire Json above to replace it. As we have different headers and body contents as the prompt utilises the gpt model on two different areas as well as an encoding model (ada), so decided to copy all values within the data variable.

1 answer

Your answer

romungi-MSFT 48,911 Reputation points Microsoft Employee Moderator

2023-12-07T08:07:36.46+00:00

@Emmanuel Falola Did you try to enable developer mode on your browser and check if the request from the studio matches that of postman or VS code? I have used this before to confirm if I am calling the API with the correct details, like the headers, body, etc.

Answer 1

Azar 29,520 MVP Volunteer Moderator

Hi Emmanuel Falola

This might be due to several things, Plz Ensure that the endpoint configuration in Postman or VSCode matches the configuration you used during testing in the Machine Learning Studio. this can be the reason sometimes.

Also check that the input data you are sending via Postman is formatted correctly. Ensure that it matches the format expected by your deployed model.

Confirm that your requests are properly authenticated and authorized. If your endpoint requires authentication, make sure you are including the necessary tokens

Check this and if it helps kindly accept the answer thanks again.

Emmanuel Falola 0 Reputation points

2023-12-06T13:27:57.9133333+00:00

I've mirrored the configurations from the ML Studio for my VScode but still run into the same issues

For the formatting question, the data I am sending I believe to be correct as I am getting some relevant information back, its just not the same as the ML Studio prompt flow responses I was getting.

I don't believe its an authentication issue as I am not getting any errors, I am receiving responses from the LLM they are just not what I expected

Share via

Customised prompt not present when prompt flow deployed

1 answer

Your answer