I'm using my Own Data with Azure OpenAI - Is it possible to get the filepath in the chat response

Question

I'm using my Own Data with Azure OpenAI - Is it possible to get the filepath in the chat response

Jamil Hallal 21

I'm using my own data with Azure OpenAI and the data is a list of documents stored in the blob storage. I need to have the filepath in the chat response of the post request to:

https://mytenantname.openai.azure.com/openai/deployments/my-deployment/extensions/chat/completions?api-version=2023-06-01-preview

Currently I receive the citation but the filepath is always null. The following is a part of the JSON response:

"id": null, "title": null, "filepath": null, "url": null, "metadata": {"chunking": "orignal document size=29204. Scores=3.6135864 and None.Org Highlight count=91.Filtering to chunk no. 4/Highlights=7 of size=193"}, "chunk_id": "4"}

If I check the filepath in the index in the cognitive search, I can see that the filepath is populated properly with the actual filepath.

You can also refer to the response example in Microsoft documentation and you can see that the filepath is null. Is there a way to get this filepath in the response?

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-08-07T10:56:57.11+00:00

Jamil Hallal Greetings!

I'm looking into this and update you as earliest with my findings.

Appreciate your time and patience.

romungi-MSFT 48,906 Microsoft Employee Moderator

@Jamil Hallal I think that field is returned if you pass the optional field filepathField which is part of fieldsMapping

See this implementation of BYOD feature webapp that returns the file name when it is deployed as webapp from the AOI portal.

        "dataSources": [
            {
                "type": "AzureCognitiveSearch",
                "parameters": {
                    "endpoint": f"https://{AZURE_SEARCH_SERVICE}.search.windows.net",
                    "key": AZURE_SEARCH_KEY,
                    "indexName": AZURE_SEARCH_INDEX,
                    "fieldsMapping": {
                        "contentFields": AZURE_SEARCH_CONTENT_COLUMNS.split("|") if AZURE_SEARCH_CONTENT_COLUMNS else [],
                        "titleField": AZURE_SEARCH_TITLE_COLUMN if AZURE_SEARCH_TITLE_COLUMN else None,
                        "urlField": AZURE_SEARCH_URL_COLUMN if AZURE_SEARCH_URL_COLUMN else None,
                        "filepathField": AZURE_SEARCH_FILENAME_COLUMN if AZURE_SEARCH_FILENAME_COLUMN else None
                    },
                    "inScope": True if AZURE_SEARCH_ENABLE_IN_DOMAIN.lower() == "true" else False,
                    "topNDocuments": AZURE_SEARCH_TOP_K,
                    "queryType": "semantic" if AZURE_SEARCH_USE_SEMANTIC_SEARCH.lower() == "true" else "simple",
                    "semanticConfiguration": AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG if AZURE_SEARCH_USE_SEMANTIC_SEARCH.lower() == "true" and AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG else "",
                    "roleInformation": AZURE_OPENAI_SYSTEM_MESSAGE
                }

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-09-04T08:06:24.7766667+00:00

@Jamil Hallal Did you get a chance to see suggestions from romungi-MSFT?

Do let us know if that helps or have any further queries.
BH 0 Reputation points

2023-12-13T01:00:16.7666667+00:00

Is romungi-MSFT's answer still valid? Because this did not work for me. I tried to include the fields mapping field in my API call and got this error:
"Validation error at #/dataSources/0/AzureCognitiveSearch/parameters/fieldsMapping/contentFields: Input should be a valid list"
Jamil Hallal 21 Reputation points

2023-12-13T09:19:57.9733333+00:00

@AshokPeddakotla-MSFT the proposed solution didn't work for me. I'm still not able to get the filepath in the answers.

1 answer

Your answer

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-08-07T10:56:57.11+00:00

Jamil Hallal Greetings!

I'm looking into this and update you as earliest with my findings.

Appreciate your time and patience.
romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-08-11T13:54:32.4833333+00:00

@Jamil Hallal I think that field is returned if you pass the optional field filepathField which is part of fieldsMapping

See this implementation of BYOD feature webapp that returns the file name when it is deployed as webapp from the AOI portal.

"dataSources": [ { "type": "AzureCognitiveSearch", "parameters": { "endpoint": f"https://{AZURE_SEARCH_SERVICE}.search.windows.net", "key": AZURE_SEARCH_KEY, "indexName": AZURE_SEARCH_INDEX, "fieldsMapping": { "contentFields": AZURE_SEARCH_CONTENT_COLUMNS.split("|") if AZURE_SEARCH_CONTENT_COLUMNS else [], "titleField": AZURE_SEARCH_TITLE_COLUMN if AZURE_SEARCH_TITLE_COLUMN else None, "urlField": AZURE_SEARCH_URL_COLUMN if AZURE_SEARCH_URL_COLUMN else None, "filepathField": AZURE_SEARCH_FILENAME_COLUMN if AZURE_SEARCH_FILENAME_COLUMN else None }, "inScope": True if AZURE_SEARCH_ENABLE_IN_DOMAIN.lower() == "true" else False, "topNDocuments": AZURE_SEARCH_TOP_K, "queryType": "semantic" if AZURE_SEARCH_USE_SEMANTIC_SEARCH.lower() == "true" else "simple", "semanticConfiguration": AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG if AZURE_SEARCH_USE_SEMANTIC_SEARCH.lower() == "true" and AZURE_SEARCH_SEMANTIC_SEARCH_CONFIG else "", "roleInformation": AZURE_OPENAI_SYSTEM_MESSAGE }
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-09-04T08:06:24.7766667+00:00

@Jamil Hallal Did you get a chance to see suggestions from romungi-MSFT?

Do let us know if that helps or have any further queries.
BH 0 Reputation points

2023-12-13T01:00:16.7666667+00:00

Is romungi-MSFT's answer still valid? Because this did not work for me. I tried to include the fields mapping field in my API call and got this error:
"Validation error at #/dataSources/0/AzureCognitiveSearch/parameters/fieldsMapping/contentFields: Input should be a valid list"
Jamil Hallal 21 Reputation points

2023-12-13T09:19:57.9733333+00:00

@AshokPeddakotla-MSFT the proposed solution didn't work for me. I'm still not able to get the filepath in the answers.

Answer 1

Kelly Yuan 0

@Jamil Hallal Hey I met the same issue, but interestingly, I can get URL field response via Postman by adding fieldsMapping urlField parameter, but it doesn't work in the custom connector. Any new findings?

Kelly Yuan 0 Reputation points

2024-01-11T17:46:37.9433333+00:00

Updated: just found some chunks that include url some don't.

Share via

I'm using my Own Data with Azure OpenAI - Is it possible to get the filepath in the chat response

1 answer

Your answer