Filter data in Azure Open AI Completion Extensions by content type

Miguel Lopes Isidoro 41 Reputation points
2023-10-26T17:28:09.06+00:00

Hi,

I am using the Azure Open AI Completition Extensions in a project I am in.

Basically, I have a Azure Blob Storage with a container with the following types of information (files both in PDF and Markdown formats):

  • Generic articles
  • Events
  • Documents
  • People (Name, Email, Mobile Phone, Skills, Past Projects)

I then used the "Add Your Own Data" feature to create a data source, index and indexer that indexes the content that is target in the Chat Completions API calls.

I am having satisfactory results to every prompt except for people related questions. If I ask questions like the ones below the results return a lot of information (mostly irrelevant) but not People results.

  • Who is Miguel Isidoro?
  • Who has expertise in TypeScript?

What I would like to do is the following:

  • Have a way in the Blob Storage to distinguish between the above content types
  • Have an index with a content type field
  • Have a way for the indexing process to populate this new field according to the content type
  • Have a way in the Completion Extensions API to filter by a content type

I would like to continue using the Completion Extensions API.

Is this possible? If yes, how?

If not, what is the alternative to be able to filter by a specific content type?

Thanks,

Miguel

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
1,100 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
3,371 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Janarthanan S 700 Reputation points
    2023-10-28T04:32:48.78+00:00

    Hi @Miguel Lopes Isidoro

    Yes it is possible to Azure OpenAI on your data lets you restrict the documents that can be used in responses for different users with Azure Cognitive Search security filters. When you enable document level access, the search results returned from Azure Cognitive Search and used to generate a response will be trimmed based on user Microsoft Entra group membership. You can only enable document-level access on existing Azure Cognitive search indexes.

    You can find it in the detailed documentation as mentioned below Document level access control.

    https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data

    The content filtering system integrated into Azure OpenAI Service runs alongside the core models and uses an ensemble of multi-class classification models to detect four categories of harmful content (violence, hate, sexual, and self-harm) at four severity levels respectively (safe, low, medium, and high).

    Please find content filtering system in the Microsoft documentation.

    https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters

    I hope the solution is useful to you and then accept the answer.

    Regards,

    Janarthanan S


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.