Tool for Azure cognitive search similar to Logstash?

Ketan Suthar 21 Reputation points
2022-04-22T15:16:02.543+00:00

My company has lots of data(Database: PostgreSQL) and now the requirement is to add search feature in that,we have been asked to use Azure cognitive search. I want to know that how we can transform the data and send it to the Azure search engine. There are few cases which we have to handle:

How will we transfer and upload on index of search engine for existing data?
What will be the easy way to update the data on search engine with new records in our Production Database?(For now we are using Java back end code for transforming the data and updating the index, but it is very time consuming.) 3.What will be the best way to manage when there's an update on existing database structure? How will we update the indexer without doing lots of work by creating the indexers every time?
Is there anyway we can automatically update the index whenever there is change in database records.

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
694 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Grmacjon-MSFT 15,606 Reputation points
    2022-04-23T01:44:12.977+00:00

    Hi @Ketan Suthar ,

    Thank you for using Azure. I see you also posted this question on StackOverflow and it has received a few responses. Adding some additional info and resources:

    How will we transfer and upload on index of search engine for existing data?
    There are two basic workflows for populating an index: push your data into the index programmatically, or pull in the data using a search indexer. Read this doc for more details: Data import in Azure Cognitive Search

    What will be the easy way to update the data on search engine with new records in our Production Database?

    You can import search documents into a specified index using HTTP POST. For large numbers of updates, batching of documents (up to 1000 documents per batch, or about 16 MB per batch) is recommended and will significantly improve indexing performance. Read this doc for more info. 

    POST https://[service name].search.windows.net/indexes/[index name]/docs/index?api-version=[api-version]     
  Content-Type: application/json     
  api-key: [admin key]

    Is there anyway we can automatically update the index whenever there is change in database records? The new Incremental enrichment feature in Azure cognitive search can help with this.

    Please let us know if you have further questions
    Best,
    Grace

    0 comments