We have a use case to Load data from Cosmos DB to Elasticsearch

avi.p.garg 20 Reputation points
2023-11-22T05:47:34.2166667+00:00

Hi,

We have a use case to sync up data from CosmosDB to Elasticsearch through ADF. Can someone suggest the best possible way to connect Elasticsearch from ADF. Also if we have any other best option to connect cosmosdb to elastic search.

Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,168 questions
0 comments No comments
{count} votes

Accepted answer
  1. Deepanshukatara-6769 7,980 Reputation points
    2023-11-22T06:02:29.65+00:00

    Hi , hope doing well

    Please check below for your query

    Method 1: Using Azure Functions

    1. Create an Azure Function: Develop an Azure Function that retrieves data from Cosmos DB.
    2. Transform and Send to Elasticsearch: Within the Azure Function, perform necessary transformations and push the data to Elasticsearch using the Elasticsearch REST API.

    ADF can trigger this Azure Function at specified intervals or events.

    Method 2: Using Azure Data bricks

    1. Azure Databricks Cluster: Set up an Azure Data bricks cluster.
    2. Data Transformation: Use Databricks to retrieve data from Cosmos DB, transform it, and prepare it for Elasticsearch.
    3. Send to Elasticsearch: Utilize the Elasticsearch Python library or REST API to send the prepared data to Elasticsearch.

    Please accept answer if it has helped

    1 person found this answer helpful.

2 additional answers

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,401 Reputation points Microsoft Employee
    2023-11-24T05:33:02.2766667+00:00

    Hi avi.p.garg,

    Thank you for posting query in Microsoft Q&A Platform.

    Since, we don't have in built connector for Elasticsearch, we need to consider making REST API call or writing our own code in Azure Function or write script and run it using Custom activity.

    Kindly consider checking below links to know more custom and Azure Functions activities in ADF.

    Use custom activities in an Azure Data Factory or Azure Synapse Analytics pipeline

    Azure Function activity in Azure Data Factory

    Also, It's good feedback to consider implementing connector for Elasticsearch. I forwarded feedback to internal team. I would request you to have a feedback item log in below portal. Product team closly monitors feedbacks and consider for future implementations.

    https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

    Hope this helps. Please let me know if any further queries.


    Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.


  2. SivadasanUnnikrishnan-3968 0 Reputation points
    2024-06-26T04:51:34.53+00:00

    Azure functions seems to be a reliable solution, but we need to accommodate various aspects

    1. Trigger failure from Cosmos to Elastic Search.
    2. if your functionapp is on an appservice the scaling might not really work as expecred there is a huge chance that it can get throttled and fail.
    3. We need a separate service which sweeps through the data and make sure data in sync.
    0 comments No comments