We have a use case to Load data from Cosmos DB to Elasticsearch

Question

Hi,

We have a use case to sync up data from CosmosDB to Elasticsearch through ADF. Can someone suggest the best possible way to connect Elasticsearch from ADF. Also if we have any other best option to connect cosmosdb to elastic search.

Thanks

Answer 1

Hi , hope doing well

Please check below for your query

Method 1: Using Azure Functions

1. Create an Azure Function: Develop an Azure Function that retrieves data from Cosmos DB.
2. Transform and Send to Elasticsearch: Within the Azure Function, perform necessary transformations and push the data to Elasticsearch using the Elasticsearch REST API.

ADF can trigger this Azure Function at specified intervals or events.

Method 2: Using Azure Data bricks

1. Azure Databricks Cluster: Set up an Azure Data bricks cluster.
2. Data Transformation: Use Databricks to retrieve data from Cosmos DB, transform it, and prepare it for Elasticsearch.
3. Send to Elasticsearch: Utilize the Elasticsearch Python library or REST API to send the prepared data to Elasticsearch.

Please accept answer if it has helped

Answer 2

Hi avi.p.garg,

Thank you for posting query in Microsoft Q&A Platform.

Since, we don't have in built connector for Elasticsearch, we need to consider making REST API call or writing our own code in Azure Function or write script and run it using Custom activity.

Kindly consider checking below links to know more custom and Azure Functions activities in ADF.

Use custom activities in an Azure Data Factory or Azure Synapse Analytics pipeline

Azure Function activity in Azure Data Factory

Also, It's good feedback to consider implementing connector for Elasticsearch. I forwarded feedback to internal team. I would request you to have a feedback item log in below portal. Product team closly monitors feedbacks and consider for future implementations.

https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c

Hope this helps. Please let me know if any further queries.

---

Please consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Answer 3

Azure functions seems to be a reliable solution, but we need to accommodate various aspects

1. Trigger failure from Cosmos to Elastic Search.
2. if your functionapp is on an appservice the scaling might not really work as expecred there is a huge chance that it can get throttled and fail.
3. We need a separate service which sweeps through the data and make sure data in sync.