How to not duplicate data when updating Sentinel data connector

Question

We implemented the Sentinel data connector using the Azure function app.

We have an issue now, when we need to make any updates to the data connector and make a deployment, the customer needs to create a new workspace to avoid data duplication.

Is there a way to avoid this and make the new version of the data connector replace the old one without needing customers to clean up existing data or create a new workspace?

Answer 1

@Xiuyang Bobby Sun

Kindly follow: Custom data ingestion and transformation in Microsoft Sentinel, which offers "Filtering"

Ingestion-time transformation provides you with the ability to filter out irrelevant data even before it's first stored in your workspace.

You can filter at the record (row) level, by specifying criteria for which records to include, or at the field (column) level, by removing the content for specific fields. Filtering out irrelevant data can:

• Help to reduce costs, as you reduce storage requirements
• Improve performance, as fewer query-time adjustments are needed

Ingestion-time data transformation supports multiple-workspace scenarios.

Kindly follow Tutorial, as Transformations allow you to modify incoming data before it's stored in Azure Monitor.

If you don't have any further queries and the suggested answer is as per your business need, please "Accept the answer", This will help us and others in the community as well.

Thanks,

Akshay Kaushik