Integrating large watchlist created using csv file(size: 5MB) available in azure storage account with solution package?

Question

We have created a custom watchlist using a .csv file available in the storage account with a size of 5MB(as there is a limitation of 3.8MB for a local CSV file) and used watchlist data in the workbook. Now, we want to ship it in our Sentinel Solution package to make it publicly available in Microsoft Sentinel.

We are aware of creating a watchlist template with limited data. So, should we follow the same process to create a watchlist template with larger data, or is there any other way to ship a watchlist template with a solution package created using an Azure storage account?

If we add the raw content of a 5MB CSV file directly in the watchlist template will it create any issues in deployment?

Can you please provide guidance on it?

Answer 1

Hi Nirali Shah,

Thanks for reaching out to Microsoft Q&A.

To create a watchlist from a large file that's up to 500 MB in size, upload the file to your Azure Storage account. Then create a shared access signature URL for Microsoft Sentinel to retrieve the watchlist data. A shared access signature URL is an URI that contains both the resource URI and shared access signature token of a resource like a csv file in your storage account. Finally, add the watchlist to your workspace in Microsoft Sentinel.

This approach should not create any issues in deployment as long as the file is properly referenced and accessible from the storage account.

For more information, see the following articles:

• Create watchlists in Microsoft Sentinel
• Built-in watchlist schemas
• Azure Storage SAS token

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hi Nirali Shah,

Welcome to Microsoft Q&A, thanks for posting your query.

To integrate a large watchlist created from a CSV file (up to 500 MB) stored in an Azure Storage account, you should avoid adding large amounts of raw content directly into the watchlist template.

Instead, follow these steps that may solves your issue

If you are facing issues with validation when using large raw content, it is likely due to the size limitations of the template. By using a SAS URL, you can bypass these limitations and successfully deploy your main template. The SAS URL allows the watchlist template to access the CSV file stored in the Azure Storage account during deployment.

1. Upload the CSV file to an Azure Storage account.
2. Create a SAS URL for the CSV file in the Azure Storage account. This URL will allow the watchlist template to access the CSV file during deployment.
3. In the watchlist template, use the "rawContent" property to specify the SAS URL of the CSV file in the Azure Storage account.

Here is an example of how the "rawContent" property can be used to reference a CSV file stored in an Azure Storage account using a SAS URL:

"rawContent": { "type": "AzureStorageUri", "value":

 "https://mystorageaccount.blob.core.windows.net/mycontainer/myfile.csv?sastoken" }

In the example above, "mystorageaccount" is the name of the Azure Storage account, "mycontainer" is the name of the container where the CSV file is stored, "myfile.csv" is the name of the CSV file, and "sastoken" is the SAS URL for the CSV file.

Using a SAS URL to reference a CSV file stored in Azure Storage is a best practice for integrating large watchlists in Azure. This method not only simplifies your deployment process but also ensures that you can manage large datasets effectively.

Hope the above answer helps! Please let us know do you have any further queries.