How to list over 6000 indicators with Graph API (Sentinel)

Question

How to list over 6000 indicators with Graph API (Sentinel)

Joakim Loxdal 20

Hi!

I have added 10k indicators (domainNames) to Sentinel using the security API.

When fetching them with: GET graph.microsoft.com/beta/security/tiIndicators

I will receive 200 indicators, and a link (@odata.nextLink) which gives the url of the next page (next 200 indicators).

This continues until my script has fetched 6000 indicators, at that point I wont receive the nextLink anymore.

Is 6000 a limit, and is it possible to fetch all N indicators where N > 6000?

Thank you

Joakim

Alistair Ross 7,466 Reputation points Microsoft Employee

2023-03-29T10:16:23.06+00:00

Hi Joakim

Can you provide the script that you are running. I cannot find any paging limitations for the graph api. The only limitation I know of is 15000 indicators per tenant for Defender for Endpoint.

Also are you getting any errors in the responses relating to throttling? I don't think it is this, because the api can handle 1000 requests per minute, but that is assuming that you are the only person making these calls within that time frame.

Edit: Also because you are using Sentinel, have you considered using the newer Sentinel API, which provides a richer data set https://learn.microsoft.com/en-us/rest/api/securityinsights/stable/threat-intelligence-indicators/list?tabs=HTTP

kind regards

Alistair

Joakim Loxdal 20

Hi Alistair!

Python code below with some comments.

    def get_indicators(self):
        """ Get all indicators

        Uses paging, each response gives the 'next_link' which is used to
        fetch the next page
        """

        result = []
        next_link = f"https://{self.api_url}?$filter=targetProduct+eq+'Azure Sentinel'"
        
        while next_link:
            res = requests.get(next_link, headers=auth_headers)
            res.raise_for_status() # Throws error if not 200 status code
            next_link = res.json().get("@odata.nextLink")
            
            # Add to result list
            result += res.json().get("value")

        # !!! This list is never more than 6000 indicators, even though more are available !!!
        return result

Any ideas?

I have considered using the Sentinel API, however I do not want to mix the two different APIs.

The issues I have with the Sentinel API is that actions can not be done in bulk (except list), you would have to make 1 request per indicator for update, delete and insert. Is my understanding of this correct?

Also, just some feedback, i noticed that this page (https://learn.microsoft.com/en-us/rest/api/securityinsights/stable/threat-intelligence-indicator/create?tabs=HTTP) is called 'create' in the url and navbar, but it should be called 'update'.

Andrew Blumhardt 10,051 Reputation points Microsoft Employee

2023-03-29T15:16:42.5+00:00

You might also try the Log Analytics API to query the table directly.

Usually bulk operations require paging or chunking through multiple returned pages.
Joakim Loxdal 20 Reputation points

2023-03-30T07:01:00.84+00:00

Hi Andrew!

I don't have an issue with paging, the issue is that I can not fetch over 6000 indicators WITH paging. I was wondering if 6000 is a limit or if there is an issue on my side.

Your answer

Alistair Ross 7,466 Reputation points Microsoft Employee

2023-03-29T10:16:23.06+00:00

Hi Joakim

Can you provide the script that you are running. I cannot find any paging limitations for the graph api. The only limitation I know of is 15000 indicators per tenant for Defender for Endpoint.

Also are you getting any errors in the responses relating to throttling? I don't think it is this, because the api can handle 1000 requests per minute, but that is assuming that you are the only person making these calls within that time frame.

Edit: Also because you are using Sentinel, have you considered using the newer Sentinel API, which provides a richer data set https://learn.microsoft.com/en-us/rest/api/securityinsights/stable/threat-intelligence-indicators/list?tabs=HTTP

kind regards

Alistair
Joakim Loxdal 20 Reputation points

2023-03-29T12:20:55.7866667+00:00

Hi Alistair!

Python code below with some comments.

def get_indicators(self): """ Get all indicators Uses paging, each response gives the 'next_link' which is used to fetch the next page """ result = [] next_link = f"https://{self.api_url}?$filter=targetProduct+eq+'Azure Sentinel'" while next_link: res = requests.get(next_link, headers=auth_headers) res.raise_for_status() # Throws error if not 200 status code next_link = res.json().get("@odata.nextLink") # Add to result list result += res.json().get("value") # !!! This list is never more than 6000 indicators, even though more are available !!! return result

Any ideas?

I have considered using the Sentinel API, however I do not want to mix the two different APIs.

The issues I have with the Sentinel API is that actions can not be done in bulk (except list), you would have to make 1 request per indicator for update, delete and insert. Is my understanding of this correct?

Also, just some feedback, i noticed that this page (https://learn.microsoft.com/en-us/rest/api/securityinsights/stable/threat-intelligence-indicator/create?tabs=HTTP) is called 'create' in the url and navbar, but it should be called 'update'.
Andrew Blumhardt 10,051 Reputation points Microsoft Employee

2023-03-29T15:16:42.5+00:00

You might also try the Log Analytics API to query the table directly.

Usually bulk operations require paging or chunking through multiple returned pages.
Joakim Loxdal 20 Reputation points

2023-03-30T07:01:00.84+00:00

Hi Andrew!

I don't have an issue with paging, the issue is that I can not fetch over 6000 indicators WITH paging. I was wondering if 6000 is a limit or if there is an issue on my side.

Share via

How to list over 6000 indicators with Graph API (Sentinel)

Your answer