Purview Discovery API - Status Code 400 - InvalidSearchQuery Error

Question

Purview Discovery API - Status Code 400 - InvalidSearchQuery Error

Atha Dwira Perdana 0

I'm trying to fetch data from my Azure Purview using Purview's Discovery - Query REST API, using the following request:

POST {endpoint}/datamap/api/search/query?api-version=2023-09-01

Here is my payload:

payload = {
  "limit": 100,
  "filter": 
  {
    "collectionId": "<collectionID>",
    "entityType": "oracle_table"
  }
}

I was able to get up to 210000 results, after that it always gives me this error:

Response: Status Code: 400 {"requestId":"55f1e7e2-4506-4003-b0b1-2e3d8c054886","errorCode":"InvalidSearchQuery","errorMessage":"The query request is invalid. Please check the payload and try again."} Total results fetched: 210000

any insights on this error? I'm not sure if it points to all data are already fetched or there's some issue with the API. I don't think all tables are returned from the API.

thank you in advance.

Atha Dwira Perdana 0 Reputation points

2024-12-12T02:05:24.6633333+00:00

Thanks for the reply @Ganesh Gurram , I forgot to mention I used pagination using continuation token for my payload-loop, here's the latest working continuation token before it stops, converted from base64

{"sessionId":"<sessionID>","offset":210000,"requestHash":null}

I've experimented with smaller increments with my payload, but it seems to always stop at offset 210000
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-12-12T06:54:58.09+00:00

@Atha Dwira Perdana
Thank you for the update.

Here are few considerations that might help to resolve the issue:
API Limitation - The API might have a cap on the number of results returned. Try reducing the limit to smaller values (e.g., 10,000 or 50,000) to check if pagination continues beyond 210,000.

Continuation Token - Ensure the continuation token is valid and fresh. Double-check the sessionId and requestHash to ensure they are up-to-date and valid.

Rate Limiting - Introduce a small delay between requests to avoid throttling, and check for rate-limiting headers (x-ms-ratelimit-remaining) in the response.

Hope this information helps.
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-12-12T22:34:59.5966667+00:00

@Atha Dwira Perdana - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-12-14T02:01:42.0466667+00:00

@Atha Dwira Perdana - Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

1 answer

Your answer

Atha Dwira Perdana 0 Reputation points

2024-12-12T02:05:24.6633333+00:00

Thanks for the reply @Ganesh Gurram , I forgot to mention I used pagination using continuation token for my payload-loop, here's the latest working continuation token before it stops, converted from base64

{"sessionId":"<sessionID>","offset":210000,"requestHash":null}

I've experimented with smaller increments with my payload, but it seems to always stop at offset 210000
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-12-12T06:54:58.09+00:00

@Atha Dwira Perdana
Thank you for the update.

Here are few considerations that might help to resolve the issue:
API Limitation - The API might have a cap on the number of results returned. Try reducing the limit to smaller values (e.g., 10,000 or 50,000) to check if pagination continues beyond 210,000.

Continuation Token - Ensure the continuation token is valid and fresh. Double-check the sessionId and requestHash to ensure they are up-to-date and valid.

Rate Limiting - Introduce a small delay between requests to avoid throttling, and check for rate-limiting headers (x-ms-ratelimit-remaining) in the response.

Hope this information helps.
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-12-12T22:34:59.5966667+00:00

@Atha Dwira Perdana - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ganesh Gurram 7,295 Reputation points Microsoft External Staff Moderator

2024-12-14T02:01:42.0466667+00:00

@Atha Dwira Perdana - Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Answer 1

Hello @Atha Dwira Perdana

Thanks for the question and using MS Q&A forum.

The error you're encountering, InvalidSearchQuery, typically indicates that there is an issue with the structure or content of your query payload. Since you mentioned that you were able to fetch up to 210,000 results before receiving the error, it could suggest that the API has limitations on the number of results returned or that the query is becoming too complex or exceeds certain thresholds.

Here are a few suggestions to consider:

Check Payload Structure: Ensure that the payload is correctly structured according to the API documentation. Double-check the required fields and their formats.

Limit Results: Since you are hitting a limit, consider implementing pagination in your requests. Instead of trying to fetch all results at once, you can use the limit and offset parameters to retrieve results in smaller batches.

Review Collection ID: Make sure that the collectionId you are using is valid and that it corresponds to the data you expect to retrieve.

API Version: Confirm that the API version you are using (2023-09-01) is compatible with the features you are trying to access.

Error Handling: Implement error handling in your code to capture and log more details about the request and response, which may provide additional insights into what might be going wrong.

For more details refer to this: Troubleshoot Microsoft Purview Data Map API request timeout exceptions

Hope this helps. Do let us know if you have any further queries.

Share via

Purview Discovery API - Status Code 400 - InvalidSearchQuery Error

1 answer

Your answer