Azure Databricks to Cognitive Search connection timing out

alvaro mantilla 20

I have a question on Azure Databricks and Azure Cognitive Search connectivity and hope this is the right forum to ask it.

We are getting a timeout message when creating an index , "ServiceRequestError: (<urllib3.connection.HTTPSConnection object at 0x7f081820e1a0>, 'Connection to xxxxx-aisearch-02.search.windows.net timed out. (connect timeout=300)')". I recreated the scenario on my own subscription and get the same error message.

The command we are running is a simple create index one: "create_index('test22_index_db', config)". We've passed the search URL, Search Key and other parameters in the config variable.

The configuration uses private endpoints and all the configuration settings seem correct. Connectivity to other AI services is working fine, Document Intelligence and OpenAI services. Connectivity to storage accounts and key vaults is fine as well.

Connectivity works when private endpoints are removed from AI Search.

Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-03-05T21:12:33.72+00:00

Hello alvaro mantilla,

Welcome to the Microsoft Q&A forum.

Per the error message, a connection timeout to the search the endpoint.

Can you please provide your connection string.

and please check your endpoint name.

Exaple: if your service name is myazuredemo, the correct endpoint should be https://myazuredemo.search.windows.net/

Reference: https://stackoverflow.com/questions/72240508/creating-cognitive-search-index-in-databricks-connection-httpsconnection-failed
Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-03-07T00:31:02.35+00:00

Hello alvaro mantilla,

I am checking to see if you have any further questions here.
alvaro mantilla 20 Reputation points

2024-03-08T22:29:51.7266667+00:00

Please see my test script below:
%python

!pip install azure-search-documents==11.4.0b8

config = { "SRCH_ENDPOINT" : "https://XXXXXXXXXXXXX.search.windows.net/",

"SRCH_KEY" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXX"

}

%python

from azure.search.documents import SearchClient

from azure.search.documents.indexes import SearchIndexClient

from azure.search.documents.models import Vector

from azure.search.documents.indexes.models import (

SearchIndex,

SearchField,

SearchFieldDataType,

SimpleField,

SearchableField,

SearchIndex,

SemanticConfiguration,

PrioritizedFields,

SemanticField,

SearchField,

SemanticSettings,

VectorSearch,

HnswVectorSearchAlgorithmConfiguration,

)

from azure.core.credentials import AzureKeyCredential

%python

def create_index(index_name, config):

index_client = SearchIndexClient(endpoint=config["SRCH_ENDPOINT"], credential=AzureKeyCredential(config["SRCH_KEY"]))

fields = [

SimpleField(name="id", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True),

SearchableField(name="content", type=SearchFieldDataType.String),

SearchField(name="contentVector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single),

searchable=True, vector_search_dimensions=1536, vector_search_configuration="my-vector-config"),

]

vector_search = VectorSearch(

algorithm_configurations=[

HnswVectorSearchAlgorithmConfiguration(

name="my-vector-config",

kind="hnsw",

parameters={

"m": 4,

"efConstruction": 400,

"efSearch": 500,

"metric": "cosine"

}

)

]

)

semantic_config = SemanticConfiguration(

name="my-semantic-config",

prioritized_fields=PrioritizedFields(

prioritized_content_fields=[SemanticField(field_name="content")]

)

)

Create the semantic settings with the configuration

semantic_settings = SemanticSettings(configurations=[semantic_config])

Create the search index with the settings declared above

index = SearchIndex(name=index_name, fields=fields,

vector_search=vector_search, semantic_settings=semantic_settings)

result = index_client.create_or_update_index(index)

print(f'{result.name} index created.')

%python

create_index('test23_index_db', config)

======================

If I remove the private endpoints from the AI Search resource it works immediately.

If I keep the private endpoints, after 5 mites or so running, the error message is:

ServiceRequestError: (<urllib3.connection.HTTPSConnection object at 0x7f2dc8737d00>, 'Connection to igsys-aisearch-02.search.windows.net timed out. (connect timeout=300)')

ServiceRequestError: (<urllib3.connection.HTTPSConnection object at 0x7f2dc8737d00>, 'Connection to XXXXX-aisearch-02.search.windows.net timed out. (connect timeout=300)')

ServiceRequestError Traceback (most recent call last)

File <command-2696193608322861>, line 1

----> 1 create_index('test23_index_db', config)

File <command-2696193608322860>, line 34, in create_index(index_name, config)

   31 # Create the search index with the settings declared above

   32 index = SearchIndex(name=index_name, fields=fields,

   33 vector_search=vector_search, semantic_settings=semantic_settings)

---> 34 result = index_client.create_or_update_index(index)

   35 print(f'{result.name} index created.')

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace.<locals>.decorator.<locals>.wrapper_use_tracer(*args, **kwargs)

   76 span_impl_type = settings.tracing_implementation()

   77 if span_impl_type is None:

---> 78 return func(*args, **kwargs)

   80 # Merge span is parameter is set, but only if no explicit parent are passed

   81 if merge_span and not passed_in_parent:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/search/documents/indexes/_search_index_client.py:256, in SearchIndexClient.create_or_update_index(self, index, allow_index_downtime, match_condition, **kwargs)

254 kwargs.update(access_condition)

255 patched_index = index._to_generated() # pylint:disable=protected-access

--> 256 result = self._client.indexes.create_or_update(

257 index_name=index.name,

258 index=patched_index,

259 allow_index_downtime=allow_index_downtime,

260 prefer="return=representation",

261 error_map=error_map,

262 **kwargs

263 )

264 return SearchIndex._from_generated(result)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/tracing/decorator.py:78, in distributed_trace.<locals>.decorator.<locals>.wrapper_use_tracer(*args, **kwargs)

   76 span_impl_type = settings.tracing_implementation()

   77 if span_impl_type is None:

---> 78 return func(*args, **kwargs)

   80 # Merge span is parameter is set, but only if no explicit parent are passed

   81 if merge_span and not passed_in_parent:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/search/documents/indexes/_generated/operations/_indexes_operations.py:713, in IndexesOperations.create_or_update(self, index_name, prefer, index, allow_index_downtime, if_match, if_none_match, request_options, **kwargs)

710 request.url = self._client.format_url(request.url, **path_format_arguments)

712 _stream = False

--> 713 pipeline_response: PipelineResponse = self._client._pipeline.run( # pylint: disable=protected-access

714 request, stream=_stream, **kwargs

715 )

717 response = pipeline_response.http_response

719 if response.status_code not in [200, 201]:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:230, in Pipeline.run(self, request, **kwargs)

228 pipeline_request: PipelineRequest[HTTPRequestType] = PipelineRequest(request, context)

229 first_node = self._impl_policies[0] if self._impl_policies else _TransportRunner(self._transport)

--> 230 return first_node.send(pipeline_request)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:86, in _SansIOHTTPPolicyRunner.send(self, request)

   84 _await_result(self._policy.on_request, request)

   85 try:

---> 86 response = self.next.send(request)

   87 except Exception: # pylint: disable=broad-except

   88 _await_result(self._policy.on_exception, request)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:86, in _SansIOHTTPPolicyRunner.send(self, request)

   84 _await_result(self._policy.on_request, request)

   85 try:

---> 86 response = self.next.send(request)

   87 except Exception: # pylint: disable=broad-except

   88 _await_result(self._policy.on_exception, request)

[... skipping similar frames: _SansIOHTTPPolicyRunner.send at line 86 (2 times)]

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:86, in _SansIOHTTPPolicyRunner.send(self, request)

   84 _await_result(self._policy.on_request, request)

   85 try:

---> 86 response = self.next.send(request)

   87 except Exception: # pylint: disable=broad-except

   88 _await_result(self._policy.on_exception, request)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/policies/_redirect.py:197, in RedirectPolicy.send(self, request)

195 original_domain = get_domain(request.http_request.url) if redirect_settings["allow"] else None

196 while retryable:

--> 197 response = self.next.send(request)

198 redirect_location = self.get_redirect_location(response)

199 if redirect_location and redirect_settings["allow"]:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py:553, in RetryPolicy.send(self, request)

551 is_response_error = True

552 continue

--> 553 raise err

554 finally:

555 end_time = time.time()

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py:531, in RetryPolicy.send(self, request)

529 try:

530 self._configure_timeout(request, absolute_timeout, is_response_error)

--> 531 response = self.next.send(request)

532 if self.is_retry(retry_settings, response):

533 retry_active = self.increment(retry_settings, response=response)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:86, in _SansIOHTTPPolicyRunner.send(self, request)

   84 _await_result(self._policy.on_request, request)

   85 try:

---> 86 response = self.next.send(request)

   87 except Exception: # pylint: disable=broad-except

   88 _await_result(self._policy.on_exception, request)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:86, in _SansIOHTTPPolicyRunner.send(self, request)

   84 _await_result(self._policy.on_request, request)

   85 try:

---> 86 response = self.next.send(request)

   87 except Exception: # pylint: disable=broad-except

   88 _await_result(self._policy.on_exception, request)

[... skipping similar frames: _SansIOHTTPPolicyRunner.send at line 86 (2 times)]

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:86, in _SansIOHTTPPolicyRunner.send(self, request)

   84 _await_result(self._policy.on_request, request)

   85 try:

---> 86 response = self.next.send(request)

   87 except Exception: # pylint: disable=broad-except

   88 _await_result(self._policy.on_exception, request)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/_base.py:119, in _TransportRunner.send(self, request)

109 """HTTP transport send method.

110

111 :param request: The PipelineRequest object.

   (...)

114 :rtype: ~azure.core.pipeline.PipelineResponse

115 """

116 cleanup_kwargs_for_transport(request.context.options)

117 return PipelineResponse(

118 request.http_request,

--> 119 self._sender.send(request.http_request, **request.context.options),

120 context=request.context,

121 )

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-a3bc208e-7f1b-435a-a849-8b96ea9f8908/lib/python3.10/site-packages/azure/core/pipeline/transport/_requests_basic.py:386, in RequestsTransport.send(self, request, proxies, **kwargs)

383 error = ServiceRequestError(err, error=err)

385 if error:

--> 386 raise error

387 if _is_rest(request):

388 from _azure.core.rest.requests_basic import RestRequestsTransportResponse

ServiceRequestError: (<urllib3.connection.HTTPSConnection object at 0x7f2dc8737d00>, 'Connection to XXXXX-aisearch-02.search.windows.net timed out. (connect timeout=300)')
Bhargava-MSFT 31,116 Reputation points Microsoft Employee

2024-03-12T21:54:02.38+00:00

Thanks for providing the additional details. The issue seems to be due to network configuration or firewall settings preventing the connection from being established.

Can yu verify that the virtual network and subnet associated with the private endpoint are properly configured and allow traffic to and from the Azure Cognitive Search service.

and also, please check if there are any firewall rules in place that might be blocking traffic to the private endpoint.
Sigita Lapina 0 Reputation points

2024-07-08T10:11:14.05+00:00

Are there any updates on this?

Share via

Azure Databricks to Cognitive Search connection timing out

Create the semantic settings with the configuration

Create the search index with the settings declared above

Your answer