begin_analyze_document encounters (InternalServerError) An unexpected error occurred.

Question

Currently, in the process of processing one kind of documents, all with the same format and almost the same length.

We were able to process the first few docs last week. However, this morning, we keep encountering this error. How should we go about this?

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "parse_preprocessed_docs":
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_plan.py", line 286, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 478, in core_dagster_event_sequence_for_step
    for user_event in _step_output_error_checked_user_event_sequence(
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 160, in _step_output_error_checked_user_event_sequence
    for user_event in user_event_sequence:
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/execute_step.py", line 95, in _process_asset_results_to_events
    for user_event in user_event_sequence:
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/compute.py", line 212, in execute_core_compute
    for step_output in _yield_compute_results(step_context, inputs, compute_fn, compute_context):
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/compute.py", line 181, in _yield_compute_results
    for event in iterate_with_context(
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 463, in iterate_with_context
    with context_fn():
  File "/home/apol/.pyenv/versions/3.10.11/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 84, in op_execution_error_boundary
    raise error_cls(
The above exception was caused by the following exception:
azure.core.exceptions.HttpResponseError: (InternalServerError) An unexpected error occurred.
Code: InternalServerError
Message: An unexpected error occurred.
Exception Details:	(InternalServerError) An unexpected error occurred.
	Code: InternalServerError
	Message: An unexpected error occurred.
	Target: 0
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 465, in iterate_with_context
    next_output = next(iterator)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 131, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 125, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/src/ops/common/documents.py", line 84, in parse_preprocessed_docs
    text_elements, table_elements = parse_ocr(preprocessed_doc)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/src/utils/document_processing/parsing.py", line 84, in parse_ocr
    ocr_result = extract_doc_with_form_recognizer(
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/src/utils/caching.py", line 81, in wrapper
    output = func(*args, **kwargs)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/src/utils/services/ocr.py", line 54, in extract_doc_with_form_recognizer
    return poller.result()
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/polling/_poller.py", line 251, in result
    self.wait(timeout)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/polling/_poller.py", line 270, in wait
    raise self._exception  # type: ignore
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/polling/_poller.py", line 185, in _start
    self._polling_method.run()
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/polling/base_polling.py", line 772, in run
    raise HttpResponseError(response=self._pipeline_response.http_response, error=err) from err
The above exception was caused by the following exception:
azure.core.polling.base_polling.OperationFailed: Operation failed or canceled
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/polling/base_polling.py", line 757, in run
    self._poll()
  File "/home/apol/undp-knowledge-hub-environment/app/dagster/.venv/lib/python3.10/site-packages/azure/core/polling/base_polling.py", line 789, in _poll
    raise OperationFailed("Operation failed or canceled")

Answer 1

Hello! Thanks so much for the quick response! Was able to try out the first bullet, but not the others since we are running this within a Dagster job.

Anyway, we just discovered that the fix for this is lowering OCR_BATCH_SIZE (e.g., to 20).

Answer 2

Hi @Apol Estrella,

I'm glad to hear that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread as a solution. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Question: begin_analyze_document encounters (InternalServerError) An unexpected error occurred.

Solution: Was able to try out the first bullet, but not the others since we are running this within a Dagster job. We just discovered that the fix for this is lowering OCR_BATCH_SIZE (e.g., to 20).

If you encounter any issues in the future, don't hesitate to reach out to the community forum for assistance. Thank you.

---

Do not forget to click Accept Answer and Yes for was this answer helpful.