Sparkpool library Error for adding a zen-engine 0.22.0 in requirement.txt

Question

Hi,
PySpark version: 3.3.1.5.2.20240522.3
Python version: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0]

I am getting an error when adding zen-engine in requirements.txt in sparkpool in azure synapse spark pool. The error indicates that there is an issue with building the zen-engine==0.22.0 package during the installation process. The error log points out a failure in preparing metadata for zen-engine from the pyproject.toml

• Following is the error message using requirements.txt:
  Collecting zen-engine==0.22.0 (from -r /usr/lib/library-manager/bin/lmjob/sandbox/condaenv.gcir2h2z.requirements.txt (line 35)) Downloading zen_engine-0.22.0.tar.gz (124 kB) 124.8/124.8 kB 19.4 MB/s eta 0:00:00 Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'error'
• Following is the error while using notebook in sparkpool running in the session: Collecting zen-engine Using cached zen_engine-0.23.0.tar.gz (125 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [6 lines of output] Checking for Rust toolchain.... Cargo, the Rust package manager, is not installed or is not on PATH. This package requires Rust and Cargo to compile extensions. Install it through the system's package manager or via https://rustup.rs/ [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details.