This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
I need use some geospatial python packages on my Synapse SparkPool such as geopandas. I was able to install it as described in the documentation geospatial-processing-analytics using the yml file :
name: aoi-env
channels:
- conda-forge
- defaults
dependencies:
- gdal>=3.3.0
- pip>=20.1.1
- azure-storage-file-datalake
- libgdal
- shapely
- pyproj
- pip:
- rasterio
- geopandas
- apache-sedona
However it doesn't work when trying to read data from Data Lake Gen2, it return the following error: 'No such file or directoryDriverError' and I'm pretty shure the path is correct.
I think it may be a dependency problem, so I tried to pass geopandas as a dependency but it doesn't work because it fails to install. After 30 min running the installation process it is cancelled
name: aoi-env
channels:
- conda-forge
- defaults
dependencies:
- gdal>=3.3.0
- pip>=20.1.1
- azure-storage-file-datalake
- libgdal
- shapely
- pyproj
- geopandas
- pip:
- rasterio
- apache-sedona
Hello @lucas.pontes ,
Thanks for the question and using MS Q&A platform.
Could you please share the code snippet which you are trying to read data from Data Lake Gen2 and also share the stack trace or screenshot of the error message which you are experiencing?
Hello PRADEEPCHEEKATLA-MSFT,
There is the snippet and a screenshot of the error
path = 'abfss://landing@datalaketeste1.dfs.core.windows.net/agrotools/geodata/solos_br.gpkg'
df_pedo = gpd.read_file(path)
Hello @lucas.pontes ,
Thanks for sharing the additional details.
Could you please confirm - whether you have successfully installed geopandas library using the above environment[dot]yml file?
As per the repro from our end, we have found the error details from the Apache Spark Application =>SystemReservedJob-LibraryManagement => Stdout logs
Dear PRADEEPCHEEKATLA,
The same happens when I pass geopandas as a dependency. When I pass geopandas after pip (on yml file) it works to install but doens work when I try to open a file using geopandas on a spark notebook
Hello @lucas.pontes ,
This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.
Hello @lucas.pontes ,
Did you get a chance to open a support ticket for this issue?
Hi,
I used a work around. Just open the file with pandas and converted it to geopandas. It's not ideal but works for now
Hello @lucas.pontes ,
Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.
Hi
How did you read that vector file into pandas?
I am getting the same error. I know I have geopandas installed and working as I can read in sample datasets and explore them using a webmap. I know the permissions are correct, since I can list and read all files that make up the shapefile I am trying to read (.shp, .prj, .dbf, .shx, .cpg). I still get the same "no such file or directory" error when I use a geopandas.read_file command. Please post a workaround if you have one. I like Sedona for processing, but geopandas for visualization. Thanks, this thread has been really helpful!
I am facing same issue do anybody happens to find the solution?