Reading csv file from DBFS Community Edition with pandas

Question

getting an error while reading the file in dbfs with pandas but its working with spark by using same path

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs:/FileStore/tables/loan_classification.csv' --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) File <command-1879621862619562>:1 ----> 1 df= pd.read_csv('/dbfs:/FileStore/tables/loan_classification.csv') File /databricks/python/lib/python3.9/site-packages/pandas/util/_decorators.py:311, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs) 305 if len(args) > num_allow_args: 306 warnings.warn( 307 msg.format(arguments=arguments), 308 FutureWarning, 309 stacklevel=stacklevel, 310 ) --> 311 return func(*args, **kwargs) File /databricks/python/lib/python3.9/site-packages/pandas/io/parsers/readers.py:680, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options) 665 kwds_defaults = _refine_defaults_read( 666 dialect, 667 delimiter, (...) 676 defaults={"delimiter": ","},

Answer 1

@Kumar - Thanks for the question and using MS Q&A platform.

This is a known limiation with Databricks community edition.

Note: This is a known issue with pandas reading the csv files from dbfs:/FileStore/tables

This article provides examples for reading and writing to CSV files with Azure Databricks using Python, Scala, R, and SQL.

And also, checkout the SO threads addressing similar issue:

https://stackoverflow.com/questions/66310652/data-bricks-community-edition-cannot-read-uploaded-table-was-working-before

https://stackoverflow.com/questions/63552922/not-able-to-cat-dbfs-file-in-databricks-community-edition-cluster-filenotfounde

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.