Error when creating table in Unity Catalog enabled cluster with 14.3 version

Question

I am trying to create a table in schema using dbutils.fs.ls("abfss://container-name@storageaccount.dfs.core.windows.net/fullpath/") in Unity Catalog enabled cluster with 14.3 version, but I keep getting this error message:

"AnalysisException: INVALID_PARAMETER_VALUE: There are more than 1001 files. Please specify [LIMIT] to limit the number of files returned."

How can I resolve this issue in Unity Catalog enabled workspace? I have looked at this reference article https://community.databricks.com/t5/community-platform-discussions/invalid-parameter-value-there-are-more-than-1001-files-please/td-p/78182, but it doesn't provide a solution. Even using os, osutils doesn't seems to work.

Regards,

Ashwini A Gaikwad

Answer 1

@Ashwini Gaikwad - Thanks for the question and using MS Q&A platform.

It seems like you are trying to create a table in schema using dbutils.fs.ls() in Unity Catalog enabled cluster with 14.3 version, but you are getting an error message "AnalysisException: INVALID_PARAMETER_VALUE: There are more than 1001 files.

As per the repro from our end: I tried both 14.3 and 15.4 runtime to simulate the error and it worked fine (snapshot below). 

If the issue persists, you can try creating a new cluster with 15.4 runtime and see if it works.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

@Ashwini Gaikwad - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Error when creating table in Unity Catalog enabled cluster with 14.3 version

"AnalysisException: INVALID_PARAMETER_VALUE: There are more than 1001 files. Please specify [LIMIT] to limit the number of files returned."

Solution: The issue is resolved.

I have raised a MS ticket Case 2409160050004001 and I have received below response from them.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.