How to reseting a Py Spark List in Azure Databricks

Question

I am picking up records one by one from csomosdb and updating them, below this is what I am doing

for i in updatedf.collect():
item=list(container.query_items(
query='SELECT * FROM EnRecords e WHERE e.CarKey= "1111345" AND e.LCNumber=@acnum',
parameters=[ dict(name='@acnum', value=i[1])],
enable_cross_partition_query=True
))
if len(item)>0:
enrolrecords_json_string=json.dumps(item, indent=True)
dbutils.fs.put("dbfs:/mnt/NewEnrolRecords.json", enrolrecords_json_string, True)

  with open("/dbfs/mnt/NewEnrolRecords.json", 'r+') as fp:
    enrolrecords = json.load(fp)
    enrolrecords[0]["Products"][0]["ProductCode"]!= i[3]

    enrolrecords=[]
  f.close()

When looping over I can't clear enrolrecords list. It is holding on to the values from the first pass. Any suggestion?

Answer 1

Hi @Arindam Pandit ,

Thank you for posting query in Microsoft Q&A Platform.

You mean to clear enrolrecords object inside with open() block? If yes, enrolrecords variable is going to hold dictionary in it as json.load() function is going to retrun json data as dictionary object in python.

To clear dictionary we can use clear() function. kindly use code as enrolrecords.clear() and see if that helps.

Please let us know how it goes. Thank you.