Hello @Tiago Matte Debastiani ,
This should be easy. Removing folders by AzStorage cmdlets can be done by this cmdlet:
Remove-AzStorageDirectory
More info about it: https://learn.microsoft.com/en-us/powershell/module/az.storage/remove-azstoragedirectory?view=azps-9.0.0
You already have a full list of directories in $dirsToList, so you don't need to collect this information again. The cmdlet will show an error if you try to delete a non-empty folder, but I still suggest that you verify if the folder really is empty and then run the remove command. You can also use -WhatIf until you are sure that only empty folders are removed.
If you need more help or the working script please let us know.
Vukasin