pyspark convert scientific notation to string

Question

Something what should be really simple getting me frustrated.
When reading from csv in pyspark in databricks the output has a scientific notation:

Name Code
AA 6.44E+11
BB 5.41E+12

how to convert it to string? Here is the expected output. Note, Code can have any lenght. Need both, Name and Code as a result in dataframe.

Name Code
AA 644217000000
BB 5413150000000

I tried this pyspark-how-to-remove-scientific-notation-in-csv-output but it getting me error on "avg".
The simpler the better. TIA

Answer 1

hey @braxx , @HimanshuSinha

Even i'm facing the same issue. where you able to find any turn around for this issue ?

thanks,
abhishek

Answer 2

Hello @braxx ,
Thanks for the ask and using Microsoft Q&A platform .

You mentioned that the file is CSV , but there is no comma seprated field which you have shared .

You can try the below code .

from pyspark.sql.functions import format_string

---

df.select('Name',format_string('%.1f',df.Code.cast('float')).alias('converted')).show()

Output :
+----+---------------+
|Name| converted|
+----+---------------+
| AA| 643999989760.0|
| BB|5409999945728.0|
+----+---------------+

Please do let me know how it goes .
Thanks
Himanshu

-----------------------------------------------------------------------------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators