pyspark convert scientific notation to string

braxx 426 Reputation points

Something what should be really simple getting me frustrated.
When reading from csv in pyspark in databricks the output has a scientific notation:

Name Code
AA 6.44E+11
BB 5.41E+12

how to convert it to string? Here is the expected output. Note, Code can have any lenght. Need both, Name and Code as a result in dataframe.

Name Code
AA 644217000000
BB 5413150000000

I tried this pyspark-how-to-remove-scientific-notation-in-csv-output but it getting me error on "avg".
The simpler the better. TIA

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,394 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. HimanshuSinha-msft 19,196 Reputation points Microsoft Employee

    Hello @braxx ,
    Thanks for the ask and using Microsoft Q&A platform .

    You mentioned that the file is CSV , but there is no comma seprated field which you have shared .

    You can try the below code .

    from pyspark.sql.functions import format_string'Name',format_string('%.1f',df.Code.cast('float')).alias('converted')).show()

    Output :
    |Name| converted|
    | AA| 643999989760.0|
    | BB|5409999945728.0|

    Please do let me know how it goes .


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

  2. Abhishek Kumar E 1 Reputation point

    hey @braxx , @HimanshuSinha-msft

    Even i'm facing the same issue. where you able to find any turn around for this issue ?


    0 comments No comments