Remove characters in from values pyspark

Question

Hi Expert,

How to remove characters from column values pyspark sql

I.e gffg546, gfg6544

Answer 1

Hi @Shambhu Rai

You can use this with Spark Tables + Pandas DataFrames:

Example:

import pandas as pd  
  
df=spark.table("your.table_name")  
  
pddf = df.toPandas()  
  
pddf['ColumnName']=pddf['ColumnName'].replace(regex=[r'\D+'], value="")   
  
display(pddf)  

Output without replace:

Output after using the replace function:

Documentation for converting spark to pd

https://docs.databricks.com/spark/latest/spark-sql/spark-pandas.html

--------------------------

If this is helpful please accept answer.

Answer 2

Sir,
Last point how to handle 10-25 in the same column name when we need 25 as an output in same column