I am trying to run pyspark script. In this script i am executing sql query and creating dataframe after i am trying to change last column position into first but it is giving me error. Can you please help me how to do this?

code:

if(masterjobname.endswith('ADDR_Phani')):
            df = sqlContext.sql(query)
            print("Target:  "+targetpath )
            w = (Window().orderBy("SOURCE_COLUMN_VALUE"))
            df = df.withColumn("SYSTEM_ID", dense_rank().over(w))
            cols = df.columns.tolist()
            cols = cols[-1:] + cols[:-1]
            df = df[cols]

score:0

No matter if your variable is or isn't a list:

list(var)

You can use it as exception handling; i.e. ensure that it is indeed a list object.

score:1

Okay i got the answer.

df.columns.values.tolist() gives the list of the columns.

solution:

if(masterjobname.endswith('ADDR_Phani')):
            df = sqlContext.sql(query)
            print("Target:  "+targetpath )
            w = (Window().orderBy("SOURCE_COLUMN_VALUE"))
            df = df.withColumn("SYSTEM_ID", dense_rank().over(w))
            cols = df.columns.values.tolist()
            cols = cols[-1:] + cols[:-1]
            df = df[cols]

Related Query

More Query from same tag