Good explanation. I have one query, I other videos, you have also used below format StructType().add(field='id',data_type=IntegerType()) In this video, you have slightly format StructType([StructField(name='id','dataType=IntegerType()) Are both these same ?
Hi Sir your videos are helpful for me.I learned very much with your videos.... One humble request if is possible means you can do it, Atleast one video per day or 5 videos per week. Thanks in advance
Abyone help me. For me getting type error while excecutung below code error TypeError: __call__() takes 1 positional argument but 2 were given Code: from pyspark.sql.types import StringType, StructField, StringType, IntegerType data = [(1,'Narendra',2000),(2,'Modi',5000)] schema = StringType([\ StructField(name='id',dataType=IntegerType()),\ StructField(name='Name',dataType=StringType()),\ StructField(name='Salary',dataType=IntegerType())])
My humble request... please continue..
Sure. Thank you ☺️
Good explanation. I have one query, I other videos, you have also used below format
StructType().add(field='id',data_type=IntegerType())
In this video, you have slightly format
StructType([StructField(name='id','dataType=IntegerType())
Are both these same ?
Yes
Yes, but different syntax, and we have few more ways to define schema
Ur always giving informative videos.. Keep it up maheer
Thank you ☺️
Good explanation and great effort & very useful videos Thank you!!
Thank you I needed this video 👍
Beautiful explaination.
Hi Sir your videos are helpful for me.I learned very much with your videos.... One humble request if is possible means you can do it, Atleast one video per day or 5 videos per week. Thanks in advance
Good Vedio . Thanks Maheer
Welcome
Hi i have one question.how to convert 11/11/2022 1102 to YYYY-MM-DD HH:MM:ss in pyspark
Hi @subhanishaik8163
By using date_format():
df = df.withColumn('date_time_str' , lit('2022/11/11 1102'))
df1 = df.withColumn('New', date_format(to_timestamp(df.date_time_str, 'yyyy/MM/dd HHmm'), 'yyyy-MM-dd HH:mm'))
OUTPUT:
date_time_str New
2022/11/11 1102 2022-11-11 11:02
2022/11/11 1102 2022-11-11 11:02
Completed🎉🎉🎉
👍🏻
Abyone help me. For me getting type error while excecutung below code
error
TypeError: __call__() takes 1 positional argument but 2 were given
Code:
from pyspark.sql.types import StringType, StructField, StringType, IntegerType
data = [(1,'Narendra',2000),(2,'Modi',5000)]
schema = StringType([\
StructField(name='id',dataType=IntegerType()),\
StructField(name='Name',dataType=StringType()),\
StructField(name='Salary',dataType=IntegerType())])
df = spark.createDataFrame(data,schema)
df.show()
You are using "schema=StringType", I think thats a typo. Use "StructType()"