80. Databricks | Pyspark | Tips: Write Dataframe into Single File with Specific File Name

[@code_nation]

As pandas is slow, we can use this function too, I changed the separator to pipe format but if you want it as comma only then remove the sep from options,
in path make sure to give file name with format at the end of path: Ex. path = "/mnt/dl2container/folder/file.csv"
def to_single_file_csv(dataframe, path):
tmp_path = path.rsplit('/',1)[0]+'/tmpdata'
dataframe.coalesce(1).write.options(header = "True", sep = "|").csv(tmp_path)
file = dbutils.fs.ls(tmp_path)[-1][0]
dbutils.fs.cp(file, path)
dbutils.fs.rm(tmp_path, True)

[@rajasdataengineering7585]

Thanks for sharing the UDF!

[@lalithroy]

Hi Raja thank you for making videos in your own voice. Could you please make a videos on delta live tables as industry is moving towards it.

[@rajasdataengineering7585]

Sure Lalith, will make videos on DLT

[@sravankumar1767]

Superb explanation Raja

[@rajasdataengineering7585]

Thanks Sravan 👍🏻

[@nagulmeerashaik5336]

Thank🙏... Do more videos this series plssss....

[@rajasdataengineering7585]

Sure, will post more videos 👍🏻

[@sachinjosethana]

great explanation

[@rajasdataengineering7585]

Glad it was helpful!

[@sabastineade2115]

This was really helpful, can we do the same when saving output into S3 Bucket in AWS?

[@rajasdataengineering7585]

Yes we can do

[@balajia8376]

Thanks Raja.. will it work for parquet format?

[@rajasdataengineering7585]

Yes Balaji, it will work

[@nestam8669]

Hi Raja. Will there be any performance degradation while converting from spark df to pandas df?

[@rajasdataengineering7585]

Yes there is performance difference while applying any transformation in pandas dataframe as it has limitations with data distribution and parallel processing

[@pankajshende679]

Even I created folder before writing data from pandas df, I have getting error cannot save file in non-existent directory. could you please help why getting this error.

[@brahmendrakumarshukla3136]

Thanks Raja, Could you also help here to dataframe write in .xlsx file

[@rajasdataengineering7585]

Sure, will do

[@sachinjosethana]

Could you share the videos for Delta Live tables

[@rajasdataengineering7585]

Sure, will post videos on DLT soon

[@khandoor7228]

good job thanks!

[@rajasdataengineering7585]

Thanks 👍🏻

[@pankajjagdale2005]

how to overwrite this file ?

[@rajasdataengineering7585]

We can use mode("overwrite")

[@pankajjagdale2005]

but mode overwrite not working in pandas i tried that way @@rajasdataengineering7585

[@darnellyork7019]

ᎮᏒᎧᎷᎧᏕᎷ

[@kap58627]

Here is solution in spark: from pyspark.sql import SparkSession
# Create a SparkSession with the required configuration
spark = SparkSession.builder \
.appName("SingleFileOutputWithoutSuccessCommittedFiles") \
.config("spark.sql.sources.commitProtocolClass",
"org.apache.spark.internal.io.cloud.PathOutputCommitProtocol") \
.getOrCreate()
# Read your data into a DataFrame (replace 'your_data' with the appropriate data source)
df = spark.read.csv("your_data.csv")
# Perform your transformations on the DataFrame (if needed)
# Coalesce the DataFrame into a single partition
# This will ensure that the data is written to a single output file
df_single_partition = df.coalesce(1)
# Write the DataFrame to your output location
# (replace 'output_path' with the desired location)
df_single_partition.write.csv("output_path", header=True)
# Stop the SparkSession
spark.stop()