Mule 4 | Batch Processing | Part 2 | Demo | Different Phases | Aggregation
Вставка
- Опубліковано 30 чер 2024
- Batch processing in mule is a vast and quite difficult topic to understand. At times, it may be very slow, to improve performance a sound understanding of batch processing is required. This video takes a deep dive into batch processing and explains most of the concepts at length.
Note: At 12:01 I said it's 3x the original file, instead, it's 5x the original file. Sometimes my math doesn't work well :D
Good to know concepts:
Optimizing Max Concurrency - Make sure you watch the parallel for each loop to understand how to optimize the max concurrency.
🔗 • Mule 4 | Parallel For ...
Streaming: - Useful to understand the streaming model in Batch Aggregator which supports the forward-only iterator
🔗 • Advanced Concepts in M...
Batch job Part 1:
🔗 • Mule 4 | Batch Process...
⏱ Video Timestamps
==========================
0:00 Start
0:32 Use case introduction
3:28 Exploring the persistent queues create by the batch job
4:45 Exploring object store, that has batch instance details
7:50 Simulating crash scenario and observing the recovery
12:25 Playing with batch block size and gaining improvements
17:35 Handling errors using accept policy and understanding max failed records parameter
28:25 Checking the object store for failure instances
41:00 Fixed-size batch aggregator
43:44 Streaming aggregator
46:38 Realizing the forward-only iterator in streaming aggregator
52:25 Variables behavior in batch job
01:05:33 Choosing between batch/for-each/parallel for-each
📌 Related Links
==========================
🔗 Cache in Mule 4: • Mulesoft 4 | Cache Sco...
🔗 Transactions in Mule 4: • Mule 4 | XA | Local Tr...
🔗 Classloading Isolation in Mule 4: • Refined Error handling...
🔗 API Design Best Practices: • API Design | Best Prac...
🔗 Custom Policy: • Mule 4: Create Custom ...
🔗 API Gateway and Autodiscovery: • Mule 4: API Gateway an...
🔗 Global Error Handler: • Mule 4: Create a Globa...
🎬 Popular Mule 4 Playlists
==========================
💥 Advanced Concepts in Mule: bit.ly/AdvancedMule
💥 Mule 4 Custom Connectors: bit.ly/Mule4CustomConnectors
💥 Dataweave Series: bit.ly/dataweave2
Let's connect:
=========================
💥 Twitter: / vishwas_p13
This channel is underrated. You have explained the concepts the way no one ever did. Wonderful..
This video is a pure gold. I've learnt so much details about Batch Job... Many, many THANKS!
Hello Vishwas, the concept is clearly articulated. Thank you.
What a wonderful class sir...👏👏👏👏
You are awesome Vishwas 👏
Thank you so much Vishwas.
Very informative, Thanks
Beautifully crafted and delivered. Thanks for making it simple.
Very useful video
Thank you sir it very good explanation
Hi Vishwas, Thanks a lot for your effort and awesome explanation...
I have a small doubt. If server get down while reading records from file what will happened?
Hi Vishwas How batch executes in a clustered environment ( On Premise ) or Multi Worker deployed Cloudhub application ? As per MuleSoft documentation scheduler kicks off only on primary. Does that mean none of the other nodes share load during the processing? Lets say if a 2 node cluster is being overloaded due to current batch scheduler, how can we improve performance? Is it by scaling it horizontally or vertically?
Hi Viswas, Thank you so much for this insightful video. I was trying with xml file but getting error. In case of json file, i believe mule by default put all object inside array while writing to file but not sure how it can be handled in case of xml. Please advise.
Hi Vishwas, for variable behaviour the logger is before the set variable. I have tried and is displaying as [1], [2] and so on..
Thanks Vishwas for another wonderful session. Would it be possible to share sample batch input file to do hands on by ourselves.
This is the link from where I've downloaded the CSV
www.stats.govt.nz/large-datasets/csv-files-for-download/
There are plenty of files available with different sizes.
@@Vishwasp13 Thanks a lot. Could you please do a video on Integration design patterns and how can we achieve them using Mule
Very nice .. i have one question.. processing 1000 records but 100 transactions failed continuously now want to terminate this batch job and when i start text time i dont want to re continue instead want to start from beginning.. how to do it
Hi Vishwas,
For me when I am forcefully stopping the server, the execution starts from 1st record again and new batchInstanceId gets generated.The only difference I have is the number of records in csv file is 10.Will this make any difference??
Yes, 10 records will get processed in a breeze, try using a file with a huge number of records.
Sir if we have database as source and we have huge records suppose some Laks of records and we need to sync to Salesforce how can we load those Laks of records in load and dispatch phase or what is the best practice to consider here in this scenario
Mule does that automatically in best possible way. The load and dispatch phase is not in your control. The only thing thats in your control is configuring the batch job instance.
Okay like do we need take only few records and then iterate the records using anytjing or we can directly select all the records from the database in a single run
Vishwas can you do a video on vpc and load balancing
Getting following error even though server started success and deploy success
“Server not available to handle this request, either not initialized yet or it has been disposed.”
i have been watching your mule4 series videos, you've explained the concepts very clearly
Thank you Vishwas
can i have your mailid?
Nice Job.. Can you please share source code link.. Thanks
Unfortunately, I don't have the code saved for this demo.
@@Vishwasp13 Can you please make some more videos on concepts likes US /EU control plane , Runtime Fabric,Healthcare Toolkit,Service Mesh 1.1,Government Cloud,Tcat Server etc. Appreciate your efforts :) .. Big Thanks :) :)