Відео

The anti-pattern is using load balanced connections to distribute the load across the cluster when the data is already well distributed. This uses a huge amount of resources but provides no benefit.

There are a few differences but the main difference is changing what the boundary of the transaction is. With standard NiFi, the transaction is the processor. If there’s any kind of failure you deal with it for that processor. With stateless, a failure means that all of the processing in that group is rolled back and the input gets routed to failure. For a case like consuming from Kafka, the message that is received doesn’t even get acknowledged in stateless until successfully processing through the entire group. So if there’s a failure the message won’t be acknowledged and can be consumed again.

You’ll need to check the nifi-app.log to understand what the error is.

@@nifinotes5127 Thank you, can this be the reason? 2024-08-28 13:15:25,230 ERROR [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.NullPointerException: Cannot invoke "jakarta.servlet.ServletContext.setAttribute(String, Object)" because "webApiServletContext" is null

If you want to distribute the data evenly across the NiFi cluster and you have more NiFi nodes than Kafka partitions you’d want to load balance. If you had say 8 partitions and 10 NiFi nodes, though, it probably wouldn’t be worth the cost of load balancing, you’d be better off just distributing across the 8 nodes that are pulling from Kafka.

Hey @resoldab I would certainly like to be! Has just been a challenge finding the time to create videos lately with all that we have going on at Datavolo! Hopefully I can get back to it soon though. Anything in particular you’re interested in hearing about?

No, Kafka’s Exactly Once Semantics do not apply to cross-cluster processing.

@@nifinotes5127 Thanks so much for the fast answer. I felt it logically but didn't find it written somewhere clearly.

It does, but even GPT-4 struggles to produce quality Jolt Transforms. Some specific training examples may help. But I suspect the better solution will be to focus on training it to write Python (or possibly groovy) with script based processors.

It does, but even GPT-4 struggles to produce quality Jolt Transforms. Some specific training examples may help. But I suspect the better solution will be to focus on training it to write Python (or possibly groovy) with script based processors.

You have a couple of options, detailed in the Python Developers Guide: nifi.apache.org/documentation/nifi-2.0.0-M2/html/python-developer-guide.html#dependencies

Me either! :) it’s been pretty amazing working with these new Gen AI technologies!

Thanks Roberto. There are options, assuming the web service allows batch queries. Generally the pattern would be to use SplitRecord to break up that 2.2MM records into something like 100 records per flowfile - or whatever size you want to be a batch to your web service. Then you’d use ForkEnrichment, transform the data on the ‘enrich’ route, use InvokeHTTP, etc to get the enrichment data, and then use JoinEnrichment to join the enrichment data back with the original data. There are examples in the ForkEnrichment docs.

@@nifinotes5127 thank you so much! Such a prompt and comprehensive response!

@@nifinotes5127 thanks again Mark! What's the reason behind splitting a big flowfile into a 100 records chunk? And another question: would it be possible to enrich my multi rows flowfile with a complex SQL query against an Oracle database? I have to pull a column out of the enrichment db based on a rather complex query, and I would like to do this with a batch approach

@onesoulgospelchoir6742 the splitting to 100 records is to provide a reasonable size for the bulk request to the web service. You probably don’t want a single web request with 2.2MM records. Even creating that request would take a lot of memory. You could do enrichment using a database with bulk query but it’s a bit complicated. You’d need to form your own query, likely using something like SELECT * FROM TABLE WHERE COLUMN IN (…). If more help is needed id recommend the slack channel or NiFi mailing lists. UA-cam comments are not very conducive to long explanations 🙂

There have certainly been some tweaks and improvements. Some refactoring so that it's faster and definitely improvements to the lifecycle of creating processors that that dependencies are loaded in the background instead of on creation, etc. But this is the first release with the Python stuff available. So now that we're putting it out there into full circulation, I expect to see people contributing and a lot of growth to come in the very near future.

@@nifinotes5127 But in this version 2.0 M1, is this feature not available? I tried to replicate it, but it doesn't work.

Which feature are you referring to? Python based processors? They are available in this version.

No, you shouldn’t need Maven.

Sorry @jamesmedel6566 looks like I dropped the ball on responding to you. The first milestone release (2.0.0-M1) was just released, including the Python bits. There is a docker image for it on Dockerhub, as well.

Hey @juniorskates94, not that I know of. That probably would have been a good idea to create a docker image, but I didn’t think about it. And unfortunately probably won’t have the cycles to do so for a bit. In the meantime, I recommend reaching out to dev@nifi.apache.org so that you can provide more details about the error and we can have a better conversation there than we can here.

I’m not sure that I understand the question. Processors do support CRON Scheduling natively.

I’d recommend shooting a note to users@nifi.Apache.org or using the Slack channel. Makes it much easier to converse than UA-cam comments :)

@@nifinotes5127 please share slack channel link

nifi.apache.org/mailing_lists.html you can go to this page and at the bottom is the Invite Link for slack.

@@nifinotes5127 Thanku soo much for the help

The batch output ensures that all of the data is transferred out as a single batch of data. This ensures that the following processor (PublishKafkaRecord) has access to all of the data when it runs. Otherwise, it may receive only some of the data, and push just that partial data in the transaction. By doing it as a batch it will send all data in the same transaction.

The Stateless Engine is a mechanism for running a nifi flow in a 'stateless' way. In this paradigm, the content is not persisted across restarts, the flow executes sequentially, and there are some other differences. Here's a link to the README to learn more: github.com/apache/nifi/blob/main/nifi-stateless/nifi-stateless-assembly/README.md

I do not have any examples. Generally, the best approach to understanding the REST API is to just perform the task in your browser with the Developer View open. This will show you exactly what is posted to the server and the response. There is also a client library in Java and there’s nipyapi for python (tho this isn’t officially a part of Apache nifi, it’s just something developed by a community member).

You should be able to look in the logs and see why it’s invalid.

@@nifinotes5127 it states: keystore properties is invalid because must set either 0 or 3 properties for keystore. truststore properties is invalid because invalid truststore password or type specified for file {nifi.truststore location} though these services are running and enabled without ExecuteStateless.

@@PSNOYEA are you setting your keystore and truststore passwords as properties in your executestateless processor? If you're using parameters the property:value pair needs to match your parameter so it's picked up properly. Possibly even if you're hard coding the sensitive value in the controller service/processor, you need to change it to a parameter and then define it as a property in the execute stateless processor.

No, I don’t believe so. I don’t know that RabbitMQ supports such a thing. If it does, it may require updates to the processors, but I don’t know enough about RabbitMQ to say for sure

In Stateless, the entire pipeline is single threaded so the consumer will not run while other day is processing. If the processing takes too long, though, the transaction will fail. In that case, the publisher will fail, and the entire transaction will be rolled back. Then the data would be pulled again.

The commit happens in the PublishKafkaRecord processor, as the results are pushed to the destination topic.

@@nifinotes5127 That is, even if I push the data to another topic, the offset will be commited in the topic from which it was originally read? As non obvious side effect?

Correct.

No, there will be no change to script components. It’s not possible to run native python within the jvm

I would guess later this year. But being an Apache project there is no definitive timeline.

I suppose it depends on what your definition of "high traffic/low latency" is :) Effectively, it amounts to a Remote Procedure Call (RPC) for each FlowFIle, over a local socket. So a native Java processor will naturally be more efficient. But for well designed flows, the performance has generally been quite good. There are definitely areas that we intend to further improve performance, though.

Agreed, you very much should be worried about those things, for any software that you write. :) You will naturally see better performance with native Java processors, as there's no need to communicate to a separate process. But of course, well-written Java will outperform poorly-written Python, and well-written Python will outperform poorly-written Java. So it will be a matter determining what the most critical concerns are in each case.