- 394
- 231 936
StreamNative
United States
Приєднався 18 жов 2019
Founded by the original developers of Apache Pulsar and Apache BookKeeper, StreamNative delivers the most comprehensive event streaming platform powered by Apache Pulsar. The StreamNative Platform is a cloud-native offering that complements Apache Pulsar, providing advanced capabilities to help accelerate real-time application development and to simplify enterprise operations at scale.
TRAINING
Apache Pulsar Training: streamnative.io/training/
StreamNative Academy: www.academy.streamnative.io/
TRAINING
Apache Pulsar Training: streamnative.io/training/
StreamNative Academy: www.academy.streamnative.io/
Data Streaming Summit 2024 - Closing Keynote
Data Streaming Summit 2024 - Closing Keynote
Переглядів: 18
Відео
Data Streaming Summit 2024 Highlights
Переглядів 4214 днів тому
Check out key highlights of Data Streaming Summit 2024! You can watch full sessions at ua-cam.com/play/PLqRma1oIkcWgN9agdJ0DQhX2gPf8K2ynk.html
Data Streaming Summit Morning Keynote
Переглядів 4014 днів тому
Data Streaming: Past, Present, and Future Ben Gamble, Field CTO, Ververica Dustin Nest, Technical Trainer, StreamNative Hugo Smitter, Principal Platform Architect, FICO Kundan Vyas, Staff Product Manager, StreamNative Matteo Merli, Pulsar PMC Chair and CTO, StreamNative Naveen Punjabi, Director of Partnership, Google Cloud Sijie Guo, Co-founder and CEO, StreamNative
Measuring Physical Operations in Real Time
Переглядів 2514 днів тому
Leveraging existing cameras infrastructure, Safari AI uses computer vision to detect, track and measure KPIs like guest occupancy, staff engagement, parking utilization, queue wait time and more for our Enterprise customers. We use Pulsar as our main data storage and streaming backbone to collect ML data from edge devices and process it with Flink to timely serve and alert our customers with me...
Scaling Kafka Replication at Uber's Monumental Scale
Переглядів 11814 днів тому
Uber employs one of the largest Apache Kafka clusters in the world, acting as the pivotal hub connecting the entire Uber ecosystem. We aggregate system metrics, application logs, database changelogs, and event data from rider/driver/eats apps. This intricate process guarantees the seamless downstream availability of critical data through the Kafka platform. In 2016, we pioneered our own Kafka r...
Truly Scalable Operational Data Layers for Data Pipelines
Переглядів 2514 днів тому
As streaming systems scale to match the ever-increasing volumes of data in applications, how should data engineers think about the scale properties of the sources and destinations of streaming data? In this session, we’ll discuss scaling from the perspective of an operational data layer (both a destination and a source), or - more tangibly - the global source of truth for data aggregated from a...
From Terabytes to Petabytes: Scaling Strategies with Apache BookKeeper
Переглядів 3414 днів тому
The demand for scalable data storage is relentless as businesses grapple with exponentially growing data volumes. Salesforce's innovation with Apache Bookkeeper has emerged as a game-changer, pushing scalability boundaries. The traditional way Bookkeeper uses its metadata store - Apache ZooKeeper, struggled to keep pace as storage scaled beyond the Petabyte mark. Salesforce's innovation on the ...
Uncorking Real-Time Analytics with Pulsar, Pinot and Flink
Переглядів 4114 днів тому
Imagine a world where analytics is not just for the boardroom but for everyone, everywhere, every moment. Pinot is not your average OLAP database; it’s a turbocharged engine designed to power features that users interact with, delivering insights faster than you can say “real-time.” We’re talking about lightning-fast queries, sky-high concurrency, and data fresher than your morning coffee. In t...
From Swamps to Lakes: Building a Cleaner Data Ecosystem
Переглядів 2414 днів тому
As companies scale their data streaming operations, many find themselves stuck in a data swamp-an unmanageable mess of low-quality, disorganised data. In this session, we’ll explore how to transform these swamps into clean, high-value data lakes by focusing on early-stage data quality, controls, and automation. Using real-world examples, we’ll show how you can reduce risk, improve operational e...
Harmonious Integration of Pulsar with ClickHouse & StarRocks
Переглядів 8914 днів тому
When building traditional real-time data analytics systems, we primarily integrated Kafka and Flink through tightly coupled, predefined static aggregations. This approach had several limitations, making it difficult to respond flexibly to surges in social events or recovery scenarios, and scaling was highly constrained. To address these issues, we adopted Pulsar, which provided messaging and li...
Powering Billion Scale Vector Search at Milvus with Apache Pulsar
Переглядів 5214 днів тому
Milvus vector database is the data infrastructure behind many GenAI applications including RAG chatbots, image/video search and recommender systems. This talk will share how Milvus, the open-source vector database, leverages Apache Pulsar in its distributed architecture to deliver high-performance vector search at billion vector scale. The pub/sub design pattern decouples the data ingestion and...
How to Process Streaming Data: A Quick Guide
Переглядів 6314 днів тому
In an era where data flows in real-time, organizations face the challenge of efficiently processing streaming data to derive timely insights. This talk, “How to Process Streaming Data: An Unbiased Guide,” provides a comprehensive and practical overview of the leading stream processing tools to handle high-velocity data streams. We will explore key concepts, including event-driven architectures,...
The Power of Apache Pulsar. Harnessing Dapr to build high scale messaging at FICO
Переглядів 6414 днів тому
Learn about FICO’s experience migrating from Apache Kafka to Apache Pulsar with the help of Dapr to build high scale messaging services into our platform. We cover how we harnessed Dapr for seamless, efficient, and scalable Pulsar integration. We’ll explore: • An overview of the FICO® Platform and strategic business objectives. • Key contributions to the Dapr community, enhancing its support fo...
MotherDuck & the AI Lakehouse
Переглядів 3514 днів тому
Follow along as I talk about what we mean by "Big Data is Dead" - and what it means for your data warehouse. The world has changed and we cannot continue to use old patterns and expect success! By leveraging DuckDB with MotherDuck's powerful serverless compute, we can extend the data warehouse into the data, reading not only parquet, csv, and json but also Iceberg & Delta. Furthermore, since we...
Take control of your business monitoring with Apache Pulsar, Apache Flink and RisingWave
Переглядів 4714 днів тому
Vehicles in the automotive industry generate an impressive amount of data during their lifetime. Aircraft engine maintenance is a long, complex process that generates data that allow tracking progress and identify issues. Despite their differences, these use cases share the same need for efficient processing of the produced data, in order to monitor the processes and ensure the expected quality...
VERA: The Engine Revolutionizing Apache Flink
Переглядів 4414 днів тому
VERA: The Engine Revolutionizing Apache Flink
Achieving Data Interoperability with Real-time Stream Processing
Переглядів 3614 днів тому
Achieving Data Interoperability with Real-time Stream Processing
Lessons from building streaming first Data Infra on a tight budget at CityStorageSystems
Переглядів 3614 днів тому
Lessons from building streaming first Data Infra on a tight budget at CityStorageSystems
Process Realtime State Efficiently with Apache Pulsar and Apache Spark transform WithState Operator
Переглядів 2714 днів тому
Process Realtime State Efficiently with Apache Pulsar and Apache Spark transform WithState Operator
Learnings from running high-volume data streaming with Kafka & Flink
Переглядів 7114 днів тому
Learnings from running high-volume data streaming with Kafka & Flink
Making Kafka Connectors Dance with Apache Pulsar
Переглядів 9314 днів тому
Making Kafka Connectors Dance with Apache Pulsar
Kafka Mirror Maker vs Pulsar Geo-Replication: Best Practices for Disaster Recovery
Переглядів 2414 днів тому
Kafka Mirror Maker vs Pulsar Geo-Replication: Best Practices for Disaster Recovery
Unlocking Real-Time Insights: StreamNative & Timeplus Integration for Instant Analytics
Переглядів 4828 днів тому
Unlocking Real-Time Insights: StreamNative & Timeplus Integration for Instant Analytics
Three Challenges of Data Streaming in the Cloud and How to Overcome Them
Переглядів 62Місяць тому
Three Challenges of Data Streaming in the Cloud and How to Overcome Them
Data Streaming Summit 2024 Morning Keynote
Переглядів 2242 місяці тому
Data Streaming Summit 2024 Morning Keynote
Cost Effective Kafka: Strategies for Cost Reduction and Increased Efficiency
Переглядів 642 місяці тому
Cost Effective Kafka: Strategies for Cost Reduction and Increased Efficiency
SweetStreams Episode 4: Real-Time AI with Zilliz and StreamNative, guest Timothy Spann of Zilliz
Переглядів 772 місяці тому
SweetStreams Episode 4: Real-Time AI with Zilliz and StreamNative, guest Timothy Spann of Zilliz
thanks for the video. is there any good UI for Pulsar? pulsar-manager is barely working
This is awesome
Timecodes, please!
Nice
Thanks for the episode. Timothy is always fun to listen to and watch :)
Hey, I want to adjust broker configuration (bundle splitting algo ) , but those attributes not present in pulsar helm chart , what should I do ?
You said there are pulsar functions using node js too? I don't see it anywhere. Could you please clarify.
Thanks for the demo. Can you share the producer app?
Thank you for the video. The explanation is good. i am looking for more information apart from running few commands. Can someone please let me know any other resources from stream native for pulsar? As i am a beginner, i would like to explore more.
Thank you for this short introduction video for Apache Pulsar.
Can this functions be used to process data like fetching data from salesforce and store it into database or storing data from CSV to database?
Very useful information. Thanks, Julien. Would be helping if you can create a video on schema based transactions instead of simple text messages.
This is a good resource to understand Pulsar Broker and Bookie GC. Is there a English version of this presentation or video?
great video Asaf !
anyone using this ?
Great walkthrough, thank you.
Great video
Great! Can you upload source code to public version control?
Why There is no Transaction Support for C++ Client, As this Transaction support is the biggest problem due to which many existing products not using Pulsar !! please work on it !
great!pulsar
Good problem and the solution plan. I hope that in 2024 we'll be able to try it in action. Having the mentioned links in the UA-cam video would be very useful. It applies to every video. Also having presentations on SlideShare would be just wonderful! > Pulsar Function authors will get OTel interface, and use any type they want. Nice! Also a bit off-topic. Is there any work going on to support WASM in Functions?
It would be great if the video description included time codes.
Thank you for your comments! We added the time codes on the description.
1. Stop on error - needs alert system and manual invervention 2. Ignore error - may cause data loss. 3. Retry - out of order processing, potential infinite message redelivery 4. DLQ - out of order processing again, finite number of redeliveries. 5. Retry queue
nice ne , can you please these sample code
please share the sample project repo
Hi, I have followed these steps and completed the Geo replication setup. When i try to produce message from US-WEST cluster , I am not able to consume from US-EAST. Can you please help me to fix the issue
Thank you, Julien. We need more materials with such production quality on Pulsar!
Thank you so much for the feedback! I'm glad you enjoyed the video! This is not the last video: stay tuned 🙂
Hello. Please, Send this project here for a sample
This stuff is really nice
Nice talk Tarun!
Nice work! The first few synchronous "pulls" from the newly created Pub/Sub subscription did not work because the first "hello" was published before the subscription was created. Generally, synchronous "pull" is guaranteed to return messages even if you set how many messages you want to receive.
Amazing introduction. Thank you
Such videos are very useful. Thank you.
Thanks to Sijie and Matteo for the presentation!
Very cool - blue/green deployments should have better support within Pulsar.
🤔 Promo SM!!!
Thank you for a great talk!
Yeah Sijie and Addison!
How about commenting at least something you're doing on the screen? And reduce speed a little? Material itself is interesting. Demo is far from the good one
With this receive approach, you are processing messages one by one. What if you get 10k messages all at once. it takes a lot of time to process them. how can we process them in batch? like a Semaphore of 20 and it call receive only when any semaphore becomes available.
How to find documentation (ebook) and examples about Apache InLong? It seems amazing but I couldn't get good documents about it.
👀
Hi Ioannis, If we wanted to do a load test, how could we do it? It is said in the document that we can do it with pulsar-perf.
please provide timestamps
nice
Java is so unnecessarily overly complicated
Thanks for sharing ! This is gold !
no audio for few key minutes :(