Here, Service discovery is to find which nodes are exciting at a given time. So each call needs to go thru service discovery. Is there a way to optimize it ?
There is something missing in the talk, I think the proposed architecture is increasing throughput by making the publisher choose randomly one node at a time to emit the message and if for example there are 4 nodes ,a publisher can in theory emit 4 different messages concurrently
You can just use any service discovery service for that matter, but if you want to use Redis, just make new hosts save their hostname to a redis set key, and have other hosts read from that set constantly. You'll have to handle the case of hosts going offline the same way just in reverse basically
Even internal PubSub services, like the ones provided by cloud providers, will be much more expensive than running your own solution like the one i suggested
I don't get it.. How is it more scalable? Channel names are distributed between nodes? Publisher and subscriber is connected to every node, so unless someone explains me, it looks just like cluster: message is replicated to each node.
Looks like still messages were duplicated. Other option could have been distribution of channels across nodes and then using service discovery connect to nodes which hold those channels
It depends on your service discovery solution, but in our case, the Redis key that holds that specific node's hostname would simply expire after a few seconds because it won't be refreshed anymore
Here, Service discovery is to find which nodes are exciting at a given time. So each call needs to go thru service discovery. Is there a way to optimize it ?
There is something missing in the talk, I think the proposed architecture is increasing throughput by making the publisher choose randomly one node at a time to emit the message and if for example there are 4 nodes ,a publisher can in theory emit 4 different messages concurrently
At some point you gotta ask yourself it you are using the right tool for the job.
ok. Now, how do I make a service discovery?
You can just use any service discovery service for that matter, but if you want to use Redis, just make new hosts save their hostname to a redis set key, and have other hosts read from that set constantly.
You'll have to handle the case of hosts going offline the same way just in reverse basically
Very Cool, What would it take for you to want to use a SaaS over building & managing your own cluster?
Mostly cost. SaaS based PubSub messaging can get extremely expensive under high usage
@@shaharmor Is that compared to running these redis clusters and front ends for them, or is your PubSub just internal?
Even internal PubSub services, like the ones provided by cloud providers, will be much more expensive than running your own solution like the one i suggested
@@shaharmor so what are you using for the last mile connection? WebRTC data channels with Socket.io signalling?
Great talk!
Nice pattern
I don't get it.. How is it more scalable? Channel names are distributed between nodes?
Publisher and subscriber is connected to every node, so unless someone explains me, it looks just like cluster: message is replicated to each node.
A publisher sends each message to only one node, compared to Redis Cluster which forwards that message to all nodes
Really nice talk
Nice solution In that time when you did it it was quite hard to do that, Can I ask that Did you switch to streams?
Hey, we haven't switched to streams, and our pubsub cluster solution is still running in production handling millions of messages per second
Looks like still messages were duplicated. Other option could have been distribution of channels across nodes and then using service discovery connect to nodes which hold those channels
Well, arent messages heavily being delivered multiple times to subscribers in this case ?
What happens when a node crashes and has no chance to deregister itself from the service discovery ?
couldn't you have 'service discovery' node ping the registered nodes and remove it on timeout ?
It depends on your service discovery solution, but in our case, the Redis key that holds that specific node's hostname would simply expire after a few seconds because it won't be refreshed anymore
"Talk is cheap, show me the code." Linus Torvalds
Increíble solución