Scaling Redis at Twitter
Вставка
- Опубліковано 30 сер 2014
- Twitter runs some of the largest Redis clusters in production. To adapt Redis to Twitter's use cases, we have come up with both configuration best practices and several new features. This talk is to provide a case study of running Redis at scale- with numbers and stories, and what we have in mind for the future.
About Yao Yu
Yao has been on the cache team at Twitter for over three years. She worked on in-memory caching technologies, including Twitter's fork of of Redis, Twemcache, and created an analytic framework for cache backends. She has strong bias toward clean abstractions, efficient data processing and good code aesthetics. @thinkingfish
This event was organized by the San Francisco Redis Meetup group (www.meetup.com/San-Francisco-R.... If you are doing something interesting and fun with Redis and would like to share your stories get in touch with ben.arent@rackspace.com
Event produced by Rackspace and filmed at Geekdom San Francisco on July 17, 2014.
Follow @GeekdomSF for upcoming events and inquire about SF coworking membership at geekdomsf.com/
Great work presenting. Thank you for being so clear and fact based.
Great session - thank you for sharing this!
Amazing session! Very knowledgeable
Really insightful. Thanks for putting this up.
Very interesting. thanks for the insight.
Very great talk. Thanks so much for it
Great talk. Redis+Caching.
Fascinated talk
Very nice video, appreciate the effort...would be nice to see how redis is used in conjunction with storm to parallelize operations...
29:00 this guy who asked the question about persisted connection is sharp. I would add Whether the proxy is a layer 4/7 or Whether its a TLS termination proxy or not also participates in the latency. If the connections are persisted and stateful from proxy/server then latency will be great. The const comes from establishing tcp/tls ..
Bro, I have seen your comments on like 10-20 videos easily and this is still high assuming you might not have commented a lot.
This much common in viewing history means I'm unknowingly following your path and might become as knowledgeable as you in future.
Great talk, thanks !
Great video and also very interesting! I just wish I knew what it all means. Without studying comp sci this all seems very foreign.
One question actually. Which library did she mentioned around 24:46 which is used for talking between services? I can't hear it properly.
I have been brought here by University of California, San Diego's Course on Big Data !
Very good
At 49:00, the perception that relational DBA's don't like stored procedures is inaccurate. What they don't like is _Dynamic SQL_ (i.e., the relational term for 'on-the-fly scripting'). Deployed stored procedures can be optimised when they're slow, as long as their return remains compatible. It's actually preferable to have stored procedures, than a comparable amount of code sent down to the database unwrapped to do the same thing. Relational databases scripting languages are mature and production-ready, and there're optimisations possible on procs that aren't possible on standalone commands. They have to be skilfully used, that's all.
NB: Strictly speaking a closer term for on-the-fly scripting could be temporary stored procedures, but those are hardly used, that's why I don't consider them here.
You are probably right. I didn't have this as first-hand information so I quite likely got it wrong. I agree that it's a big improvement for reliability/performance if the scripting is deployed instead of ad-hoc.
you know, abstract level.. so cute
What does it mean, redis can see what the server can do and its not doing and make a service out of it :/
Main problem with Redis is that it scales terribly. The fact that it runs on a single core only is a huge limitation, as you have to run multiple instances per physical server to full utilize it, which becomes a management nightmare if you have anything more complicated than a trivial deployment.
You people should try Aerospike. It's an open source project that's used at massive scale in the advertising industry. It is an in-memory system that solves _all_ your requests --- it has the same high performance as Redis, it has clustering that works, it is multithreaded. Since it's been in deployment for years, a lot of the hard problems of memory allocation, network optimization, etc have been solved.
Instead of putting out yet another open source project, how about working with us to contribute a Redis protocol compatibility layer?
Contact me at brian@aerospike.com
Looks like a solid project with clean code. I only read a little bit but I do appreciate the work.
I don't think we will simply use it to replace our existing and in-development stack though. Long story but I'm sure you are familiar with how the ecosystem argument goes.
Yao Yue
Would you be open to a conversation? We have customers in advertising working at very similar scale, and we've satisfied all of your wishlist --- years of production experience. In Aerospike, we believe there is a better Redis --- one with clustering and flash out of the box, today. Please reach me directly, we can have an honest technical discussion.
I've read some of Aerospike, it looks great that Aerospike is 10x faster than Cassandra and MongoDB
Very amazing tech behind what’s really a pretty shitty service.
FIRED
In your retirement years, you’re going to look back upon your manchildhood with despair.
@@QUINTIX256 yes bruh already feeling dreadful about all the layoffs, it wasn't meant to be funny. I was just stating what happened.