Plus one for admitting a lot of hard truths in such a diplomatic and polite tone. Scaling from 200 to 2000 for a year sounds like a monstrosity to me! - Whoa! Great job! I want to greet the one who was in charge of this situation!
On rationalizing "using your baby" vs switching to a better supported service, I like the quote on writing by William Faulkner: "You must kill your darlings". We get attached to complex, clever, difficult code, but the best code is dull, simple, unclever, or non-existant.
The takeaways for me were: Own your uptime Use a common profiling shape that works for all used programming languages Configure distributed tracing Use a common metric dashboard Consistent logging Performance IS important even if it's a small subset of a call chain Stop logging everything
One of the key takeaways (that I didn’t see in other comments) was the point regarding tradeoffs. Everything that is being decided (small or big) has tradeoffs. Thus, decisions need to be taken deliberately instead of just accepting it as the way things are headed.
Politics - succinctly - is the process of deciding who gets what, when and how. (Harold Lasswell) .. whenever there are scarce resources (including people), and people fight over them, politics is what happens.
Strongly disagree. *Economics* is the process of deciding who gets what, when and how. Politics only matters when the government tries to control the economy.
This is a much better definition. I've heard the Company > Team > Self before and always wonder a) where is the end customer in that equation because they are what really matters and b) there are plenty of cases where the self's decision could go against the company but it would be the right one (ie. illegal behaviour at the company)
I think a majority of the issues brought up could be minimized by using EventSourcing rather than REST as the internal distributed system pattern. A service is responsible for writes, and letting other services consume those through e.g. feeds (using Atom, this would still be RESTful btw). Those dependent services can then project those events into whatever database they need locally, and do not depend on uptime or scalability of the other service. This also minimizes need and impact of migrations, as all databases are just projections of events, and can be rebuilt by simply deleting them (we do this in production all the time, in a rolling fashion). Blue/green deployments is also an option (and VASTLY simpler using EventSourcing) to constantly do major upgrades to services without downtime.
@@MartinMortensen event sourcing brings it own demons, but in this particular use-case, based on what uber are trying to achieve with rapid development of decouple features, you could make the argument that event sourcing would be a better trade off than distributed micro services
My thoughts after watching this video: 1. Are engineering teams at Uber comprised only of junior engineers and fresh grads? or 2. Maybe Uber doesn't have much of an engineering management culture? or 3. Maybe Uber's product organization wants things moving so fast that the engineering organization has to throw common sense away?
Good talk. Politics is an "obvious" (to me) of teams owning their *own* uptime -- not uber's uptime as a whole. Suggestion: I have this same problem : you were not looking at the audience and engaging with your eyes. Lift up your head and not down at the monitor all the time.
Multiple languages for services are advantageous as it encourages a better separation of the transaction. The challenge is an environment with bias on one language that leads to 2nd tier broken support for the other languages as no one wishes to fix the dominant platform when it errs. Much like RFCs vs. an implementation of a protocol being the reference.
Great talk. Interesting how many of the problems you discuss really do boil down to the "politics" category of human problems: emotional sunk-costs arguments for sticking with the known-but-broken vs. taking the cost of switching to the new-but-not-broken. Or as Douglas Adams said, "People are a problem."
Can someone explain what cross language context propagation means? He talks about passing context down to intermediary calls but I don't grasp what the alternative is?
The is what happens when you embrace microservices with all its buzzwordy glory. Some decisions need to be centralized eg decomposition & language to use.
43:29 I go with the notion that politics specifically "playing politics" is the notion of utilizing people to achieve a goal but making it so that the people that are important to you are satisfied. Bad politics reduces the amount of people being satisfied, good politics conversely increase the number of people satisfied.
Very informative It was unusual to see someone with the chief systems architect role in modern IT organisation It'll be good to get an understanding of different roles within Uber's IT organisation and how it is structured E.g. are these just his observations in retrospect or does he have the authority to influence future decisions based on these insights
"It's so hard to get a count of how many services are running because it's changing so rapidly" Wait what, you don't have an automatic count you do it manually?!?!
Insightful talk -- thank you. Regarding Matt's question: 'How can we safely insert the "ride with a puppy" step into the Uber architecture?'. Perhaps a dab of functional programming? That is, we assert that each microservice should be a composition of (pure) functions -- aka "pipe". These "micro-pipes", would give us the *synchronous* composability we want while preserving *asynchronicity* in the space between the services achieved through event sourcing. Google seems to call this pattern "functional microservices".
Plot twist : After reading a comment on UA-cam from a random guy with no real experience of high traffic they decided to change everything because I thinked that they're "fairly dysfunctional"
Double Plot Twist: After they noticed another guy used the word “thinked” in his arguments, they reconsidered what the else random guy is saying, and indeed fired half of people
Why not use an event-driven, non-blocking I/O to speed up REPO synchronization, aka nodejs? Is the company focus dependent upon centralized or decentralized philosophies?
Charles B. Sharpe it's not that easy. They obviously use node js but node js is single threaded and can run into similar scaling problems as any language. The complication is concurrency/caching and cpu metrics/resource management. Very complicated environment
Actually, Netflix has a presentation where they talk about that. Even when you're doing a microservice based architecture, you end up having some services that are far more critical than others. Aspects of monolithic systems begin to re-emerge. In monoliths we addressed these "critical components" with various high availability tricks. Netflix took a slightly different approach. They built Hystrix to provide systemic fallback and short-circuiting of cascade failures, which are an emergent property of a microservice based architecture.
@@Calphool222 I happened to stumble onto this talk after the one Netflix one :) I know it has been a long time since this comment but still. Really loved the netflix talk. Wish they would've done QA later.
So somebody discovered distributed systems again, called them microservices, but nobody seems to know how to make them. Is it another fad that will go away?
This was a very disappointing video. Unfortunately it seems like Uber architecture was designed without much thought in a way junior devs would approach it. Given the very basic nature of problems they're having, I am surprised that Matt hasn't mentioned problems with transaction support and REST API versioning.
So Uber is a great business idea, but as usual it was poorly implemented. But it succeed in spite of its poor implementation because it was a great business idea. The world is not a meritocracy.
Thanks for this presentation and for a completely unexpected reason. I thought that guys doing architecture at Uber actually know to do architecture. Who the fuck scales to 1k microservices in a year??? Do they even have a clue how and when to separate a microservice? Reason for a new microservice is a new bounded context. How in the hell did you manage to find 1k bounded context in the app that routes the vehicles for taxi rides? Uber is a distributed ball of mud. This guy is an antipattern himself.
Interesting how people have one personality during structured presentations and another when asked questions: the way the presenter dealt with the questions at the end was rude and condescending and probably gives an insight into the culture present at Uber
Huh? He's pulling back the curtain and describing the difficult problems Uber is facing, most of which were emergent properties of being engineering purists. The ignorance of Conway's Law and a hundred other systems thinking phenomena in engineering circles is epidemic. Code slinging is an important baseline skill that everyone needs to have, but it's *not* what it takes to lead armies of technical people, and having intentionally ignorant HR managers is *not* a solution.
It's the complete opposite. This is how a great engineer thinks. Seeing the trade-off between all the different types of methodology, technology and architectures. It is impossible to design a fully working system at a large scale. Every layer you add increase the complexity of the project exponential not matter what architecture you use or how great your starting UML was. Trade-offs are constantly added to the ones you were already facing. On top of that you are building off the previous architecture. Any flaw in the system or outdated methods or technologies only get propagated with growth. Small scale architecture is ridiculously simple relative to large scale architecture. Personally, I often find that maintaining an MVC project with 1 or 2 teammates is more time consuming and produces worse results then just coding it by myself and they are handling 2000 coders... it is crazy...
After 15 minutes... why isn't this guy sooo embarrassed that he crawls under a rock and hides? THIS guy is responsible for anything? Sounds like a junior engineer after having completed his first project (which might be close to the mark).
Plus one for admitting a lot of hard truths in such a diplomatic and polite tone. Scaling from 200 to 2000 for a year sounds like a monstrosity to me! - Whoa! Great job! I want to greet the one who was in charge of this situation!
On rationalizing "using your baby" vs switching to a better supported service, I like the quote on writing by William Faulkner: "You must kill your darlings".
We get attached to complex, clever, difficult code, but the best code is dull, simple, unclever, or non-existant.
you've probably never wrote a killer perl one-liner
@@SenorQuichotte ahaha no one would doubt my 200 character awk one liners are a piece of work.
The takeaways for me were:
Own your uptime
Use a common profiling shape that works for all used programming languages
Configure distributed tracing
Use a common metric dashboard
Consistent logging
Performance IS important even if it's a small subset of a call chain
Stop logging everything
Honesty is incredibly valuable.
Step 1) Hire two thousand developers.
Step 2) ???
Step 3) Profit.
One of the key takeaways (that I didn’t see in other comments) was the point regarding tradeoffs. Everything that is being decided (small or big) has tradeoffs. Thus, decisions need to be taken deliberately instead of just accepting it as the way things are headed.
Yeah that was cool. Make your trade offs tailored to your organisation so the bad stuff doesn't have such a big impact
Architecture 101
one of the best presentations about the danger of (micro)services
@@qm3ster ((((:-))))
I fully agree with all the statements. We faced all of these issues. One by one. In the painful ways.
Politics - succinctly - is the process of deciding who gets what, when and how. (Harold Lasswell) .. whenever there are scarce resources (including people), and people fight over them, politics is what happens.
Politics is what happens when people agree not to kill each other over resources. Scarcity has no validity. Distribution has no validity.
Strongly disagree. *Economics* is the process of deciding who gets what, when and how. Politics only matters when the government tries to control the economy.
Well said!
This is a much better definition. I've heard the Company > Team > Self before and always wonder a) where is the end customer in that equation because they are what really matters and b) there are plenty of cases where the self's decision could go against the company but it would be the right one (ie. illegal behaviour at the company)
"Immutability changes everything."
I think a majority of the issues brought up could be minimized by using EventSourcing rather than REST as the internal distributed system pattern. A service is responsible for writes, and letting other services consume those through e.g. feeds (using Atom, this would still be RESTful btw). Those dependent services can then project those events into whatever database they need locally, and do not depend on uptime or scalability of the other service. This also minimizes need and impact of migrations, as all databases are just projections of events, and can be rebuilt by simply deleting them (we do this in production all the time, in a rolling fashion). Blue/green deployments is also an option (and VASTLY simpler using EventSourcing) to constantly do major upgrades to services without downtime.
I don't even know where to start. But eventaorucing and microservices is by no means less complexity.
@@MartinMortensen event sourcing brings it own demons, but in this particular use-case, based on what uber are trying to achieve with rapid development of decouple features, you could make the argument that event sourcing would be a better trade off than distributed micro services
@@richardhp77 Event sourcing does not exclude microservices.... on the contrary
Just realized we have the exact same problems like Uber, only thing that differs is we don't have Uber as a product 😂
My thoughts after watching this video:
1. Are engineering teams at Uber comprised only of junior engineers and fresh grads?
or
2. Maybe Uber doesn't have much of an engineering management culture?
or
3. Maybe Uber's product organization wants things moving so fast that the engineering organization has to throw common sense away?
Good talk. Politics is an "obvious" (to me) of teams owning their *own* uptime -- not uber's uptime as a whole.
Suggestion: I have this same problem : you were not looking at the audience and engaging with your eyes. Lift up your head and not down at the monitor all the time.
Multiple languages for services are advantageous as it encourages a better separation of the transaction. The challenge is an environment with bias on one language that leads to 2nd tier broken support for the other languages as no one wishes to fix the dominant platform when it errs. Much like RFCs vs. an implementation of a protocol being the reference.
I watched 21 minutes of this talk and I still can't figure out if I learned anything.
Great talk. Interesting how many of the problems you discuss really do boil down to the "politics" category of human problems: emotional sunk-costs arguments for sticking with the known-but-broken vs. taking the cost of switching to the new-but-not-broken. Or as Douglas Adams said, "People are a problem."
tldr; like, we thought micro-services meant nano-services, so like, that's bad.
Isn"t a bunch of criticisms of microservices in fact a failure to choose the right size for a microservice ?
Great content , pretty much captures every pitfall of using the microservices
Can someone explain what cross language context propagation means? He talks about passing context down to intermediary calls but I don't grasp what the alternative is?
Great story, great presentation.
Very nice talk. Eclipse allows cross cutting changes. IntelliJ does not. I really miss workspaces.
That's a COVID-like growth of microservices there
The is what happens when you embrace microservices with all its buzzwordy glory. Some decisions need to be centralized eg decomposition & language to use.
Pardon my ignorance. What is WIWIK?
"What I Wish I Knew"
43:29 I go with the notion that politics specifically "playing politics" is the notion of utilizing people to achieve a goal but making it so that the people that are important to you are satisfied. Bad politics reduces the amount of people being satisfied, good politics conversely increase the number of people satisfied.
Very informative
It was unusual to see someone with the chief systems architect role in modern IT organisation
It'll be good to get an understanding of different roles within Uber's IT organisation and how it is structured
E.g. are these just his observations in retrospect or does he have the authority to influence future decisions based on these insights
29:37 Ah, ORM induced performance problems.
"Don't bring out the sticks. Carrots are the only way to go." Nailed it.
Great talk from experience with microservices - thanks
node, go, java, python - "Wow, that's like a lot of languages!" LMAO
It is. Less the better
Approaching microservices like RPC is why you will have so many issues. It's a mistake made by folks rooted in older distributed system approaches.
Great talk, well done.
Matt, such a great presentation! Also, you have tape on your face.
10x growth in engineers in 1 year is probably too fast... but might be hard to avoid
10:55 Wow!
*"By having multiple languages, it can fragment the culture."*
don't do it
@@rentsy3444 Thanks!
"It's so hard to get a count of how many services are running because it's changing so rapidly"
Wait what, you don't have an automatic count you do it manually?!?!
Insightful talk -- thank you. Regarding Matt's question: 'How can we safely insert the "ride with a puppy" step into the Uber architecture?'. Perhaps a dab of functional programming? That is, we assert that each microservice should be a composition of (pure) functions -- aka "pipe". These "micro-pipes", would give us the *synchronous* composability we want while preserving *asynchronicity* in the space between the services achieved through event sourcing. Google seems to call this pattern "functional microservices".
1. break laws
2. have a toxic culture
3. get a-holes to invest in an app made by ppl who end up doing all the work but getting no credit.
Subtext of this talk: the Uber Engineering mega-team is fairly dysfunctional.
Plot twist : After reading a comment on UA-cam from a random guy with no real experience of high traffic they decided to change everything because I thinked that they're "fairly dysfunctional"
Double Plot Twist: After they noticed another guy used the word “thinked” in his arguments, they reconsidered what the else random guy is saying, and indeed fired half of people
One of the most interesting Microservices sessions I ever saw at all. Thanks a lot.
When I hear
"Buying new computers (scaling) is cheaper than hiring software engineers."
I have shocked about this fact. Bom!
Why not use an event-driven, non-blocking I/O to speed up REPO synchronization, aka nodejs?
Is the company focus dependent upon centralized or decentralized philosophies?
Charles B. Sharpe it's not that easy. They obviously use node js but node js is single threaded and can run into similar scaling problems as any language. The complication is concurrency/caching and cpu metrics/resource management. Very complicated environment
Uber does no profit and just pumped up and hyped. Why is everyone taking it as an example?
Seems like they need a new microservice to coordinate all their microservices. :)
Tim Scanlin nice sarcasm
Actually, Netflix has a presentation where they talk about that. Even when you're doing a microservice based architecture, you end up having some services that are far more critical than others. Aspects of monolithic systems begin to re-emerge. In monoliths we addressed these "critical components" with various high availability tricks. Netflix took a slightly different approach. They built Hystrix to provide systemic fallback and short-circuiting of cascade failures, which are an emergent property of a microservice based architecture.
@@Calphool222 I happened to stumble onto this talk after the one Netflix one :) I know it has been a long time since this comment but still. Really loved the netflix talk. Wish they would've done QA later.
All of these sounds like Uber problems than microservices problems.
I really dodged a bullet staying away from Silicon Valley.
Great talk ... the number of ads in it are ridiculous though.
this guy talks like Richard from silcion valley, guess the show was accurate after all
Great presentation
So somebody discovered distributed systems again, called them microservices, but nobody seems to know how to make them. Is it another fad that will go away?
This was a very disappointing video. Unfortunately it seems like Uber architecture was designed without much thought in a way junior devs would approach it. Given the very basic nature of problems they're having, I am surprised that Matt hasn't mentioned problems with transaction support and REST API versioning.
everything is a tradeoff, everything is a choice
the frequency of him saying "like", "kinda", "sort of", "totally" and "you know" is - like - staggering, you know?
is like kinda sort of totally staggering, you know?
I can feel his frustration
ge looks so happy!
He is an excellent presenter. I have attended one of his talk in 2015
too high level and too abstract. Touching so many things at a time and not diving into details. This presentation could have been better.
like like like like like like
I can't bear it over 13:25
me too. i gave up on this presentation. I couldn't stand the "street" talk. like like you know like
thanks for sharing
Lots to think about, thanks.
I'm surprised that he's not a jibbering wreck of shredded nerves.
He kind of is though
like
Cool stuff
best have so many defs, like best libs, best cuz i know it, or best ....
So Uber is a great business idea, but as usual it was poorly implemented. But it succeed in spite of its poor implementation because it was a great business idea. The world is not a meritocracy.
Thanks for this presentation and for a completely unexpected reason. I thought that guys doing architecture at Uber actually know to do architecture. Who the fuck scales to 1k microservices in a year??? Do they even have a clue how and when to separate a microservice? Reason for a new microservice is a new bounded context. How in the hell did you manage to find 1k bounded context in the app that routes the vehicles for taxi rides? Uber is a distributed ball of mud. This guy is an antipattern himself.
How many "likes" can you say in a minute lulz
Interesting how people have one personality during structured presentations and another when asked questions: the way the presenter dealt with the questions at the end was rude and condescending and probably gives an insight into the culture present at Uber
Wow...! Are the *investors* watching...?! this is _like_ a *Technical Debt Time Bomb* of a "stack"! Uber is going to *implode* so spectacularly... 😧
sup
Like, please stop saying like, like all the time.
regular ads? downvoted
rambling so many things..yikes!
This guy doesn't seem like a very good engineer.
previous CTO of Voxer
Huh? He's pulling back the curtain and describing the difficult problems Uber is facing, most of which were emergent properties of being engineering purists. The ignorance of Conway's Law and a hundred other systems thinking phenomena in engineering circles is epidemic. Code slinging is an important baseline skill that everyone needs to have, but it's *not* what it takes to lead armies of technical people, and having intentionally ignorant HR managers is *not* a solution.
It's the complete opposite. This is how a great engineer thinks. Seeing the trade-off between all the different types of methodology, technology and architectures.
It is impossible to design a fully working system at a large scale.
Every layer you add increase the complexity of the project exponential not matter what architecture you use or how great your starting UML was.
Trade-offs are constantly added to the ones you were already facing. On top of that you are building off the previous architecture. Any flaw in the system or outdated methods or technologies only get propagated with growth.
Small scale architecture is ridiculously simple relative to large scale architecture.
Personally, I often find that maintaining an MVC project with 1 or 2 teammates is more time consuming and produces worse results then just coding it by myself and they are handling 2000 coders... it is crazy...
After 15 minutes... why isn't this guy sooo embarrassed that he crawls under a rock and hides? THIS guy is responsible for anything? Sounds like a junior engineer after having completed his first project (which might be close to the mark).
This dude can't present well at all. Decent info though.
This conference is crap
Sounds like Matt proud a lot from 3:25 . Probably, that chaos + a big pressure as a result of that become the reason of Uber engineer suicide..
like
like a like
like a like a like
I like you, do you like me?