Designing my Server Architecture for Scalable Web Applications
Вставка
- Опубліковано 20 лип 2024
- brilliant.org/DevonCrawford
^ This video is sponsored by Brilliant! To master computer science and support my channel use my link :)
In this video I decided to share my research over the past few weeks since I have encountered many roadblocks when designing my server architecture. My current design is just a single node js + express application which handles all routes for my front end, back end + any other web apps I want to integrate. This is design is okay for small websites but as soon as you want to do multiple apps it becomes problematic. Pretty sure my new design is called a "micro-services" approach. and lately I've just done a lot of research on what tech would best suit my needs. A few things I didn't get the chance to mention in this video.. 1) I don't actually know JavaScript that well tbh.. any real algorithmic processing I will do in c++ or c and then integrate that code using the node ffi library. 2) I only want to use JavaScript, node js and express to act as the glue between my programs and the website. I prefer languages like C, C++ or Java since the compiler will yell at you for anything you've done wrong. I'll do all my real processing in the languages which I know best. 3) See below for a bunch of resources that I found helpful when learning all of this stuff!
And also, I would really like to hear everyone's feedback and possible improvements or ideas you may think of when designing such a system.
Technology used in this video
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Server: Node js + Express (Javascript)
Front End: Next js + React (HTML, CSS, JS)
Reverse Proxy & Load Balancer: Nginx
Website Host: Digital Ocean Cloud computing platform
Data: JSON, AJAX (asynchronous calls from client to server)
Next Video:
Previous Video: • Nerds go Clubbing..
Research / Resources
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Production best practices (express): expressjs.com/en/advanced/bes...
Client vs Server side rendering: medium.freecodecamp.org/what-...
Next js: nextjs.org/
Nginx config explained: www.digitalocean.com/communit...
Social
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Website: devoncrawford.io
Twitter: / devoncrawford13
Snapchat: / devoncrawfordyt
Instagram: / devoncrawford_
Github: github.com/devoncrawford
Discord: / discord
Gear
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
- Video Editing Software (Premiere Pro CC): prf.hn/l/BOomWo3
- All Adobe Apps (I use this plan): prf.hn/l/Xv5qk5Q
- Keyboard (Velocifire TKL01): amzn.to/2AQTjJQ
- Monitors (Dell U2518D) : amzn.to/2AQQmco
- Webcam (Logitech C920): amzn.to/2Cw588c
- Desk Microphone (Blue Yeti): amzn.to/2FPFmzi
- DSLR (Canon T7i): amzn.to/2T54mpZ
- Wide Lens (Tokina 11-16mm F2.8): amzn.to/2MlpvcO
Music
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
swad beach - aruarian dance (swadmix)
Link: / aruarian-dance-swadmix
Inteus - I Pull up in a What is That
Link: / i-pull-up-in-a-what-is...
SwuM. - NEW DRUM KIT OUT NOW!
Link: / lofi-trap-drum-kit-out...
lzr - goin up
Link: / goin-up - Наука та технологія
Devon, I am a 44 year-old software developer in the Enterprise world. I have to say your videos are some of the best I've seen for what you seem to want to accomplish.
You're showing the people who watch this video the fact that you have no fear of digging through documentation, putting in lots of hours, overcoming frustration.
So many people wanting to learn how to code just assume that "experts" just go on their keyboard and just start typing away without frustration. But you seem to show the reality of it.
I would love to see you talk more about just the mental process you go through from square one to finish, the habits/practices you follow and the mental challenges you have to overcome to convince yourself that you can do this stuff.
I think that's the biggest hurdle for a lot of people is that feeling of imposter syndrome.
You were going to do really well in your career. Keep up the good work man.
I watch devon's videos too Can I get a job please? *.*
@@XenoContact im confused now.
I want you to know as a 31 year old developer currently struggling with doing new things out of my comfort zone this line "people wanting to learn how to code just assume that "experts" just go on their keyboard and just start typing away without frustration. But you seem to show the reality of it." is what I needed to hear right now, thanks
@@germanhoyos4422 You're never too old to learn something new - go get 'em :-)
I feel your comment is very important for people who know nothing in programming mainly this part "You're showing the people who watch this video the fact that you have no fear of digging through documentation, putting in lots of hours, overcoming frustration" many people fail to realize even the most competent of programmers sit reading through documentation to make stuff work. You may know every piece of syntax and code for a given language but making it work how you need is not always going to be easy without documentation :) I feel there is often too much feeling of imposter syndrome for new coders and sometimes are afraid to read documentation feeling they are cheating however it could not be much further from the truth :) the documentation is there for people to learn from, the original library creators give the documentation to aid others in working with their library and such. Do not be afraid to read documentation and watch endless videos it will make you more competent in your goals and project. I myself am stuck deciding on what project to even embark on to expand my basic c# knowledge after learning from unity making games I now want to expand into software and other things ! The key here is to read and learn and read some more :) happy learning !
Screw Python, programming in paint is the real deal in 2018
Teodor Bajenaru but...
“Python’s where it’s at”
(According to ads I get at least)
Its_ Me lol haha, I get that ad too 😂
So funny! I see that add every single day, it pisses me off lol
Someone has built a working program by making a bitmap through paint.
I also get the Unity C# course too and I did end up buying it. It is actually pretty great.
> have photoshop in the taskbar
> launches Paint to draw a workflow
kzm 夢を I probably do that same kind of stuff in microsoft word and it is easier because in word, you can snap the boxes to be in line with each other and make sure that the boxes are in the same dimensions. Also switch from portrait to landscape 🤣
Doug Cannon man you can do the same in photoshop once you're used to it and it's easier imo
just a matter of habit i guess
this is galaxy brain
Hahahaa
first thing I did when I installed my first Linux distro - googled "alternative to MSPaint".
Pinta is great btw :D
Amazing video with an amazing ad, just amazing! You don't know how much you help me with these videos each time you upload, not just because of the knowlegde, but also the motivation to graduate and learn so much more! Thanks a lot man.
yoo i watched you since minecraft and it’s crazy how you grew this channel so fast and so big! I remember when this channel was at 1K :D.
I’m learning so much from these video’s and this is the kind of stuff i wanna do later so this is really fun to watch and you make it even more entertaining. You really motivate me to do this kind of stuff like coding. Thanks for this amazing content! :)
I'm a final year computer science student this year, and this is the most informative channel for me on UA-cam. This is exactly the same topics i'm a struggling with and learning to myself at the moment. Keep up with the good videos dude, there are people out there who really benefit from them.
Great stuff, Devon. Your coding channel always delves into new tech and I want to check out next.js after this. Cheers for another great vid!
Digging, hope you continue to architect and build it and take us along the journey. Loved this and the CS degree video.
I like how you shoot your videos and edit them, inspires more developers like me to do the same with more creative ways. Keep up the good work!
I would do it just like this, but put each of the services in a docker container. That way you can have the container restart itself if the service crashes. Load balancing between containers instead of ports.
I also vote for this idea. It really is a step up.
More work eventually ends up in more convenience.
And run the containers with Kubernetes!
Another thing that might consider, is to use javascript in asynchronous way.
Also, docker swarms could be used if you happen to have more machines available to run your apps at some point.
heads up, docker containers don't automagically restart, there's some further configuration and setup needed to catch process ending errors and restart (pm2, forever, etc).
I like your approach to programming related topics. It's fluid. You sketch first then dive deeper later. Seems very productive
Dude.. your videos are so unique and fun, please keep doing what you doing! Great vid!
All developers should do this every so often, in any field. This not only gives other people insight in your way of thinking, but it also serves the 'rubber ducky' purpose. Because you're explaining it to others, your mindset changes about the problem and you get additional different views as well. Good practise man, awesome!
I’m a photographer and have no knowledge what so ever in coding, I just enjoy learning from you man. Keep it up.
You make me get excited to get back to programming and learning new stuff
Such an inspiration , these new type of youtube channels are amazing
and you are on the top of them in my opinion
KEEP IT UP
Ohh, how awesome! I am also getting started with NodeJS etc, you are basically doing exactly what I want as well. But haven't got the time to actually get my head wrapped around it due to work.
Looking forward to the next videos, I hope I can learn a lot of stuff from that already!
Don’t know you at all, but this type of video is so valuable. It’s a piece that is missing from the “self taught” coding world. It’s like there are the basics courses and then there are completed project courses and then there are “watch expert live coders do amazing things”. This right here is the messy middle of “how do you Think about a project”. Thank you for this “design meeting”. I’ll be following you.
Awesome videos where you give a great overview of what you need to do/learn to implement something! Keep doing these videos!
This is a pretty good start. As long as you have your ports in order, and you're only working with one server, this is a pretty solid plan. Though I would just call this application organization, scalability kinda refers to multiple servers, load balancing, high availability, and managing requests with idempotency
Man, you're videos are just awesome! You inspire me to start a project without knowing anything ✌🏼
dude, you are insane! don't stop doing ur vids
oh jeeeeeeeesus he liked ;0
I am beyond happy that I was able to keep up with this. Thank you!
Hey Devon. I really appreciate posts like this. Thank you.
I honestly watch these having no idea what your talking about but somehow am just genuinely interested in what your talking about 🤷🏽♂️😂. It’s like I’m in a class that I don’t need for my degree but your a really cool professor so I automatically get sucked in to paying attention. Idk if that makes sense but keep up the good work!!! Been subbed to you for a good min.!
man I really appreciate your work in making these videos. love watching them! Maybe you can even add some caching tools as well like Redis or Varnish to make it even cooler! keep up to good work :)
I'm learning more and more programming right now, and I'm building up to deploy a website. I keep coming back to this video every 2 days or so to see how much more I understand of what's going on.
Congrats on such a cool sponsor, keep it going man! You do inspire me, спасибо
This is so interesting. I would watch your videos every day! Thank you for sharing :)
Can't wait to see progress on this!
I've seen a few others mention this already but definitely look into Docker and with it Docker Compose. Using Docker Compose you can not only automate the process of booting all of these services at once but also save yourself work in the future when deploying individual services becomes as easy as running a simple docker command. I use Docker for work and my personal server which is set up much like your proposal, and it works beautifully. Keep up the good work though man, love seeing how others approach the same problem.
I love seeing other people working through this kind of thing. Good luck.
This was awesome. I've been pissed that I could not find a video like this sooner. I wasn't even sure where to begin.
It's amazing how I spent a full week researching this.
My goal is the same, to separate my project into multiple servers. One serves the frontend, another serves the backend. From there every application can be dedicated and/or bundled up somewhere, ideally organized to keep maintenance seamless when interacting with applications.
I'm going to look into what you recommended to do and I thank you for finding out all of this.
I would love to see your project files (excluding your applications of course) or if you made a boiler template out this, but that's a lot to ask for. Regardless. Thank you.
You are just amazing. I really love your videos man. You are my inspiration.
Keep this style of videos coming, they ROCK!
dude love this kind of video of yours no one does this except you , i get a grasp (somehow lol) on some web apps .
Where have you been?? I'm excited for this video
Same
bruh
I've checking everyday ur channel and ur videos makes my day better Thanks!!
I laughed so hard when you captioned "Some High Quality Documentation", then I know what's going to happen next. Legendary MS Paint drawings.
This is literally what I’m trying to learn rn thank you so much
Love your videos! I just finished building a web app with scalability in mind as part of a capstone project. Looks like you're taking the right approach at tackling this and I'm excited to see the progression of your solution. I'm not sure if you mentioned it but look into docker. Containerization definitely helps with scalability and easy/fast deployments. What are you using to host your server? You can look into launching each app as independent microservices on their own AWS instances which would allow you to future scale them independently as they grow. Amazon offers great pricing on server hosting as you only pay for what you use.
Btw I used the same exact tech stack as you, so if you have any questions feel free to ask!
Wow, awesome man.
I wish you did more videos, I'm starting to study Computer Science this semester and I like your videos very much! (although I hardly ever understand what you're doing) hahaha
keep doing them!
hugs from Brazil! ;)
Glad you switched the topic to programming. Looking up to more videos!
Jesus Christ Devon! What the heck happened?! You grew so much! Hope you remember me haha, your first fan you have ever met, sort of by accident. But I’m happy for you man. You grew more than I thought and I’m extremely impressed. Love the content and I’m gonna binge to catch up
- nick, that weird dude from X15
Really like these vids dude keep them coming :)
Every time I watch one of your videos I get so amped on programming and geeky computer stuff. So amped that I actually do my homework
Dude you really go crazy with that Paint workflow xD
wow! again! You know i was thinking about some programm which visualise the music from comp to a LED /RGB tape. And i thought it is easy. Bro. Love you that you showed us that none of the products are easy to do. Hope for new vids
Nice content explaining the process! More systems
Another step to make your API scalable is versioning. If you have multiple apps talking to the same endpoints, changing the endpoints (either response/request or route) without versioning would require you to update all applications to understand the changes.
With versioning, however you can create a new version of an endpoint and avoid breaking other apps the uses the older version. This allows for incremental updates without the need to break anything.
Thanks for making this video! I'm a pretty junior programmer, and I get intimidated and overwhelmed easily with all the stuff that's available out there. (Do I use restify? express? what's the difference? Apply that to all the different things out there... ) Your videos really help me remember about how programming is really fun and useful, and makes me more motivated when I get back to the cubicle again tomorrow. :)
I love when you upload these kinds of vids!!! 👍👍👍👍
Love these videos dude.
I dont understand much as Im just at the start of my CS schooling, but this kind of stuff is so interesting to see how it's actually done behind the scenes. Also 10/10 music selection at the paint part
i have no idea what hes saying but he presents all this info in such an entertaining way.
I've just completed my AWS Developer Certification, AWS does this all for you. Use an Auto scaling group with a min and max amout of instances/servers, use an Elastic Load Balancer to assist with routing and load of incoming traffic. You could also go serverless and use AWS lambda, API Gateway and Dynamo for your backend calls, then use S3 to host your static files.
Dude I'm been watching for a while and been using the same frameworks as you even before you did and now you're using next.js and react and I'm using them as well, what a coincidence lol
Devon you are doing a wonderful job by making such videos that are inspiring as well as motivating for our young programmers. You are an inspiration and motivating factor for me. Wish you best of luck for your future hope you have a wonderful future ahead. Keep up the good work!
you made me take on a whole new perspective on programming... that it should be a fun learning experience :)
Half video it's me actually understanding your code, the other half it's me looking your beautiful face, admiring how smart you are.
I don't really have any experience with what you're doing, but I think I understand the basics.
I probably don't know what I'm talking about, but I really like the idea of trying out IPFS and neo4j, no idea what I could do with them until I try though! I do really like what you're doing, maybe I should dive in and try out a bit of everything, see what I like! Thanks for insiring me to try new things! :D
Really like this type of video dude, it's nice to see someone else go through their thought process.
This seems like the best way to host on a single server. What would be the process if an app crashes or something?
I am so excited.. Please be more regular with videos. It is soooo awesome !!! Please try to be regular... Thanks !!!
love the vids, please dont stop motivating me through college :) engineering student loans are insane
Best Computer Science channel in all of UA-cam!
Awesome stuff man.
dude these vids are soo great
Dude this is not boring at all I've been trying to find a full stack app architectire but suck at this planning on my own.. Very cool I am def. Gonna try next on a new project
awesome video when you say beginning of scalability, you look like a singer.
I always learn some more in each of your video so thanks Crawford
Honestly this may make my current project work even better, but I do recommend using docker with the same setup you beautifully drew in paint.
Do you think you can post some kind of list of all the music you listen to? I enjoy all your music choices, specifically the one at 4:30, i tried to find it on soundcloud but could not, either way a playlist of all your music you use in your videos and or one of all the music you listen to in general (like a public soundcloud playlist?) would be really awesome, love all your vids!
Hi, I separate my programs and deploy them on separated docker containers, so every frontends and backends are served on their own thread and with their own resources. I'm learning how to use kubernetes, that allows to create a "pod" which is a group that has containers and you can expose that pod as service and you get a single link to consume your backend and by default you get load balancing through the containers that are part of the pod
If you want it to be even more scalable, you could create a fully static frontend (using something like GatsbyJS or react-snap for the frontend. This lets you use React while still getting some SEO built in at compile time). The problem with Next.js is that it requires you to run on an actual server with the NodeJS runtime installed, whereas if you built your app into static HTML/JS/CSS, you can serve the files from anywhere, basically for free. This makes your frontend extremely scalable and cheap to serve.
The only tradeoff is that you don't get the awesome SSR that comes from using Next.js -- however, you can still have some good SEO optimization by computing the render() functions at compile time (that's what GatsbyJS or react-snap do for you).
Anyways, great video as usual! I really enjoy watching your stuff.
Just looked more into Next.js and realized they have a static HTML export feature. You can disregard my comment :D
Yeeeeey he's back!!! Hyped :)
You should do some "what's inside my windows and linux" distributions video.. like what you prefer using for programming etc etc..
Exciting stuff !!
Oh you finally hit the microservice idea boy?
Go on with this :3 can't wait to see updates
I know bits and pieces of what you are trying to accomplish, but I too would be stuck on a few things that I don't know how to do. I look forward to seeing your journey with this project.
OMG man, I have same trouble and you make me think a lot. Thank's bro!
Hey dude how you doing with regards to your front end. If you are using react you can host it as static site that resides somewhere like an S3 bucket, you could then use a CDN to further scale the frontend if you get more requests for the client side app. That could possibly lessen the load on your server.
@Scott I agree, but also it would beneficial to leverage a cloud provider. Google Cloud has a very good Kubernetes Service if you want to go the container route. I personally would use AWS API Gateway and EC2 for scalable backend servers as well as their Application load balancers instead of Nginx for setting up routing paths. Static resources can be stored in AWS S3 and Cloudfront for CDN. Frontend can also be hosted on EC2.
I'm currently interning at a bank in Australia and what you said about the complexity is true. We also use Nginx to route our different applications/API's we build.
Sounds like a docker problem
Exactly
Any specific reason why you wanna go with server side rendering? I would go for client side rendering. Especially for the web application, which probably will need some form of rendering after the initial load. So it prevents you from always reloading the complete page when you want to fetch some new data.
SEO. If you want your SPA to rank on a search engine without various hacks, server side rendering with Next.js is painless.
Most search engines already address this issue. You can also implement caching and ld+json to fix any other issues you have in particular.
I mean, if you want to rank high on duck duck go, i guess you have no choice.
SEO is a good point, although I think you can still achieve acceptable SEO fairly easy with client-side rendering.
In the end it's up to Devon. I just wanted to question his idea, that he tires to get to the bottom of it, before deciding for good.
And before I forget :) regarding scalability (because it was like the main point of this "design meeting"):
Most of the time server-side rendering will be more performance heavy on the server. Of course this varies from framework to framework and how you structure your code.
This doesn't mean that when using client-side rendering the performance of the site (as seen by the user) will be better. Usually quite the contrary. But like the name already says: The server does all the rendering. So you might run into problems if a lot of users try to access your page at the same time, when having a weak server.
@@Anony584 SEO isn't a problem anymore for client side rendering. Google has confirmed that it waits for JS loads when indexing pages.
Awesome video Devon
Make more this kind of video
You need to make more videos brother, i'm really digging them right now
Thank you Devon, I've subscribed to Brilliant.org :D and get an annual subscription, again thank you!
If you want to build a big website or just make a small website super performant, i'd recommend next.js for the front-end (which you already use) and making the back-end using Go since it's super fast and faster than NodeJS.
You went hard in the paint, dude.... .HAM...
Hi Devon, Might I suggest using typescript if you prefer a compiled/transpiled language as you state in the description? It is a strongly typed language similar Javascript and it is transpiled into Javascript. Good luck on your coding journey!
Very informative and impressive video about how to design an architecture. Do you have any idea to use nfs or gpfs file system for your application ? If yes, what would you choose and what things you will consider to use one ?
Docker container per app + Nginx on the main machine to route to each one. If anything grows big and consumes a lot of resources, you can extract it to separate server without any efforts.
I personally have some good experiences with Azure Service Fabric. It's comparable to Docker or Kubernetes but allows you to create your applications in microservices. With these services it's much easier to make scalable applications because Service Fabric can automatically start or remove services based on your load.
This is exactly how I do it. I'd suggest vue over react . I've used react, angular, and vue and vue is really great, especially if you're new. I've never tried next.
I want more of this! :D
Didn’t even realize you crossed 100k congrats 🎉🎊
Holy Shit, same dude, last time I checked, he was at 14k subscribers WAW
Same I swear the he was at 34k like less hen a month ago
This is super helpful thanks
Hi, I from Brazil, I doing software engineering, I love watching your videos, it always inspires me to study more