When You Give Programmers a HUGE Database

The first 500 people to use my link will receive a one month free trial of Skillshare: skl.sh/conaticus12231
Discord: / discord
Github: github.com/conaticus
Twitter: / conaticus
Join this channel to get access to perks:
/ @conaticus
0:00 Project Overview
0:19 Moving CSV to Postgres
1:23 Optimising Distance Calculations
2:16 Ranking Courses
4:11 Sending Mass Data to Frontend
4:38 C# API
In this video Rami and I work on an MVP. The MVP is a university finder for college students, to give them more precise results that tailor to their preferences. There is a lot of data to manage and handle efficiently and I have documented our process of optimising this.

[@conaticus]

The first 500 people to use my link will receive a one month free trial of Skillshare: skl.sh/conaticus12231

[@phusicus_404]

Hi, is this project in production now? I'd like to actually search courses.

[@AquaQuokka]

Inaccurate, no thigh-high socks.

[@ClasicRando]

Not sure if this was considered, but the postgres COPY command would have likely made the first part of the video much easier

[@Ta9i]

or asking chat gpt to make seeding script

[@haydra9380]

i love your progress of never tried c# to start knowing it in a prev video to actually rating it the best backend lang and now; writing a whole API with it

[@karlkarlsson9699]

Instead of dijkstra you could use A*, its easier to pronounce.
But you really dont need to do pathfinding as the nooks and crannies of roads will even out and some would take the train so it wouldnt be a good comparison anyways.

[@unicodefox]

and then you end up downloading the Network Rail open data feeds to find the duration between the nearest train stations to

[@bigerrncodes]

Its just deek-struh for pronunciation sake

[@unicodefox]

@@bigerrncodes dyk-stra*

[@whannabi]

​@@bigerrncodesdjeek-strah

[@zwwz1424]

@@bigerrncodesthe way non dutch people pronounce his name is so hilarious

[@mwguy]

Just use postgis extension to postgresql for filter/sort by geo location

[@BradenJohnYoung]

Is this what qualifies as a huge database these days?

[@Laflamablanca969]

Only for the soy devs

[@olgierd245]

Waiting for more! Keep up the good work, I'm curious what comes out of it.

[@Yassinius]

2:10 You should pronounce it as "Daykstra" you silly Brit.

[@yjawhar]

Could have loaded the data into SQLite through the CLI, cleaned it there, then moved it to Postgres. Could have even left it on SQLite since it's mostly read operations taking place on the data

[@borhaneddineguemidi6402]

For a quick a dirty way to calculate the distance you could use the openstreet/osrm open source project

[@Dom-zy1qy]

2:25 What are we caching ? The data that was returned from the users query? What would the key be ?
Would we just use the request params as the cache key and cache the response to those params? Then perform the distance filtering? Havent used redis or any cache layer before so just trying to figure out what is going on here.

[@AndrewTSq]

I think there is a new API for storing files on the users harddrive now, so you could probably send more than 20MB with ease :D

[@neoplumes]

If you're going to store the database in redis, why not use persistent redis and get rid of postgress?

[@ph03n1x_dev]

you should've asked me how to pronounce "dijkstra's algorithm" 😂 it's a dutch name. great video btw❤

[@conaticus]

😂thanks, i hope dijkstra doesn't see this video

[@ph03n1x_dev]

@@conaticus i think it's impossible for him to see this.... 😂

[@baxkill]

@@ph03n1x_dev What happened to him? is he having problems with his UA-cam account?

[@Strawberry_Htet]

I also once calculated the geocoordinate. Bruteforced to map out the house numbers to lat and long😂. LoL

[@lutzmmobil]

For sending data to the frontend, why don't you just consider using a type of pagination meaning x entries per page, page offset by x * pagenumber something like that. You don't NEED too get all results at once, that would be horrible, but you also don't need to let that limit yourselves in showing more results.

[@kehan6252]

1:14 can someone please tell if this is bad code or not? It seems alright to me

[@cxarra]

If you are using your own path-finding algs for mapping, you might as well write a separate front end for it and *boom* mapping app. That is to say, you should almost certainly not take this approach and instead leverage an existing services' API such as google map's. If you really do want to go down the route of hosting and running your own mapping algs, you're going to need far more data and compute than what you're currently working with not to mention hosting it at scale will wind up costing you a TON more than just leveraging an api (which will be far more accurate in any event).

[@0xsixneufquatredeuxzero]

What is the music from 1:27?

[@Yett1hhh]

using Djnago for the backend woulda save alot IMO

[@denitechYT]

HUGE database eh Con?👀

[@fullstack_journey]

wouldn't it be easier to use elastic search.

[@delofon]

2:11 "daixtra" or smth like that

[@SteveBClark]

Why, you don't live stream it
... I love your live stream

[@ikbenwelthuis]

Please tell me you're using Data Transfer objects because I didn't see them anywhere.

[@Rami-fq4lr]

we're using result sets which contains the data we need that way we don't have to use auto mapper or map one object to another

[@ljreinworth]

conaticus upload (real)

[@johanrong]

Which countries does this include?

[@conaticus]

Just the UK for the moment! That's all the dataset provides for now sadly

[@johanrong]

@@conaticus Alright

[@re.liable]

Interesting

[@GdeVseSvobodnyeNiki]

You should check google S2 or similar lib for querying entities within the range on a map. It's crazy efficient because you store coordinates as a bigint instead of real latitude and longitude. There are shitload of underlying math, but lib pretty much abstracts you from it.

[@yungifez]

Should i build this but for Canada 😂😂

[@pakmaya1390]

Nice

[@riigel]

rebuild everything in rust 😅

[@bruhbruh0000]

nice

[@boris5937]

1min out of 5min video being an ad? RAGGGHHHHH

[@AliveOP]

W

[@Duluper]

Idk what to comment

[@ayushmishra6077]

Why are his lips so red
No gay