Page Ranking and Search Engines - Computerphile
Вставка
- Опубліковано 6 сер 2024
- "Google It" has passed into common language, but how does Google rank pages? Dr Max Wilson explains page rank.
How Search Engines Treat Data: • How Search Engines Tre...
Fitts's Law: • Mouse Pointers & Fitts...
Public Key Cryptography: • Public Key Cryptograph...
Man in the Middle Attacks: COMING SOON
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
He wears sunglasses indoors, so you know he's a real hacker.
Hahahahaha
As a marketing data analyst, I can tell you that despite that Google may use these complicated, multi-faceted algorithms, at the end of the day if there is a retailer that matches a given search criteria, then one of them will invariably show up on the first page.
Sure, it's logical that Google needs to point people to purchases in order to be able to sell their own ad business... but I miss the old Google (circa 2000) when if you searched for, say, "sweater," you'd get nothing but results for how sweaters are made, the history of sweaters, and how sweaters are a secret government plot.
I'm honestly afraid to search for things I'm legitimately interested anymore - not because I believe Google cares about my life, but because I'm sick of getting retargeted ads on every single website after I perform a search.
+Higgins2001 I'd say less people are writing about the history of sweaters, their manufacture, etc., and more retailers are selling them and have websites on which they may be sold.
+Higgins2001 I too miss the days when the Google search was comprehendible: I used to know that it would favor things where the words I typed were near one another in the text of the page, and if I put it in quotes it would ONLY show me quoted strings (e.g. an arbitrary choice of "toasty horse" returns "Toasty's horse rider" which is not what I typed.) Now I have to fight against it guessing what my habits, preferences, and location are. And especially that it will always point me to a new product to buy-often when I'm actually looking for information on an older model, for instance.
+Higgins2001 It could be, that a great portion of that is not google changing the algorythm, but rather more people buying stuff online on more online platforms that didn't exist back then. So while people in 2000 may have searched for "sweater" to learn about how they are made mostly, today most people might search for "sweater" because they want to buy one online. In 2000 they bought the swaeter in a RL shop. This might be a huge factor, it might be a tiny factor, I don't know, and im not a marketing data analyst.
+Jason Olshefsky Try DuckDuckGo, for the habits, preferences and location i mean. That search engine doesn't use these factors (AFAIK).
+Higgins2001 I use NoScript to make it hard for website owners to figure out who I am, so they can't tell google and have them to targeted adds. This doesn't just apply to Google.
Google knows when I click on a link in their search engine and what videos I watch on UA-cam. That already gives them a lot of information.
This is awesome :) 17 years ago if you told someone that the idea explained in this video would spawn a multibillion dollar company, I don't think they would believe you
So many Tom Scott shout-outs XD
+Stormfox there were?
+Rob Tyler In the fake web page URL's featured in the nice animations.
Bungis Albondigas Oh thanks, I'll re-look at the video
+Bungis Albondigas haha saw them, they weren't even subtle!
Stormfox seaxyc
What about images? How does a search engine determine how to rank images?
+HexerPsy I guess by filename, description on website it was found, website it was found, text on website, Metadata of file. Maybe even Color (if you search for yellow car, pictures with much yellow in the center and named car.jpg).
Stoppi Hmmm... probably...
+HexerPsy I think they sort images based on the page the image is on, the text around it and the name of the image, they also store a bunch of metadata about it.
username_unavailable
I agree
+HexerPsy further i think they search for "similar" pictures and look how often they were clicked in the search to categorize the picture and add their own meta this way to their DBs
How do search engines keep their (necessarily) high response speed if they cross check pages with the user profile? Especially for more complex checks, for example: You mentioned the search for a horse show and the conclusion from you being a CS that it's probably a present. This seems like something which just takes too long to evaluate.
+wlfbck20 The search engine probably first presents the best result it can now and makes the checks after you clicked a link in the results in background, like if it was a present and you clicked a link where you can order it, it evaluates the effectiveness of the results and the order of the results after the effect.
+wlfbck20 Most of it's computing is already done when you search for something. Google already knows where you are and if you are a CS. So the conclusion that the horse might be a present can be made really fast, because the complicated stuff (finding out wich sites are for horse presents or where you live and so on) is already computed. It's the same strategy as the "spiders". Google doesn't search the entire web everytime you search for something. They already did it, and know what to show you as soon as you search, because they can just look at their "handy lists".
+wlfbck20 To leverage caching, they probably already categorized your profile as one of many similar profiles ahead of time. From there, it just takes one person in that group (of millions?) to take the latency hit of caching it. In other words, when you search, they may not uniquely cross check with only your profile, but a representative profile that your profile contributed towards.
+wlfbck20
They have auto complete right :)
so its processing the search before you click the button.
keep these videos coming, really interesting subject and this guy explains it really well
You could use an evolutionary approach to balancing the different weighting factors.
Have 100 servers running versions of your algorithm with different weightings, each server getting 1% of all your search engine's total traffic, then mate the ones which get the top results the most often to produce the next generation.
Before Google started getting big, I really liked Dogpile. It was a metasearch engine. It searched using other search engines and compiled the results.
The way this video described search engines in the first part was really interesting (no new information for me, but was presented in a really interesting way).
I like the shoutout to Tom Scott there ;) He definitely deserves it!
Thanks. Very interesting video and well explained. Please keep making them about various topics you only touched : relevance, what methods were used in the past, why a few methods maybe work great in small scale but not in large scale (of users) etc.
It's hard to dislike a guy who's infatuated with horses :D
Would be keen to see a discussion on speed of searches and image searches (as mentioned in the comments below)
Great video, thank you ! Are eigenvectors used ? If so, please could you make a video about it ?
I realistically would not mind knowing the full 60 year history. by the way thanks for the time you already have invested
Very happy with the amount of father ted in these videos
so once you build a graph with webpages and their ranks, what can you use to make a conclusion out of the graph ?
I recommend you check out the case studies around organic SERP CTR. I've seen a lot of sites move from the 20s up to first page with SerpClix.
You said there were so many other things you couldn't cover and that we should leave a comment about the type of things we want to hear about. My request is for ALL of the things you haven't discussed. If this takes 100 videos then I'm happy with that.
I would like to learn more!
I click the first link after a search, if I don't find the content I was looking for, I click the second one, and so on. Do they (search engines) also measure this stuff?
how do they measure relevance?
Would a ranking system (user driven) work for positioning, or do they expect users to just fake their scoring?
bring us more information! =)
+RandomVidsFrapsUser Yes, that is measured too. If for example you click on a page and then return to the search results by pressing the "back" button or by searching for the same keywords again, they track how long you spent on the page that they sent you to, and based on the time spent they'll know whether the page they served you was satisfactory or not.
You're bringing back Tom Scott for another video YAY!
There is a feature that google deployed which can match an image to similar images around the web. I'd love an explanation on how it works.
A thing I'd love to know is this : is it possible to do this backwards ? Let me explain : here, we're talking about searching for term X and Y in a set of documents and finding which documents are the most relevant ones.
But, say I'm reading a new document, what kind of algorithm can I use to know what it is about, and have it put in the same place with other documents that are about the same topic ? TF-IDF only gives scores for individual words (as far as I know), and most often, a document's subject can not be only described with a unique word (think about news articles, for instance)...
FUN FACT: "Page Rank" is actually not named as such because is about webpages. It was named after founder Larry Page
How about Map Reduce is used to produce part of the index?
is there a form of evolving search engine?
I'd like to know more about how these massive indexes are actually stored on the computer.
Isn't there a meta search engine out there which puts a search to multiple engines and then compares the results combining the results?
SEO is serious business. Web development is fun.
When I search, I'll typically click have a look and either I like website #1 press back after say time T1. Then I'll click on a second link, read that and so on; either to continue for say 4 or 5 pages(links from my search) or maybe stop there after the second link.
Q. Does the length of time I stay on each opened link get measured and account for which website is better for me? I.E. if I stay on link 1 for 10sec but link 2 for 5min, then I refine my search is link 2 promoted as such?
Q. why oh why did they stop google desktop, it was brilliant in it's days...
Q. how can an organisation (don't know if it's the right word) measure the popularity of google vs Yahoo vs Bing and so on? Can we see how google always stay at the top or maybe one day fall from existence...
Can you talk about the.. The conflict between commercial interest (like advertisers wanting to be on the first page at any cost) and Major search engines wanting to not be broken...
Would that mean that a forums section with external links takes away from a site's reputation? Or does user content not matter to google?
I want to see more SEO related videos!
Now that people ask questions rather than keywords, I wonder how the outputs of a search engines differ.
Talk about the "Deep Web"
+GuildOfCalamity Yes please
That's pretty much the pages Google doesn't index
3:38 to hell with the topics in the video , time to read all of this :D
I'd like to learn about how online javascript programs such as games are found by Google. They don't contain much text except for their title so how does Google know that they are relevant?
I wonder if there are some static weights over certain domains, like if i search for "Horse" i will get Wikipedia as one of the first searches. Or if it is just what was explained in the video?
+Crozix That's a really good point. They probably do weight the Wikipedia domain as a whole because it's such an incredibly popular site. So it might be that for single word searches, or phrase-noun searches, that the domain's weight is increased on the assumption that information on the subject is being sought. For example "Horse grooming" probably won't return Wikipedia, while "Horse Breeds" might.
+Crozix If I understand you correctly, then yes, these static weights are exatly what they were explaining. It's the link and get linked to thing. Many sites link to Wikipedia, and it is in the top 5 most visited websites. That gives it an enormous static weight. Plus, many people who search for single words (nouns) want to go to wikipedia to learn about it. Many people know what wikipedia is, so if an unknown website comes up first, explaining what a horse is, then many users will probably still click on the wiki link, because they know wiki explains stuff. This process makes it so the unknown website on top falls below wikipedia, which is what more people want. Popular websites like Wikipedia and UA-cam often show up because they get clicked alot by searches and because of their big static weight. Althought that static weight of yours is probably always changing (in small amounts). Sorry for wall of text, thanks for reading anyway;)
Stoppi That makes sense. I think the original question was about the weighting of the entire domain. For example a very very obscure wikipedia page might not have a high page ranking; it might have more links to sources than there are links to itself. But because the wikipedia domain is popular the page is ranked higher. I think your explanation is good though, and definitely still applies.
gizmoguyar Ah, I see. Well that is probably taken into account, too, yes. But another thing is, a very obscure wiki page only shows up if the search is very obscure, and therefore might not have many results, and certanly no high ranked ones. So i guess if you search for underground stuff, it is probably even more likely to get a wiki result. Also because most people know what a horse is, but maybe not so much know what Epizootic lymphangitis is, so for that search term, more people who search it want to know what it is. (I totally searched google and wikipedia for that example :D)
It is a graet video .Thank for your post video
My lovely Horse? Nice Father Ted reference!
Does semantic web actually matter?
+linkviii It does if you care about people using screen readers as well as developers' sanity.
+linkviii It makes it so much easier to read when you develop it
+Haniff Diiltiown
how are matrices used in the page rank ?
+vandos1 thankss I'm trying to learn for uni aswell want to make a search engine for a project lol
This guy really seems to be nervous or on too much sugar :-D
+pcfreak1992 he was just distracted thinking about horses
+Ben Rowe My Lovely Horse sounds a bit fishy...
+Anston Music i'll just fetch my stool.
+pcfreak1992 This guy's fantastic. He has the clearest expression of any of the computerphile guys.
+pcfreak1992 Typical Computer Science guy while not being infront of a Screen and having to talk to RL people
Pagerank is so old, its evolved into something much more as social media cannibalizes the link graph and the signals became spammed by link buyers
thing to note here is that Yahoo uses Bing internally
+Nikolaj Lepka do you have any proof for that? As far as I know Yahoo was there before Bing came along..
+Nikolaj Lepka
Hi, are you sure? Do you have any references about this subject? In my testes, yahoo and bing have different responses for the same search.
Do a quick google search about yahoo and bing.
It's something about Yahoo making a deal with Microsoft a few years ago to use bing as its back-end search technology
+Jonas Heinrich
Nice! Thanks. We developed an anti-plagiarism app and this might help to improve performance, since Bing and Yahoo has the same engine, in theory.
***** how would this information help performance?
Thanks, why not get some great SEO tools from *FollowingLike*?
It must be infuriatingly difficult to boost the right sites for a person who for example is generally curious but wants to get cold hard facts, really high quality scientifical level information and educational content on a topic that's fairly common (or the search words are fairly common), and there exists a really fantastic individual maintained expert site that is barely known in mainstream, but has all the possible info you would ever want about the subject. Internet used to be full of those sites.
Tom Scott FTW!
I don't know what they've (Google) done, but the relevancy of my searches have only went downhill since 2010.
how does that ranking system work for websites that are useful because they link to so many things, like google for example. google links to WAY more things than it gets linked to so shouldn't it be a very unimportant website?
+cubedude76 Yes. But you are not going to search for "Google" on Google. If you search for "Google" on Bing, for example, Google will be at the top of the list because of the page rank for the specific term of "google", it has the most authority, namely the keyword in the URL and other people linking to it as Google.
How was this type of research used 60 years ago?
Cool. father ted reference?
Sooo, just a question to throw out there. How do you guys think Google uses its own DNS to improve the search engine? (If at all) And is it even possible?
+MrRolnicek As far as I understand DNS and Google (not as much as you probably), their DNS is helping the spiders with locating websites and contents, but I can't really see how a DNS could improve the quality of search results. Because it doesn't read content nor why users (who use the DNS) are there. He sees WHAT they visit, but google sees that anyway, even if you don't use their DNS. Does this help?
Stoppi I guess, but how does google see what they visit "anyway"? I mean sure google knows what anyone who uses their DNS wants to visit, but unles they go there from actual google and you dont use their DNS then they wouldn't know?
MrRolnicek
Oh, I meant when you visit after a google request, cause that's probably what mostly matters for the rating of sites. But apart from that, and maybe more for adwords (google ads) and less for the search, they track you over trackers (I don't know about that, but the ghosterything detects stuff), and over google analytics, which is a service on basicly every site, they know who visitid what when from where and so on.
I read this a "Rage Pranking" for some reason. I think I'm dyslexic.
So what do you guise use? Seeks? Yacy? Starpage? DuckDuckGo?
DuckDuckGo primarily > google if I don't get the hits I want > bing if I manage to sift through every result google throws at me.
+Suvi-Tuuli Allan DuckDuckGo with a browser plugin that provides a "Search on Google" button ;)
+Suvi-Tuuli Allan DuckDuckGo & if that fails then the same search with !G at the end, DuckDuckGo's way of allowing you to do a search using Google, or other search engines using different letters after the '!'.
Neat, yeah. I hear that Bing is good for pr0n.
Woho, DuckDuckGo! :)
Bayesian Networks and other probability-based methods...
Is "My Lovely Horse" a Father Ted reference?
I always thought spiders are called crawlers. Is that simply another name, or is there actually a difference?
nice advertising for Tom Scott
spider... is that an English/British thing?.. because I have always just called them crawlers
LAST!
(at the time of posting)
My Lovely Horse!
That panda looks fittingly evil. What is Google getting when it obfuscates search results under mile long internal proxy links? Then, when that result is followed, a google site must be called up first, to receive a redirect to the real site, which takes a while if a new secure connection needs to be established or if the google server is busy.
hi guys, greetings from Chile! land of the earthquakes and tasty wine!!
+zatoichiMiyamoto you forgot about copper that makes most of our computers work :p
+zatoichiMiyamoto your earthquakes have pathetic ground acceleration.
So a small family owned plumbing company with limited SEO knowledge and limited budget could not realistically compete with large scale company competitors in the area who hire a team to manage their website everyday using all the Goolge SEO tactics.
Is Siri every going to be any good? (Natural Language question)
1:23 "my lovely lovely lovely horse"???
Did you say 60 years or 16 years of research. It sounds like 60.
Yahoo is powered by Bing.
Please slow down and speak more clearly. You have things of value to convey.
Question is about wanting this, but $_Gets this..
He bounces when he talks.
When a spider crawls the web, why doesn't the server it's requesting the code from recognize it as a bot and refuse it's request?
+Troy Adams well, it can and some probably do, but considering how much money traffic from search engines generates it'd be counter-productive for most.
They can and sometimes do. Also you can add a "robots.txt" file that tells spiders not to look at some things. However most sites want spiders to index it. Indeed the robots.txt can be used to add info that makes the spiders job easer.
+? the Platypus I've never thought the spider as a anthropomorphic computer program. Until now...
+Troy Adams: Most of the time (if we're not talking about deep web) one want's very much to be well indexed by a crawler. Websites are meant to be visited, read and interacted upon by users and what better way to have them on your website, than to let people know about you by web-searching?
*A random ordinal number*
+Kevin Fontanari Six thousand nine hundred forty-third
sharada reddy so to special me
I am your my best friend forevear
You guys should talk about the deep web.
Why does this guy bounce so much? Meth, coke, amphetamines?
Good that DuckDuckGo popped up there.
And nice plug for +TomScott
He has done it again according to the BBC ;)
The way this guy wears his sunglasses annoys me.
ayyyyyyyyyyyyyyyyyy
+Ted Chirvasiu macarena
+Ted Chirvasiu lmaoooooo
real hacker
his inability to sit still drives me nuts
+teekanne15 Computer guys aren't the most relaxed when we're trying to eloquently explain something. If I have to debug or refactor something by explaining to myself how it's supposed to work I have to get up and walk around. Maybe the brain is screaming for more oxygen.
Second!
SWARG
I am dissapointed that www.pony-horse.horse does not exist.
Markov CHAAAAAIN
obsessed with Horses smh.
Thanks for the cool vids on search engines Computerphile. I wanna know more! Can you continue the series on Google searches pls? Wanna know more on searching different languages, images, how to distinct adult content pages from regular ones (if you only spider code), personal data policy, etcetc.. This is perhaps the most important video series on Computerphile as we are all affected by Google and use it on a daily basis. Big shout out!
First!
+deldrinov ugh...
+VishnuM I know, but it was just 15s
+VishnuM if you would stop being annoyed by "first" sayers, they would go away. Just saying...
As an SEO this video is irritatingly out of date.
You're all living in a bubble.
+David S. Look at this guy. :D
keep these videos coming, really interesting subject and this guy explains it really well
SWARG
.