This week, Rand takes it back to basics and discusses how search engines deal with the COLOSSAL amounts of data they have to sift through for every single search query. Sometimes it's actually quite helpful to step back and remember how everything works.
And just because I like to keep it real (it's how I roll), an old friend makes a cameo in this video. He's poppin', lockin' and bustin' the freshest moves this side of the Mississippi. Caaaan youuuuuu dig it?!!
Happy Friday, Gang.
By: Scott Willoughby
Whiteboard Friday - How Search Engines Work
Whiteboard Friday
The author's views are entirely his or her own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.
I've got a suggestion for your Whiteboard Fridays, guys. Why don't you end each clip with all funny unsuccessful takes where someone screws (read: Rand) the scene (like the ones you see at the end of Jackie Chan's movies)
:)
gasp! Such poor history - Washington had no beard! And there was no white house, nor any president in 1776, and no interweb! I want my money back!
Should have said "chopless" (gotta love those GW chops), and 1789 - that's when Whitehouse.gov was actually registered through ICANN :)
BTW - If you had to pay to watch, Google Video just pulled a fast one on you...
I was a little worried that this WB Friday would be too boring and too basic, but I think it actually worked out OK. Probably one of those good videos to send to your Silicon Valley friend who thinks he should build his own web search engine :)
Really nice work on the editing as always, Scott.
Returning to the basics now and again sure is important. 0r ...at least if I've received the quality messages in LAST wbf, going back to the basics is approximately as important as recent reports and quality tools ;)
It will bring some new minds into your site from YouTube. Now are you marketing the video this way? If not then perhaps... Well. I bet you are.
(WARNING: RARE NEGATIVE FEEDBACK WARNING.)
There isn't a lot of content this week for the advanced readers, bounce rates will increase (JK about bnc rt) . =/
[/warning]
Fred - don't worry, I'm very self-conscious about it. I'll try to make next week a bit more advanced - already have a draft of something decent on that front.
Besides that, the Compete interview was fairly solid stuff, IMO. Nothing revolutionary, but it gives you a really good tool for when clients or bosses or friends bring up competitive metrics.
Hey, rand! what's with latte bloat?!
Good video. Not everyone who comes to seoMOZ is as advanced and some of us so it is good to have a little something for everyone.
Nice shirt too.
“If you can’t explain it simply, you don’t understand it well enough.” Einstein
Back to basics is great, reminds us all of where things start, before getting way too complicated...
One thing I never quite understood about the mechanics of Search Engines was if they really did the "math" on-the-fly or not: when you search for "cookie" do they query the database and do the "math" and then return the results, or are these things done already offline and the 0.21 seconds it takes to get the result is just the time it takes to find the search that was already stored before. Any ideas?
Oh, and they (Google at least) only give 1.000 results for each query, which I also find interesting and relevant.
Is it possible to store all variations? e.g. (0.22 sec)
I would also like to hear some tech opinions here.
Going back to the basics isn't so bad after all. I am thinking especially about everyone who has found your website and has been starting to work into the SEO world (or who wants to know more about SEO).
Of course, we have plenty of seasonned SEO posting and writing on your website, but not all of us have 5 years or more of experience in that field, which is my case.
I liked the video and the dancing robot (especially on Daft Punk as a French man).
Geez, whoever suggested the initial "How search engines work" explanation sure is crass.
*looks around nervously*
I'm digging the Catcher in the Rye reference. It's not crass; it's a literary allusion.
Perfect Video to Understand the How Search Engine Works,
A new keyword for SEOMOZ = "latte bloat", but we need to beat https://www.uwm.edu/People/twitmer/Latte-Bloat.html! :)
Let me know where I can get next/previous videos on WBF!
Thanks
Aj
https://www.google.com/ig/directory?q=seomoz&root=%2Fig&dpos=top&url=hosting.gmodules.com/ig/gadgets/file/104462838893004134091/seomoz.xml
you can have all 40 or 50 of em on your igoogle webpage with this gadget.
Enjoy!
Thanks!
I don't think you should be self-concious about returning to the basics. Elements of search have changed greatly with time. And talking about the premise of what a search engine does is good way get into, what doesn't it do or what might it do in the near future.
Nice. The amount of data poor google bot has to wade through is astounding! Check this post on just how much data google processes per day, and how they use their map reduce middleware to distribute all the number crunching across their many clusters of servers!!
oh, and yeah. Technologic - Daft Punk. excellent Choice!
Speaking of which...
I know this is a bit off topic (I'm just coming off of five hours of link research, so I hope you'll forgive me), but if you haven't seen this or this, you probably should.
https://www.youtube.com/watch?v=lLYD_-A_X5E&feature=related
this one is pretty sweet too...with 2 bodies instead of 2 hands!
Great video. I love it. My english is very poor but I understand the basic. Thanks.
Djacir Porto
https://www.megavantagem.com.br
Search Engine technology blows my mind. I love it!
Do you plan on embedding this stuff in the guides, like the Beginner's Guide?
It's a nice "break" from reading and it'd be cool to have chapter recaps in video.
I have to admit, I took that bait….. Judging by the SERPs, I think latte bloat needs a little more time to marinate in the minds of the general public ;o) On a more focused note, I would like to comment as a “newbie” to SEO this video was truly helpful. The breakdown was clear and concise. Thank you!
Very Good Video. Thanks!
Abs,
Barcia
Isn't Trustrank a part of the Pagerank ?
They're certainly trying to measure very different things, even though the TR paper definitely discusses leveraging some PageRank stuff.
Hey Rand,
Great video. I wonder how many people searched https://www.google.com/search?hl=en&q=latte+bloat I learned so much from this quick whiteboard friday.
Thx 4 the info,
Catto
That's going to be my new answer to clients.....
"FUCKED IF I KNOW"
Great video today guys - I'm sure the lack of techno jargon will help a lot of people understand just how search works behind the curtain.
If anyone wants a more in-depth look I suggest watching this video - an oldie but goodie: UW Video
There is also some good information to be found in patent filings...a great resource would be SEObytheSea
Happy Friday all!
get down googlebot!
very informative WBF and Rand got to drop a bomb... which was in fact very funny, actually.
Nice and Simple. I like the advanced Whiteboard Fridays but it is nice to get back to basics some times. A good little "remember how we all got here moment."
Rand, getting a hold of the basics, creates a great foundation later.
But, question for you...
Concerning user data under the query independent portion, are you suggesting that clickthrough and pageviews and time spent on site can count in the calculations during the query independent timeline since they have all that information stored and will calculate that along with the Pagerank and Trust rank?
and is it peetabyte or pehtabyte?
I always said peh-ta, heeh.
Have a good weekend y'all...
I'm actually not sure about the pronunciation of petabyte, but I like to think data that size is very tasty, like a pita!
I wouldn't go so far as to say that any search engine is using the clickthrough rates, or time on site, etc. for sure - as some of the engineers have noted, it's potentially very messy data. However, it certainly could be used and if so, some of it is query-dependent (like geography or the searcher's previous history) and some of it is query-independent (like time-on-site or CTR).
Are you saying that at the time of a user query the only thing the engine actually searches for is the data record for that query which already contains the results?
That was great, but I've got a couple of things to bring up.
In the crawling/indexing process, you mention that the SE has to gather information on pages, text, and links. Don't they also need to track the elements in which the words appear?
Second: you mention two-word phrases getting their own tables. This is new to me. I agree that if there were a "Ron Paul" table it would be faster to retrieve those pages that contain the exact phrase than it would be to find pages containing "Ron" and pages containing "Paul" and then finding which of those pages appeared in both lists... but not that much faster. Besides, if I search for [Ron Paul] the search engine should be returning documents that are relevant to "ron" "paul" and/or "ron paul". As such, the "Ron Paul" table really isn't going to provide any computational advantage except on searches for an exact phrase with the words in a particular order.
Or am I missing something here?
Bob - There's really no way for us to know for sure, but my understanding is that it's worthwhile to take many popular 2 and 3 word phrases and build tables with URLs that only contain both to help reduce the computational expensiveness of the query.
Thanks! I've been training clients to be able to handle some of their own SEO, and this would be a great orientation that could save a lot of time. This video is money in the bank! And they will fall in love with the Google Bot!
So, would "query-dependent" be a way of thinking how the search engines process many long tail terms?
...I believe those who are lactose intolerant can avoid latte bloat by using soy! Too funny!
That was awesome! Why, because of the little robot and music!
Ok seriously, I think it's wonderful to have a well put together resource like this next time someone asks me the question "How Search Engines Work" or the other times where your thinking if this person really understood how these things worked it might help out a bunch.
Great video- as it was already mentionend - some of us are not as advanced as other and try to keep up, but haven't graduated beyond 101, yet. Thanks for remembering that.