This week marked the arrival of Cuil on the search engine scene. Being a huge fan of search technology and how search engines work in general, I've been spending some time playing around with the new service and thought it would be valuable to expose my data on how the classic market leaders - Google, Yahoo!, Live & Ask compare to the newcomer.
When judging the value and performance of a major web search engine, there's a number of items I consider critical to the judging process. In order, these are -- relevancy, coverage, freshness, diversity and user experience. First, let's take a quick look at the overall performance of the 5 engines, then dive deeper into the methodology used and the specific criteria.
Interesting Notes from the Data:
- I'm not that surprised to see Yahoo! come out slightly ahead. Although their performance on long tail queries isn't spectacular, when you weight all of the items equally, Yahoo!'s right up there with Google. There's a reason why people haven't entirely switched over to Google, despite the far stronger "brand" they've created in search.
- Google is good across the board - again, not surprising. They're the most consistent of the engines and perform admirably in nearly every test. To my mind, despite Yahoo! eeking out a win in the numbers here, Google is still the gold standard in search.
- Ask has some clear advantages when it comes to diversity and user experience, thanks to their 3D interface, which IMO does provide some truly excellent results, particularly in the head of the demand curve.
- When it comes to index size, Yahoo! appears to have the win, but I think my test is actually a bit misleading. Although Yahoo! clearly keeps more pages on many of those domains indexed, I suspect that Google is actually both faster and broader, they simply choose to keep less in their main index (and that may actually help their relevancy results). Google's also excellent at canonicalization, an area where Yahoo! and the others all struggle in comparison.
- The biggest surprise to me? Microsoft's Live Search. I'm stunned that the quality and relevancy of Live Search is so comparatively high. I haven't done a study of this scale since 2006 or so, but the few dozen searches I run on Live each month have always produced far worse results than what I got this time around. Clearly, they're making an impact and getting better. Their biggest problem is still spam and manipulative links (which their link analysis algorithms don't seem to catch). If they fix that, I think they're on their way to top-notch relevancy.
- Cuil doesn't permit a wide variety of very standard "power" search options like site:, inurl:, intitle:, negative keywords, etc. making it fairly impossible to measure them at all on index size (though the lack of any results at all returned for terms & phrases where the other engines had hundreds or thousands speaks volumes). It also put their technical and advanced search scores in the doldrums - none of the "technorati" are likely to start using this engine, and that's an essential component of building buzz on the web Cuil's missed out on.
- Cuil was foolish to launch now. Given the buzz they had and the potential to take market share (even a fraction of a percent is worth millions), they should have had lots of people like me running lots of tests like this, showing them how clearly far behind they were from the major engines. You only get one chance to make a first impression, and theirs was spoiled. I won't predict their demise yet, but I will predict that it will be a long time before Michael Arrington or anyone in the tech or mainstream media believes their claims again without extremely compelling evidence. Their index, from what I can see, is smaller than any of the major engines and their relevancy is consistently dismal. I feel really bad for them, personally, as I had incredibly high hopes that someone could challenge Google and make search a more interesting marketplace. Oh well... Maybe next time (assuming VCs are willing to keep throwing 30+ million at the problem).
Methodology: For each of the inputs, I've run a number of searches, spread across different types of query strings. This is an area where understanding how search engine query demand works is vital to judging an engine's performance. Some engines are excellent at returning great results for the most popular queries their users run, but provide very little value in the "tail" of the demand curve. To be a great engine, you must be able to answer both.
In most instances, I've used search terms and phrases that mark different points along the query-demand scale, from the very popular search queries (like "Barack Obama" and "Photography") to long-tail query strings like ("pacific islands polytheistic cultures" and "chemical compounds formed with baking soda") and everything in between. You can see a full list of the queries I've used below each section. During the testing, I used the following scale to rate the engines' quality:
Now let's dive into the lengthy data collection process...
Relevancy
--------------------
Relevancy is defined by the core quality of the results - the more on-topic and valuable they are in fulfilling the searcher's goals and expectations, the higher the relevancy. Measuring quality is always subjective but, in my experience, even a small number of queries provides insight into the relative value of the engine's results. To collect relevancy, I simply judged the degree to which the top results resolved my inquiry, and weighted those that provided the best answers in the first few positions higher than those that had better results further down.
The following are the queries I used to judge each of the engines on performance:
- Top Buzz: gas prices, iphone, facebook, dark knight, barack obama
- Popular: laptops, photography, rental cars, scholarship, house plans
- Mid-Range: fire prevention, calendar software, snow tires, economic stimulus payment, nintendo wii games
- Long Tail: pacific islands polytheistic cultures, chemical compounds formed with baking soda, genuine buddy 50 scooter reviews, google toolbar pagerank formula, getting a novel published
- Technical: metalworking inurl:blog, cricket -site:.co.uk -site:.com.au, dark crystal site:imdb.com, top * ways, definition sycophant
Coverage
--------------------
Coverage points towards a search engine's index size and crawl speed - the bigger the index and faster the engine crawls, the more pages it can return that have relevance to each query. To judge this metric, I focused on the coverage of individual sites (both large and small) as well as queries in the tail of the demand curve.
Queries used for evaluation:
- Large Sites: site:government.hp.com, site:research.ibm.com/leem, welsh rugby site:bbc.co.uk, search engine optimization site:w3.org, tango tapas seattle site:nytimes.com
- Mid-Size Sites: site:seomoz.org/blog, site:news.ycombinator.com, site:education.com/magazine, bumbershoot site:thestranger.com, snowboards site:evogear.com
- Small Sites: site:downtownartwalk.com, site:amphl.org/, site:totebo.com, dockboard site:loadingdocksupply.com, site:microsites.audi.com/audia5/
Freshness
--------------------
Although coverage can help to indicate crawl speed and depth, freshness in results shows a keen effort by the engine to place relevant, valuable news items and other trending topics atop the results. I used a number of queries related to recent events both popular and long tail (including new pages from relatively small domains) to test the quality of freshness offered by the engine's index.
Queries used for evaluation:
- Top Buzz: los angeles earthquake, obama germany, gas prices, ted stevens, beijing olympics
- Popular Queries: new york city weather, dow jones average, seattle mariners schedule, cuil launch, nasa news
- Mid-Range Queries: warp speed engine, unesco world heritage, movie times 98115, comic con 2008, most charitable us cities
- Long Tail Queries: melinda van wingen, over the hedge comic 7/28, seomoz give it up blog, scrabulous facebook, internet startups that failed miserably
Diversity
--------------------
When search queries become ambiguous, lesser engines often struggle to provide high quality results, while those on the cutting edge can serve up much higher value by providing diversity in their results or even active suggestions about the query intent.
.gif)
Queries used for evaluation (I've only used 3 queries per level here, as more ambiguous query strings are very challenging to identify):
- Highly Ambiguous: mouse, ruby, drivers
- Moderate Ambiguity: comics, shipping, earth
- Relative Clarity: ibm, harry potter, graphic design
- Obvious Intent: seattle children's hospital map, color wheel diagram, great gatsby amazon
User Experience
--------------------
The design, interface, features, speed and inclusion of vertical results all play into the user experience. An engine that offers a unique display may rank well or poorly here, depending on the quality of the results delivered and whether the additional data provides real value. Rather than separate queries, I've judged each of the engines based on their offerings in this field (using both the data from the previous sets and my own past knowledge & experience).
.gif)
User experience was based on each of the following:
- Query Speed - the average time from hitting the search button to having a fully-loaded results page
- Results Layout - including the organization of results, ads, query options, search bar, navigation, etc.
- Vertical Inclusion - the inclusion of valuable vertical or "instant answer" style results where useful
- Query Assistance - the use of disambiguation, expansion, and similar/related queries
- Advanced Features - the ability to conduct site specific searches, search for terms only in specific URLs or titles, and narrow by website type, a given folder on a domain, etc.
For those who'd like to provide their own input about how to judge a search engine, Slate.com is running a reader contest to ask How do we know if a new search engine is any good? - I'd strongly encourage participation, as I know the audience here can contribute some excellent insight :-)
If you're interested, here's a screenshot of the Google Docs spreadsheet I created to conduct this research (and I've published the doc online here):
This kind of thing is a lot of work, and although this isn't scientifically or statistically significant, and clearly biased (as I'm the only one who did the judging), I think the results are actually fairly useful and accurate, though it would be fascinating to run public studies like this on a defensible sample size.
p.s. Want to use any of the images or content from this post? Go for it - just please provide a link back :-)
I'm not sure how "Google is still the gold standard in search" when you rank them overall lower than Yahoo. Unless of course you consider Yahoo is the Platinum standard.
Lot's of data, but very highly subjective and unscientific. Should make for a great comment thread though - drawing out the typical contrarians.
BTW - I did my own scientific testing and can vouch for Cuil as being the most highly relevant and accurate search engine as witnessed here.
Just a couple of footnotes:
Search for Danny Sullivan shows handsome picture and the "Explore by category" listings centered around Indy 500, Indy Winners, Indy Rookie of The Year, International Race of Champion Drivers & Champ Car Drivers.
I think we can all attest that Danny Sullivan is in fact handsome, moves very quickly (thus the reference to Indy), is a "winner" by most people's standards, could have been considered Rookie of the Year with his founding of SMX and Sphinn, is international (having lived in both UK and USA (not to mention the world travel), and drives a car - at least until he sold his Mini. (Not sure now though). Nevertheless - A+++ for relevance here.
Search for Rand Fishkin - delivers picture of Superb Host banner. We can agree that Rand is, in fact - a superb host. "Explore by Category" reveals the subject - "Google Employee". Although I did not know this before hand, it certainly makes a lot of sense. A+++ for relevance here.
Search for Andy Beal - reveals a plethora of "Explore by Category" references to "World American Poker Players" and such. I can attest that Andy Beal is not only an avid gambler, but also a heavy drinker and cusses like a sailor - which is why he focuses so intently on reputation management. Most people believe Andy to be a man of the highest integrity and character. Thansk to Cuil - we now know this is not the case. A+++ relevance - yet again.
Search for Michael Gray - not only delivers a number one spot to the Wolf-Howl blog, but also provides a picture of former NY governor Eliot Spitzer. An obnoxious and hypocritical womanizer, it is only logical that Cuil would pair the Spitzer photo to the Michael Gray search return. I don't know Michael Gray - but I do know Cuil, and so I can only assume - this is an A+++ relevance.
Search for Perez Hilton - Delivers a number one spot to a Wikipedia article that references the actual Perezhilton.com blog - thereby killing two birds with one stone (a number one wikipedia spot + a reference to the actual Blog url), which is something that Google is not yet sophisiticated enough to pull off.
More importantly is the number two result, which is dead on - Mahalo. The only thing missing is a picture of Jason Calacanis getting lei'd.
So there you go. My official, highly scientific and incontrovertible assessment. I give Cuil a triple AAAA+++ for search relevance and I dare anyone to prove me wrong.
The only area Cuil fell short was in a search for Jane Copland. There are several pictures of a young blonde haired lady, but Stephen Colbert is nowhere to be seen.
lol Jolly good show sean.
A vanity search for my name showed this old profile cache - showing a profile thats probably 4-5 months old. Rubbish.
A perfect example of why you should monitor your reputation. You just never know when Sean will get his hands on a computer and internet connection. ;-)
but Sean, I think you forgot someone in your scientific testing. Did you look after yourself in cuil? from what I can read, you are an English Soap Opera Actor, which explains why you are so popular in blogs and get tons of thumbs up! everyone loves actors...
"Sean Martin Michael Maguire (born 18 April 1976) is an English actor and singer of Irish descent, who rose to fame in 1988 .."
see by yourself!
;-)
LMAo...
We didn’t find any results for “paisley amoeba” Some reasons might be...
Finally, try to think of different words to describe your search.
wait.. ok.. then let's try.. "paisley does dallas"
Paisley Does Dallas Dance Music: Games Like realtors who know how the real estate market works, they deal with the ever changing landscape of search engines on a daily basis and know the ropes of what the search engines like and do not like in order to assist you to the top of the search engines, the most profitable location. Posted by Paisley Amoeba...
paisley.blogs.friendster.com/my_blo... notice the following:(HBO Family channel logo missing)
a. i wonder if HBO Family channel knows about this, because if you think the blog referenced is family friendly, go check out Urban SEO Dictionary
b. funny.. last line says "posted by Paisley Amoeba"
c. umm.. Cuil = FAIL
It all looks good and very interesting, but the speadsheet and all the detail (which I do appreciate) still doesn't answer the most important question -- why did some query get a 4 versus a 3.75 versus a 4.5 and so on.
For example, [gas prices] have both Google and Yahoo at 5, so they're perfect. Is that because Gas Buddy comes up first? If so, do the other pages that are listed not count? On Live, Gas Buddy is also first -- but it gets a 4. So what knocked it down 20%? On Ask, Gas Buddy is also first but it gets a 4.5. Again, why the difference?
Yeah - you make a fair point, but at the same time, it would be impossible to provide that level of detail for every score.
However on that particular query (and on all queries), it's not just about who's ranking first. I looked through all 10 links and what relevance and value they provided. Note that Google and Yahoo! both have links to places that show local gas prices, show trending charts, they've both got the latest news items about them, etc. The other are good, but as you scroll down, the results get worse, so it's not just about who's in first place. I tried to do this relatively holistically, albeit informally.
Don't need statistical significance to see which search engine I'll be using when Cuil is too Cuil to even rank for their own brand?
Compare
https://www.cuil.com/search?q=cuil
to
https://www.google.com/search?q=cuil
Yes, it is early days but ouch, all that hype built up around Cuil sorta bites them on the butt when the product they launch appears so inferior.
An impressive job - yep, it might be subjective but awesomely describes how search engines should be compared.
I suggest that you set up another huge poll here at SEOmoz asking people to compare the search engines - you already have poll questions! We all would be eager to se if your personal opinion coincide with the one of general SEO community.
I've been seeing reaction to this new Search Engine and the results are not good. I read on another site about the down side to cuil and tried it for myself. from my post How NOT to search the web:
"So I gave it look see to figure out if it was unique to him [Jim], or if I would have some weird results too. Turns out, it’s worse than he described. Wow, that cuil site is pretty bad. I searched for my top keywords and didn’t even show up. Then I searched for my name and got some very interesting results including other people - namely a bikini-clad super model in a provocative pose (I’m a one-piece kind of guy). For me, it sucks that my top keywords don’t result in my site showing up and its weird other people's photos are displayed with my name, but for Jim and others who depend on the internet for their livelihood (directly or indirectly) its a deal breaker."
You can't have mis-mashed results so glaringly out of sinc with the content and the searches. Its as if they delibrately set out to be black-hat and ruin everyone's reputation.
Also, being a search engine and all, they might want to look into this little thing called canonicalisation. May it please the court (sorry Sarah!):
Cuil
Cuill (notice the extra 'l')
Then again, if Yahoo can't get it right...
Hi Rand,
Great study here.
I saw this cuil first time here on your post and after reading the post did some searches. I saw that Cuil cant calculate te pagerank of the page. It shows Very different SERP than googles.
This is good! Hey, Cuil (or Cuill), have you heard of a 301 redirect?
They obviously are not worried about duplicate content issues and being indexed by Google or Yahoo!.
I'm not impressed with Cuil. The results seemed off on many searches. The thumbnail images didn't seem to serve a purpose and didn't match up MOST of the time.
Also, branded searches came up as 'not found' when the same website came up for a descriptive phrase. I don't like the magazine layout either.
The most interesting NEW search engine IMO is SearchMe.com. Seems to work great and I love the user interface. I'd like to see how SearchMe sizes up with regards to your analysis as compared with others.
Nice report! I can tell it took a lot of work.
I think Searchme.com uses google's SERP though. It just has a nice UI. Mac Style.
Great point!
I like Searchme a lot. The visual representation of the SERP engages me in the 'relevancy' based on what I see.
Rand - great collection of SE data!
Scientific or not this is the closest thing out there comparing the search engines across a range of categories. Thanks!
I commend Cuil's PR people for getting the coverage, but proclaiming yourself a "Google Killer" on your first day out of the gate does not seem like a great strategy.
What about a easing people into Cuil and letting them switch SEs gradually? A FireFox extension that let's users keep using Google but also opens a new tab with Cuil results? Or even adds a link to the results page that says "See results for your search in Cuil." I'm not technical so I don't know if this is possible.
Any hope for a blog post about online reputation management and what Cuil could do, if anything, pick their brand up off the floor?
As far as I am considered Cuil is a disappointment. I got a lot of “We didn’t find any results for…” messages while using Cuil. I agree my most of the searches contain more than 4 words, but that’s the way I search.
As far as relevancy is considered, Google rules for me. Only yesterday it happened. I was talking with a beautiful girl (OK who am I trying to impress? It was just my male friend from work.) about size of Google Index. And I remembered that in his SEOMoz blog post the ‘Evil side of Google’, Danny Dover had mentioned the size of various Google databases.
But of course I didn’t remember the exact title of the post. So just on whim I opened Cuil and typed (without quotes), “seomoz Google data Danny blog”. And what it returned,
We didn’t find any results for “seomoz Google data Danny blog”
Then I ran same search in old faithful Google, and viola! The post was right there at the first spot. That’s why I use Google.
After reading this post I just ran the same query through Yahoo and Live.
For live the PDF of that Danny posted along with blog-post got the first ranking. The url given was
www.seomoz.org/user_files/google-user-data/SEOmoz-Google-User-Data.pdf
Not what I was looking for but if you look at my search query, it contained words Seomoz, Google and data, so I think Live gave more importance to the PDF over the blog post which has URL
https://www.seomoz.org/blog/the-evil-side-of-google-exploring-googles-user-data-collection
This is just a guess. Of course other factors may be involved.
And what the hell? The post was no where to be found in top 10 results for Yahoo. First 2 results were from SEOMoz, first one was for Search Engine Ranking Factor V 2 (I think here Yahoo thought by ‘Danny’ I meant, ‘Danny Sullivan’. Sorry, Yahoo wrong guess.) and the second one was ‘The Definitive Reason Why Facebook is Worth at Least $15 Billion' - another post by Danny Dover (And I have no idea why should something about Facebook should rank at 2nd no when the search phrase gave clear indication that I was looking for something related to Google.).
Interestingly the third results was from webmasterradio.fm which gave some audio (which I hadn’t had time to hear) about,
Also this same webmasterradio.fm Page ranked second on Live. And at no. 1 spot on Hakia. So here is something to for SEOMoz team to look at, some other site is constantly ranking for a long tail search term which is obviously targeted at SEOMoz content.
Now is there any question why I prefer Google? I know it is not the most scientific way to test search engine relevancy. But then average Joe like me doesn’t have time for long analysis before deciding which search engine to choose. Past performance dictates – Go to Google and Search.
I like how if you do a Google search for the oddly-spelled "Finn McCuil" from their press releases, the #1 and #2 results are, well, not the kind of thing Cuil would probably like to have up top.
Check out Yuil, the first knockoff, already!
TechCrunch writeup: It's Yuil! Looks like Cuil, With Better Results
P.S. Sean Maguire has a better picture on Yuil vs. Cuil.
Yuil, the parody on Cuil, was shut down, cease and desist order from Cuil.
Laughing because discussion on TechCrunch was about the accuracy. Turns out results were just Yahoo! with a Cuil wrapper.
Given all the comments here, I have to wonder about how much testing the Cuil developers did prior to launch and if the marketing team did their own testing with "real folks."
Cuil seems to be a work-in-progress, perhaps forced to launch due to investors wanting a return on their investment?
It definitely seems to be a product of the kind of 'release version = enhanced beta' thinking we see so much of in the software space right now.
Unfortunately, the critical reception they've garnered right out of the gate isn't something they can fix by releasing a patch.
Cuil looks like it functions within acceptable parameters for popular queries that have next-to-no ambiguity. Which doesn't seem to Venn well with the sort of search the Google-contrarian set they want to attract (the unadorned black page says it all, no?) is likely to be performing.
I'd love to see Cuil improve, but first impressions count.
Did Cuil really make the list? I did a bunch of searches and it kept giving me a 404 not found. Truth be told, they were more niche type searches but that is how I usually rate search engines to see if they reallly spider constantly. Yahoo is a good SE but I think Google is still king because they finally gave me actual number metrics for the Keyword tool. Thank you Google, you really saved me time.
re: 404 errors or pages not found.
One SEOMoz member in Chicago got pages not found. I did the same search in California and had 2 results. We also got different listing for a different search.
It appears Cuil is giving results based off IP or pulling personal data? Are they tapping into a toolbar?
IMHO our way of searching is biased. Because I am used to google for many years (99% of the time i use it to find information) I tend to think google is more relevant.
I think a brand new google user will take one month to improve its searching skill, and take fewer and fewer time to reach its desired information after. It is not google is relevant, BUT google is relevant compared to your searching habits.
I think today so far, relevancy is paired with searching habit.
An interesting experiment to do is find people who seldom use search engine (they still exist) and ask them to find a precise piece of information with different search engines and compare the results.
You're absolutely right about the first impression. I gave Cuil a shot and got the same impression you did and I can't say I'm eager to go back. They should have done more before the launch. They may get some market share eventually but this was a huge loss. I don't think I'll try them again for a good long time. Plus the spelling of their name drives me nuts ;)
+1 to that. I tried cuil and was completely underwhelmed. Google, Yahoo and even Live provide more than enough search engine love for me (both as a user and as a site owner). Why on earth would I ever go back to use cuil again?
Interesting results, especially in regard to MSN and Ask. The Cuil numbers seem to be in line with basically everything I've seen and heard about it in the past two days.
Last night, I tried vanity searching on Cuil. I found a bunch of sites that link to me, but couldn't find me.
The good thing about Cuil is that we now that that a clean interface and fast retrieval speed aren't the only thing it takes to kill Google.
Interesting study Rand. I've been more impressed lately with Yahoo, as I've been noticing for a while that the results were often surpisingly much more relevant than Google's (although not always).
Relevance, Depth, Quality. Cuil fails all of them.
Google has spent some USD 3 Billion on datacentres in the last 18 months to two years, so 33 Million for Cuil doesn't buy you very much at all in comparison. That much is painfully clear from the slowness of the site, and frequent service outages.
Great post, thanks for the research.
In my opinion going away from the normal list approach was a big mistake for Cuil, that together with keywords not being found provided a big disappointment and was definitely something that could have been avoided!
Bottom Turn I found the same, at least my clients site is finally #1 in A search engine!
Fianlly got through the whole post. Phew. Man talk about detail. I had done my own quick review of Cuil but this one, dang Rand. Always gotta be the best huh? ;)
Some of your results suprise me, but not too much. As in almost all things we do, it's always going to depend on who you are and what you're searching for. But this was a nice look at how they compare to someone who KNOWS search engines.
I too am sad about Cuil. They learned a lesson and its a good one for all companies. If you have the backup to gain lots of PR, hype, etc. then test a million times before release. I can name at least 10 people who would have torn that thing apart and given suggestions for FREE!
Maybe the next time (if there is one) they'll learn. We aren't good at that as a society though ... so we'll see. :)
Great look at the engines here, and certainly surprising. I think it has value for others to follow your methodology and to post their own rankings, just like having 3 judges in a boxing match.
I agree, it's disappointing for a baby like Cuil to be released and to make bold claims before the time was right. I have noticed that they are updating quickly though. I find that some terms which show no results one day, have results the next day. That's even more proof that they just weren't ready yet.
Your UX ranking is slightly suspect, because I'm not sure if search suggestions pass the "mom test" yet. A search for "fun places" narrows to "fun places for kids", but it can be misleading to click on it, thinking it's a result, when all it brings is another serp. There is also no obvious way to return on Ask, if a narrowed search leads to bad results.
In my experience Yahoo! is shocking when it comes to relevancy. When I perform searches for brand or niche terms I see alot of off-topic results including malware and spam.
I was just going to mention - I personally find Yahoo to be extremely poor when it comes to relevancy.
Also, I've noticed Google clean up their results in the past few years (there used to be a lot of older results that ranked well, like forum posts, etc. that weren't really helpful) but Yahoo has made few improvements from what I can see.
I did a quick search on my keywords. Yes relevant sites came up, but the logo for my site was on someone else's, and a logo for a site nothing to do with mine was on my site. Not one of the images made sense, and it really distracts from it all.
Really? I didn't even get any relevant sites on my local/regional searches I did...
Cuil posted the wrong image for my Web site, too. Some weird illustration of two people dancing, not my logo.
I agree with you on the images. I think that destroys any relevant credibility they have. Not sure if what they are doing is even allowed. Credit is not given to the website the image comes from. It pulls the images from https://www.cuilimg.com/imgsrv?
How would anyone ever know who the image belongs to?
I did a very fast drive-by test (using my name) and a spam page filled with jibberish was on the very first page. Not very promising. Plus, no spam reporting option. Had to email the generic "feedback" address.
Like the description snippets tho...
And thanks so much, Rand, for your analysis. Very interesting stuff!
There's one factor here not taken into acount. As far as the relevancy of searches is concerned, in other countries than the US, or at least other languages than English, Ask has really poor resaults and Yahoo, although better, never beats Google. Internet is bigger than that, and results vary according to weather you use a .com, a .fr a .ar or a .ph. When we talk global, Google has little competitors, and those that matters are usually local.
I am sure the worlds got space for two brand new crappy search engines. So I have been inspired by this whole cuil thing to start my own search engine. It's gonna suck but it's gonna give you more consistant results then Cuil does.
How do you us it? just send me a private message with your serach term and I will message you back in a week or two with your results.
And you might make a few million bucks in the meantime, enough to retire somewhere offshore, I'm sure! LOL
I tried Cuil for medical terms, and it was horrible - most of the first page results were related to role playing games - not very relevant results!
Great article with good data! I also tried Cuil, but the thumbnail picture that came up next to my site had nothing to do with it. Where did it come from? Strange!
I have a website that combines all these results and produces rank based on these popularity , I think based on your comments , I should include cuil also.
Rand,excellent analysis. Did you write this before or after searching for "ridiculous search engine launches"?
I'm just starting to explore Cuil now, as i seem to be one of ten SEO's on the planet who hadn't heard much about it. I don't understand the methodology behind launching a search engine based on "relevance" when the only relevant results I get are interpretations (whose...i don't know!) of what i'm looking for. Respective search? They've got a long way to go...
Sadly I'm so out of the loop, this is the first I've heard of Cuil, but in the spirit of posting things at the end of strings where they will never be read, I'd like to put in my two cents. I like tte black background, its fresh and new and made me say "wow". But the results layout sucks and is stupid....
Although, I doubt the accuracy as in such case numbers need to be statistically significant to cover all use cases, It still a very good analysis.
I agree with a lot that is being said here.
First of all, how do you say you are going to "kill Google" the day you come out? That actually makes me laugh.
Second...their images don't match at all! I searched for a broad term and found my company logo or a screen shot of our web site on a lot of our competitors sites (seriously 6 times on 3 pages)
and Three...it is hard to get away from the list approach. Humans are creatures of habit. Converting them to a new method is very difficult. Take https://www.redzee.com/ for example. This site is a totally new way of searching with some unique and impressive ways to searching. But it's sort of like an advanced flash web site. Very impressive, but not always user friendly and practical.
Great research...I enjoyed it!
I tried a search for "comparing search engine performance" and all 10 google results were reposts of this article on different blogger sites. I did the same search on Yahoo and only one the first result came up with this article, with the rest of the results being things from academia or other sources. You can see how the two engines can vastly differ so relevance is really subjective.
I have been using both over the last few months for comparison and overall, I still prefer google because it usually finds what I'm looking for but yahoo has really improved over the past.
Anyways, your comparison is interesting nontheless.
And apparently, the founders never had to deal with cloaking during their tenure:
https://sherwoodseo.blogspot.com/2008/07/cuil-loves-cloakers.html
Thanks for putting in the work behind this study Rand.
Certainly my experience of relevancy in Cuil is pretty low and I really don't get the images they are showing next to search results. They often have very little to do with the search term entered.
Some examples of crazy results on fairly simple searches
There is a new search engine. The new visual search engines, it is only for kids.
https://www.aga-kids.com
I'm not a kid (I play one on TV), but what kind of kiddy search engine doesn't have results for "elmo"? It iterates the same 10 landing pages for any search.
I was all excited by the press on this "baby" and was looking forward to doing a weeks worth of intensive study on it. Rand beat me to it, naturally. But before I read any of this I had been poking it with a stick like a new discovery on a Pacific beach. I was finding the same type of results as Brett and Sean. Distrubing mismatch of photos and relevance.
I found articles I had written with competitors photos on them.
Totally biased research yes, but backed up with irrefutable work like Sean has done...
Well I am disappointed. But then I'm a Mariners fan so I'm used to it.
great post Rand.
One thing that would have been good to see is local search relevancy
This is an interesting study. I have a question considering that Yahoo performed better in relevance.
Are SSP(Paid Inclusion) links included or excluded in the results?
For example on the 'gas prices' Yahoo search, 'Local Gas Prices on MSN Autos' is an SSP result. In 'rental cars', 8 out of the top 10 Yahoo results are SSP. Only Hertz and Enterprise are organic results.
Although, I guess, what shows is what shows, the Yahoo relevancy and coverage are skewed by the included paid listings in my view.
holy sh!+!!!
that is a lot of work.
THANKS!
good info..
Great study! I appreciate this solid effort.
---
I am finding some peculiarities with how Cuil adds images to search results. For instance, if you search for "Denver SEO" you will see my site (Copybrighter.com) and a picture of a smiling man who I have never met and certainly have never associated with my site at all. Why is his picture getting stuck next to my result? Is this some kind of stock image triggered by the phrase "consultant"?
I found the same type of error in my own tests this morning. The deeper into the search I went the worse it got. Repeating stock type pictures with other search results that do not have anything like that on the site.
Thanks, I thought it was just me
From what I found, it will take a picture from the page.
At least it did on my website.
Well, I searched my company name and keywords for a couple of product of mine that rank. All the images it pulls up does not seem to come from my website.
I wanted to see if the image it pulled up was somehow relevant to the keyword instead. I didn't find an image search for Cuil, so I looked the keyword up on G!'s image search using the keyword. It was nowhere to be found on the first 20 pages.
The image location source for Cuil seemed to be something like this. Are they hosting their own images?
Lots of detail, and I really liked the process used. Thank you once again Rand.
But in my corner of the universe I have another reality...
Yahoo represents 10% of Google's traffic, sometimes even less. If I add up MSN + LIVE, Yahoo goes to number 3 and I don’t mean just traffic, I have the same pattern for conversions.
I get the impression that cuil does not count domain age as much as the other search engines do.
Great article Rand, but in which way do you established the point? This isn't almost clear from your lines.
did anybody allready mention that cuil does not have any https (s like secure) sites in it's index.....
very good overview Rand. thanks
Like most of us, I have also tried cuil. The first thing I have noticed, is that with one query, you could get in the results 9 links out of 10 coming from the same website on the same page.( see example ). This factor itself tells me that they are still very far away from the competition.
Love the detail, Rand. Great post. I like some of the disambiguation Cuil is doing, but I agree with you that the probably should have cleared a few things up before launching.
Great post.
I was playing with Cuil yesterday myself and I have to say I was also a little underwhelmed.
I totally agree, they should have ironed out all the creases before launching - it really has been a bit of a wasted opportunity... I guess we'll have to keep an eye on Cuil and see where they go over the next few months.
I tried it the other day and got quite excited about the prospect. Within 5 searches id decided it was crap and left.
Cuil struggles with some simple navigational queries (when the user knows the site he wants, and searches for the brand instead of typing in the URL)
Google often makes a killing out of simply replacing your browser address bar. I can't find any of my small blogs by searching for their exact domains. That's a big oversight on Cuil's part.
However, if any Cuilies are reading this, please please carry on and make a kick-ass search engine! You've got my support.
Nice work, Rand. This post is really cuil, erm, I mean, cool. (Sorry.)
I'm curious why you only used site: searches to test the converage when you stated that Cuil doesn't support site: searches. Is there no other way to test coverage?
There are alternatives, but none that are quite as useful or precise. However, certainly something to look into for further research.
Thanks :) If I had a better suggestion, I'd give it.
Great article Rand, but in which way do you established the point? This isn't almost clear from your lines.
Hmm. I think your being terribly biased to Yahoo. ...with having a Yahoo search embedded on your site...you have me suspicious.....u in cahoots with yahoo?
Plus...Cuil... although it flaunts the idea that it's 'larger' and spans the whole web....it's missing a lot of the web too.
Eh.....
And yet, he uses Google documents to create and manage his scoring models...I wonder how others would score Yahoo! versus Google and MSLive so maybe we should see how some of the other people here would rank the various sites using the same methodology that was used here...*and* post their results/opinions...