Just in time for Bing launch, one of Microsoft engineers has built a "Blind search" engine, which allows to compare search results from Google, Yahoo! and Bing without branding (and without all helpers, like "universal" aka News/Video/Images, or spelling correction, unfortunately). The concept isn't new, but I don't know other public implementations.
So, I first tried to enter a few common test queries - like+ vanity and navigational queries. Thankfully, all engines are pretty good at that already, and I was getting pretty even results. So, I have tried real stuff - the queries from my Web History, which stores what I was searching for. It will give some insight into my search habits, but that's side effect :).
So, here are the queries. I describe in detail only difficult queries, on which it was possible to differentiate the engines. Easy queries, on which all engines performed well, are provided below. Important: all the results are taken on June 13, 2009; if you try to run the queries later, the results might be different.
- [sign up for wave] - the purpose of this query is to find a Google Wave sign-up form for developers. Google shows the form on top position, and other results are generally on-topic (about Google Wave), while Yahoo! and Bing show stuff about Wave church and other offtopic. Bing shows blog entry that has a link to actual sign up form on pos. 5. Win for Google.
- [10 minute wave] - I was searching for a shortened to 10 minutes version of Google Wave IO session. Google shows the video on top position, and second result is embedding the same video. Bing and Yahoo! show complete offtopic. Win for Google.
- [zoho ceo about microsoft] - there is a famous post written by Zoho CEO, "Microsoft Silverlight vs Google Wave: Why Karma Matters"; I was expecting to find it with this query. Google shows the result on top position, Yahoo! on pos. 4, nothing on Bing. Win for Google.
- [pizza casa zurich] - searching for the page of "Pizza Casa" restaurant in Zurich. www.pizzacasa.ch in top pos. on Google, and not in top 10 on other engines. Win for Google.
- [media barcamp] - I was searching for the web page of Media Barcamp that happened recently in Lithuania (which is www.mediabarcamp.com). None of search engine in blind search shows it, so it is a rare bad Draw. Interesting though to notice that it is only because blind search shows US-served results; searching from Switzerland on Google shows the site in top-10, and searching from Belarus or Lithuania (or with Belarusian/Russian UI anywhere) shows it in top position; European Bing and Yahoo! still don't find it. Still, I would record this as a draw, since we are comparing blind search results.
- [adobe browser shots] - I wanted to test a new service from Adobe that allows to see website's screenshots from different browsers; but I didn't know the exact name of the service (which is Adobe BrowserLab with official website at browserlab.adobe.com). Yahoo! is the only one that showed the official webpage in top 10; all results of Google were helpful and ontopic (and snippets showed the name of the service), while Bing had useful snippet only on pos.3, with other results being completely offtopic. Win for Yahoo!
- [russian google blog] - I was really surprised to see the results for this query. I needed an official blog of Google Russia; I got it on pos. 2 on Google, and nowhere on Bing or Yahoo!. Win for Google.
- [disabling plugins in chrome] - I was wondering if I could disable some nasty plugin in Chrome which previews media files in a tab, instead of downloading them. As it turned out, there was no way to do this. Google showed in top position the open issue in Chromium bugtracker, which says in snippet that I can't disable plugins. The same snippet is at pos. 3 on Yahoo! and pos. 5 on Bing. Top result on Bing is offtopic, and top result on Yahoo! leads to the same question on support forum, but unanswered. Slight win for Google.
- [zurich card] - I was searching for information about ZürichCARD, which is a discount and travel card for Zurich visitors. Google shows official website of the card on top position; Yahoo! shows it on position 3, and Bing thinks I am speaking about insurance, credit or phone cards. Win for Google.
- [фидо twit] - I was checking if I remember correctly how to write "twit", which was analogue of "ban" in Fidonet (фидо = Fido in Russian). Google shows all 8 results about twitting in Fidonet/BBS, including encyclopedias and references on positions 6 and 7; Yahoo! shows topical answers with useful snippets on positions 1, 3, 5, 7; also, Google and Yahoo! returned results in Russian, which is correct for this query. Bing! shows 10 English-language results, which are completely offtopic (mostly swarmed with Twitter-related stuff). Slight win for Google, complete loss for Bing.
- [estonians on twitter] - I wanted to check if it is true that Twitter is not popular in Estonia. Bing! shows TIME magazine results on top which has nothing to do with Twitter, and then shows twitter.com/evgenymorozov, who is not Estonian in any way. All engines show at least one Estonian account on twitter. I don't know what would be the best canonical answer to the query; but among the links I see in blind search, only http://wefollow.com/tag/estonia is useful - it reveals the list of 78 Twitter users with >12000 followers who associated with Estonia tag . This link was shown only on Google, position 7. Win for Google.
- [женские голоса] - it is "female voices" in Russian; I wanted to find out what female singing voices exist. Top two results on Google speak exactly about voice ranges and their names; while both Bing and Yahoo! show me no answer (and quite high amount of commercial spam for such non-commercial query). Win for Google. Btw, I tried also Yandex for this Russian-language query, and it is quite good - good results are on positions 1 and 8.
- [пять диезов] - this is the name of Belarusian musical band "#####" (5 diez) in Russian. All search engines show some pages that mention the band; but only Yahoo! and Google show the link to the official website www.5diez.com. Google shows it on pos. 6, Yahoo! shows it on pos. 7. Slight win for Google. By the way, Yandex shows it on position 2.
- [5 диезов] - slight varition of the previous query. Google shows link to the official website on top pos., Bing shows it on pos. 4, Yahoo! doesn't show it. Clear win for Google. (For Yandex, it is also on top)
- [Warholl] - I was searching for info on Andy Warhol, but managed to misspell his name. Here we can see how deficient is representation of blind search - it hides "Did you mean: warhol" link that appears on Google results page. So, Google shows results mostly about people named Warholl, and link to the Andy Warhol museum on pos. 7; Yahoo! has the same problem (and the same spelling suggestion), but no museum page. Bing not only suggests spell correction, but auto-corrects and shows only results about Warhol (including Wikipedia and museum page as top results). I count this as a win for Bing since it founds what I was looking for better, but with a disgust - since Bing interferes with my query formulation and makes it almost impossible to search for actual Warholl (["Warholl"] and [+Warholl] don't help, only [warholl -warhol] allowed me to get rid of Warhol).
- [air france flight] - I wanted to find out the latest info about flight missing in Atlantic. Yahoo! shows offtopic apart from pos.2 where Wikipedia result shows up; Bing shows offtopic apart from pos.1 where old news (from 01 June) about the flight 447 is shown; Google has offtopic on pos. 1 and 3, but other positions are taken by news from Jun 3 and Jun 4. Google and Yahoo! show news universal on top, and Yahoo! also shows related search suggestion for [air france flight 447]. Win for Google for understanding my query intention.
- [cosenza networks] - after skimming through a post in my Google Reader about a map of Social Networks popularity in the world, created by Cosenza, I wanted to find this map. Google shows it in top position, While other engines don't even find vincos.it, official blog of the author. Clear win for Google.
- [амніставаны пашкевіч] - query about recent event in Belarusian; Google shows links to good news posts on pos. 1, 3; Yahoo! shows news from nn.by on pos. 2, and Bing shows the link to the overview page on buntby.com on pos. 1 (from which you need to click through to get to the news item, which is actually a copy from nn.by). Win for Google.
Some queries from my web history where search engines got similarly good results: [canton zurich tourism], [gm delisted], [Елена Бэсеску], [thalys], [create syndicated in livejournal], [google ion], [cagliari belarusian], [snowman unicode character].
By the way, since the Blind Search is created by a Microsoft engineer, I can guess they might be collecting submitted queries and use this data to improve their search engine quality. So may be this queryset will give different results after some time :).
You are welcome to give your own examples of difficult queries in comments.
You are welcome to give your own examples of difficult queries in comments.
![[info]](https://lh3.googleusercontent.com/blogger_img_proxy/AEn0k_sNwr_v4FDfTao5kwJk8GF7yVzb1aTstfWfbNtMVSDFl9V4RyJSelPb5GNSqqwJHs1jm2M6AbLgFq2YBI1SzMgZ8G1yMJ_Pp5Hl3r1AjUSX5w=s0-d){kind=small}
I salute the effort, but you are slightly biased from the start.
The wave query is a classic example. From the query - it has nothing to do with google wave. The query was about oceans. All 3 engines blew the first 3 queries.
My test: 6 for G, 4 for Bing, and 5 for yahoo - rest were draws.
to Anonymous:
I was biased, but only in sense of doing my queries in real life - and when I issued them, it was days before the idea of this post, I never thought about comparing the engines - I was searching for something. Formulating [sign up for wave] was easier than [sign up for google wave] - and, good for me, Google saved me from necessity of the second try. And oceans have nothing to do with signing up, so they clearly don't answer the query.
As for the funny point, [sign up for google wave] also doesn't show the sign up link on neither Yahoo! nor Bing. And Yahoo! still manages to show 3 results that have nothing to do with Google Wave.
As for your test, I would appreciate if you have shown actual queries, and explanation of the intent behind them.
New finding: entering [russian google blog] into Bing's German edition, you get porn spam on positions 3 and 4.
Post a Comment