The Cochrane Schizophrenia Group

Is Google a reliable tool for trial searches?

on July 12, 2013

By Beth Adams, Work Experience at CSzG

There are around 91 million Google searches done across the world each day. So it’s needless to say it’s a huge part of day to day life, as well as the “go to” place for information. But when we search, can we just presume we are seeing all the possible results?

English: Google Logo officially released on Ma...

English: Google Logo officially released on May 2010 (Photo credit: Wikipedia)

To test Google’s reliability, we set out to investigate the yield of search terms across all Google search engines.

We selected 11 phrases, chosen for relevance, currency, and possible controversial connotations including, ‘randomized’ and ‘randomized and schizophrenia’ for the theme of randomized trials, to have both a general and specific search. Then, we searched each phrase on the 137 Google sites across the world on the same day (7th – 8th July 2013). This allowed us to record the number of results on each of the sites to work out the comparative percentages.  You’d expect them to be consistent and reliable, but were they?

Although there was a great deal of consistency in the results for some search topics and countries,there was a great deal of puzzling variation for others. For example, one of the phrases we searched was ‘abortion’, and almost every Google site stayed mainly in the 96 – 98 % region, but the Latvian and Czech site (google.lv, google.ch) had only 24%. With almost every site there was this type of variation. With ‘Islam’, the majority had about 29% but curiously France and Britain had a mere 4% and the Bahamas were streaks ahead with 100%.  The phrase, ‘randomized and schizophrenia’ had a surprising amount of variation, although many of the values stayed around 13 – 17%, there was a fair amount of change, which was quite unusual.

Google Master Plan (frame 3)

Google Master Plan (frame 3) (Photo credit: jurvetson)

We also decided to test the reliability of searches over time by repeating the experiment on ‘randomized’, daily for 4 days. Again, there was a lot of variation. On the UK site alone the results changed every day going from 35,800,000 results to 11,700,000 back down to 32,400,000 and then up to 32,900,000.

Now, I am no expert on how Google works, so I can only speculate on the reasons for these constant changes in values. It may be because…

  • searches are more popular in certain countries and so there are more results for that search in that region
  • regional Google sites may update faster than others
  • information may be differentially filtered – this is obvious in China (doesn’t use Google) where the values were consistently low across the board, continually staying around 1%. Except strangely, for ‘political dissident’ where it had the highest value.

There must be other reasons, but what is clear is that Google is definitely a fickle source of information, especially if a researcher’s aim is to identify all randomised controlled trials for their chosen topic. It could also make life difficult for information specialists and other people working with comprehensive searches.  So…should we rely on Google as our “go to” for our information?  Debatable.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: