General Guidelines
Version 2.1 (April 6, 2007)
Proprietary and Confidential – Copyright 2007
1
Part 1: Rating Guidelines
Welcome to the Quality Rating program! If you have previously read Version 1 of
these guidelines, please pay special attention to the text highlighted in yellow. If you
are new to the program, please read the entire document very carefully.
1. The Role of the Quality Rater
As a Quality Rater, you will evaluate ‘query-page’ Tasks. For each ‘query-page’ Task, you will:
•
Research and understand the query.
•
Evaluate the page based on its relevance to the query and its utility to the user.
•
Assign a rating from the Rating Scale.
Query refers to the word or words that a user types in the search box of a search engine.
The Page or Landing Page is the page you will evaluate. It is the page you see after you click on the URL.
Task Language and Task Location. You will be given a Task language and a Task location for each
query-page Task. You must evaluate each Task in the context of its language and location.
In this document, each query will be shown in square brackets, followed by the Task language and Task
location. Examples:
[ Elvis Presley ], English (US)
[ coca cola ], Spanish (MX)
Please keep in mind that the language of the query may not match the Task language. For example, you may
be working on a German (DE) Task and see a query in English.
2. Researching and Understanding the Query
You should understand each query before you evaluate it. If the meaning of the query is unclear, you will
need to do web research to learn about it. You can do this by entering the query in the search box of one or
more search engines and looking at the results returned by them. However, your rating should not be affected
by the ranking of results you see displayed by the search engines.
You will also need to understand the possible interpretations of the query and try to imagine a user who would
type the query. Think about what the user might be trying to accomplish.
Here are some things to consider:
Task Language and Task Location
All queries have a Task language and Task location. You must use the Task language and Task location to
understand the context of the query. Users in different parts of the world have different expectations for the
same query terms. Imagine the user typing in the following query and what they would be looking for.
Proprietary and Confidential – Copyright 2007
2
Task
Task
Query
Query as Typed by the User
Possible User Expectation
Language
Location
A user in the United States types
George Bush’s official
[ George Bush ]
English
US
the query [ George Bush ].
government web page
]
[ 喬治布希
A user in Taiwan types the query
Chinese
Information about George Bush
[ George Bush ] using Traditional
Taiwan
Traditional
displayed in Traditional Chinese
Chinese characters.
]
George Bush’s official
Chinese
A user in Taiwan types the query
government web page displayed
[ George Bush ]
Taiwan
Traditional
[ George Bush ] in English.
in English
The query may have different meanings, depending on either the Task Language or Task Location.
Query
Dominant Interpretation in the Task Location
[ football ], English (US)
The dominant interpretation is American football played with a brown oval ball.
The dominant interpretation is the game Americans call soccer and which is
[ football ], English (UK)
played with a round ball.
Multiple Interpretations
Does the query have more than one interpretation? Try to imagine the possibilities. Is one interpretation the
most likely or dominant interpretation?
Query: [ windows ], English (US)
Dominant Interpretation:
Possible Interpretation:
A universally known computer operating system
A piece of glass that can be looked through
Query: [ java ], English (US)
Dominant Interpretation:
Possible Interpretation:
Possible Interpretation:
A programming language
An island in Indonesia
Coffee
Query: [ mercury ], English (US)
Possible Interpretation:
Possible Interpretation:
Possible Interpretation:
The planet
The chemical element (Hg)
The car
Broad or Specific
Is the query broad or specific? Broad queries are best matched by broad pages; specific queries are best
matched by specific or narrow pages.
Broad
Specific
Query
[ digital cameras ]
[ canon SD550 ]
Possible intent
Looking to purchase a digital camera.
Looking to purchase this specific camera
Well-matched
result
Narrow result for a narrow query. (good)
Broad result for a broad query. (good)
Poorly-
matched result
Narrow result for a broad query. (not so good)
Broad result for a narrow query. (not so good)
Proprietary and Confidential – Copyright 2007
3
Amount of Information Available
If there is a lot of information available on the Web for the query, then a page with just a link or a short article is
not a good search result. If there is very little information available, you may consider the page to be a good
result.
Timeliness
Can a query be interpreted differently at different points in time? In 1994, the user who typed [ President
Bush ], English (US) was looking for information on President George H.W. Bush. In 2006, his son George W.
Bush is the more likely interpretation. You should always rate according to the current interpretation or in the
appropriate context if the query has explicit date information.
3. Query Types
Most queries can be classified in one or more of the following three categories: Navigational, Informational, or
Transactional.
Navigational
A navigational query is intended to locate a specific web page. The user has a single web site in mind, often
the official homepage or subpage of an official site. For example:
Navigational
Query
URL of the Landing Page
Description of the Landing Page
[ ibm ], English (US)
Official homepage of the IBM Corporation
[ yahoo mail ], English (US)
Official Yahoo! Mail web page
[ ebay ], English (US)
Official homepage of eBay
[ ebay ], Italian (Italy)
Official homepage of eBay Italy
[ Harvard libraries ], English (US)
Subpage of an official site
Informational
An informational query seeks information on a topic. The user is looking for information on the query topic
(broad or specific). The goal is to learn something by reading or viewing content on the web, such as text,
images, video, etc.
Informational
Query
URL of the Landing Page
Description of the Landing Page
[ tsunami ], English (US)
Wikipedia site with comprehensive information
Informative CIA web page on Switzerland
[ Switzerland ], English (US)
Transactional
A transactional query seeks to complete a transaction on the Web – for money or free – of a product or
service. The user is mainly looking for a resource (NOT information) available via web pages. The goal is to
download, to buy, to obtain, to be entertained by, or to interact with a resource that is available on the result
page or made available through the result page.
Transactional
Query
URL of the Landing Page
Description of the Landing Page
[ Beatles poster ], English (US)
Page on which to purchase poster
[ download adobe reader ],
Official free download page on Adobe website
English (US)
Proprietary and Confidential – Copyright 2007
4
Many queries fit into more than one category. For example: [united nations], English (US). The user may
expect to be taken to the homepage of the United Nations: http://www.un.org/ (navigational intent), or the user
may be looking for recent news regarding a United Nations resolution (informational intent).
Queries that fit into more than one category
Query
Navigational Intent
Informational Intent
Transactional Intent
[ “ipod nano” ],
Looking for the official product
Looking for reviews or product
Looking to purchase the product
English (US)
page
information
[ britney spears ],
Looking for the official website
Looking for pictures, latest news,
Looking to purchase a poster or
English (US)
of this celebrity
forums, etc.
download music or video
[ united nations ],
Looking for recent news regarding
Looking for the official website
None
English (US)
a United Nations resolution
[ cloth diapers ],
Looking for information about using
None
Looking to purchase the product
English (US)
cloth diapers
4. Rating Scale
After researching the query, you will evaluate the page that loads after clicking the link provided. Most pages
will be assigned a rating from the Rating Scale: Vital, Useful, Relevant, Not Relevant, or Off-Topic.
Vital
The Vital rating is used in the following special situations:
Query: The query has a dominant interpretation. The dominant interpretation is navigational.
Page: The page to evaluate is the official web page of the query.
Here are some examples:
Vital Results for English (US) Queries
Vital
Query
Vital Page URL
Description
[ Singapore
Yes
Official homepage
airport ]
[ toyota camry ]
Yes
Official product page on the correct site
The dominant interpretation of this query
[ apple ]
Yes
is Apple Computer, Inc., and this is the
official homepage.
Multiple Vital URLs with the same landing
[ barnes and
Yes
page (different domains with the same
noble ]
owner)
Multiple Vital URLs with the same landing
[ Fleet Bank ]
Yes
page. Fleet Bank was acquired by Bank
of America in 2004.
Multiple Vital URLs with the same landing
[ yahoo ]
Yes
page (same domain)
[ download
Yes
The download page on the official site
firefox ]
Hillary Rodham Clinton’s official U.S.
[ Hillary Clinton ]
Yes
Senate page
[ knitting ]
No
none
This is an informational query
[ ipod reviews ]
No
none
This is an informational query
Proprietary and Confidential – Copyright 2007
5
There is no dominant interpretation. The
following are all strong interpretations:
American Dental Association
[ ADA ]
No
none
American Diabetes Association
American with Disabilities Act
All of these interpretations have official
homepages, but none is Vital.
American Dental
Yes
Official homepage of this organization
Association
Most queries do not have Vital results because they are not navigational, and/or do not have official web
pages or official product pages associated with them.
All queries have a Task language and a Task location. For a landing page to be Vital, it must be appropriate
for the Task language and Task location.
Vital pages may not be the best possible result for the query. In fact, it is possible for a Vital page to be not
the most helpful at all, for example, in the case of a celebrity homepage that does not have what the user
wants to know.
The Vital rating is not based on the appearance or content of the URL. Often, the URL of the official
homepage will contain the query terms. For example, the Vital result for [ ibm ] is http://www.ibm.com.
However, sometimes a URL contains the exact query terms, but the landing page is not Vital. For example,
www.diabetes.com cannot be Vital for the query [ diabetes ] because the query is not navigational and there is
no official web page for the query. No person or entity can claim ownership of the query [ diabetes ]. If no one
can “own” the query, there can be no Vital result.
You will often see URLs that contain celebrity names, but when they are not maintained by the celebrity, they
Some large international corporations have country, as well as regional or global homepages. In general, the
country specific homepage is the Vital result for that type of query. If no country specific homepage exists, a
regional or global homepage may be Vital.
Sometimes the landing page asks you to choose a language, country, postal code, zip code, etc. These
pages should receive a Vital rating, if the pages behind them would receive a Vital rating. Similarly, splash
and flash pages should also receive a rating of Vital.
It is not uncommon today for individuals to maintain various types of personal pages on the Web. Homepages,
social networking pages, and blogs have become increasingly popular. Some individuals have more than one
blog and/or more than one homepage on a social networking site (e.g. myspace, facebook, friendster, mixi).
When these pages are maintained by the individual (or an authorized representative of the individual), they are
all considered to be Vital.
Useful
A rating of Useful is assigned to pages that contain some or all of the following characteristics: highly
satisfying, comprehensive, high in quality, and authoritative. Useful pages answer the query just right; they
are neither too broad nor too specific. For many queries, they are “as good as it gets.”
Examples of Useful pages include: a page that is highly informative; a timely and informative article; a page
that allows the user to complete the intended transaction; an important subpage on the correct site; the
homepage of the correct site when a specific product is asked for. If a query “asks” for a list, then a directory
(a collection of links) can be Useful, e.g. [ fudge recipes ], [ books about sharks ].
If an ambiguous query has several equally strong interpretations and each possesses a unique homepage, the
homepages would all be assigned a rating of Useful.
Proprietary and Confidential – Copyright 2007
6
Useful Results for English (US) Queries
Query
Useful Pages
Explanation
Subpage of the Microsoft website
[ Microsoft ]
devoted to Windows
Homepage of correct site for the
[ Honda Accord ]
product
American Dental Association
American Diabetes Association
Homepages for sites with equally
[ ADA ]
strong interpretations
American with Disabilities Act
[ meningitis
Highly informative page on
symptoms ]
authoritative site
Reputable site on which to
[ broadway tickets ]
complete a transaction
[ books on whales ]
List of books on whales
Relevant
A rating of Relevant is assigned to pages that have fewer valuable attributes than were listed for Useful pages.
Relevant pages might be less comprehensive, come from a less authoritative source, or cover only one
important aspect of the query.
Examples of Relevant pages include a page with a brief article on the topic of the query or a less important
subpage on the correct site. If a query “asks” for a list, then a single item is Relevant. For example, if the
query is [ fudge recipes ], a single fudge recipe is Relevant.
A rating of Relevant is also used for a homepage that would have been Vital if there had not been a more
dominant interpretation for the query.
Relevant Results for English (US) Queries
Query
URL of the Landing Page
Explanation
[ laser printer ]
Page with info. on the HP LaserJet 2840
[ seoul, korea ]
Page with map of the city of Seoul
Amazon page that displays Tom Cruise
[ Tom Cruise ]
movies for sale.
Homepage for Dell Magazines. This might
have received a Vital rating if not for Dell
[ dell ]
Computers, the dominant interpretation.
Not Relevant
A rating of Not Relevant is assigned to pages that are generally not helpful, but are still marginally connected
with the query topic. Not Relevant pages may be outdated, too narrowly regional, too specific, too broad, etc.
to receive a higher rating. They might have less information or come from a less authoritative source.
A rating of Not Relevant is also assigned to a page that has a link to good results on the same site, but is not
a good result itself. It may be an unimportant or useless subpage on the correct site. (Another example of a
Not Relevant page is one that has a link to good results on another site without providing any utility itself,
other than the link to the “good” results on the other site.)
Please note that, in both of these cases, there is a direct link from the landing page to the good result. In contrast,
landing pages that are search engine pages (or pages with search boxes on them, but which do not have a
relationship to the query) should be rated Off-Topic. If the query has to be typed in a search box, the page has no
relevance to the query, even if using the search box leads to relevant results.
Proprietary and Confidential – Copyright 2007
7
Not Relevant Results for English (US) Queries
Query
URL of the Landing Page
Explanation
A storybook for children about a mouse-
[ Dentist for children ]
dentist named Dr. De Soto
[ japan ]
Homepage of the Japan Patent Office
[ Louvre ]
Page about the movie “The Da Vinci Code”
The value of this page is in its link to the
[ SAT college board ]
College Board website; the page is not a
good result itself.
The “Dundee United” Fans Forum on the
[ BBC ]
BBC website
Off-Topic
A page that has zero relevance to the query should be assigned a rating of Off-Topic.
Ratings are not based on the presence or absence of the query terms. A page that contains the query terms,
but is conceptually off topic, should be given a rating of Off-Topic. For example, a page on doghouses is off
topic for [hot dog].
Another example of a page that should be rated Off-Topic is one in which the query
terms occur in different places on the page, unrelated to each other. Please note that a page may also be
rated Off-Topic even if the query terms appear in the URL.
A rating of Off-Topic also applies when there is lack of attention to an important modifier or element of the
query. For example: [ universities in India ]. An article about universities in Europe is Off-Topic.
If navigation to helpful content is very difficult, a rating of Off-Topic may be assigned. For example, if the link
to good results is poorly-labeled or buried at the bottom of a long list of links, or if you need to click multiple
times to get to helpful content, you may assign a rating of Off-Topic.
Off-Topic Results for English (US) Queries
Query
Off-Topic Pages
Explanation
Page that mentions Tom Beeler and Tom
[ Tom Cruise ]
Moore and vacation cruises.
[ hammerhead
Homepage of the San Jose Sharks
sharks ]
hockey team.
Article with title “Arctic Mission: Not
[ “Mission
Impossible”
