All About Search

Mitchell F. Wyle, PhD
Euphorion Inc.

Age-old techniques for searching have become more important recently as more and more information is stored online. Searching in Internet systems has become especially daunting, and yet more important to end-users who are now tasked with navigating through complex layers of menus and hierarchies to uncover buried content.

IDC research states that searching is by far the most common user activity on the web. Already, over a third of all Internet users spend more than 2 hours weekly searching the net. Almost three quarters experience great frustration while searching.

Dozens of research reports in the popular and trade press have now surfaced expounding the "staggering" amount of frustration and time wasted by web surfers who cannot find what they need using Internet search systems.

It is clear that a surgical scalpel is needed to cut through the clutter of information items available in the major archives on the public Internet and within corporations. However the current crop of search engines are dull knives. More than 36% of Internet users spend more than two hours per week searching. 71% are frustrated in their searches. 86% believe search could be better. Over 50% of users turn to search before they depart a web site.

What does Search Really Mean?

"Search" itself is a wide, loosely-defined term. Search is no longer the exclusive domain of the large "search engine" companies that index the entire web. Most significant web sites have a local search system of some kind and many end-users go to search systems first because they find it will save them time. Search is also the "last gasp" effort of loyal web site users before they leave a web site, sometimes forever.

The unstated and underlying goal of search systems is somehow to solve the fundamental problem of effectively matching an end-user's momentary information need with all and only the information required to fill the need. However a better definition of the true information need and therefore the result of a a successful search experience is to fulfill the user's intent. A search system should not only retrieve all and only the relevant information an individual requires, but should also result in a match for the user's intent. A match is defined by a search result that elicits action by the person performing the search.

A Brief History of Search

In the 1970's we saw the rise and over-hype of artificial intelligence under which some important ideas and algorithms in search were developed. Natural language processing systems were developed that can disambiguate the part-of-speech of words in a document. The 1980's gave full-text search researchers much more text to search because it saw the rise of dense, inexpensive storage (CD-ROM technology) and also the widespread use of word processing systems. The 1990s gave us the large scale commercialization of the public Internet and the sudden popularity of Internet "search engines" such as Alta Vista and Google.

How Search Systems Work

Natural languages are composed of a finite number of words. Stringing these words together with rules (grammar) is how we communicate. Web Documents, product descriptions and other information items are strings of natural language text. Data storage systems consider the record, document, or web page the unit of storage. Searches for words within these documents are the basis of most free text systems.

Many search systems will eliminate "stop words" or the 150 most common words in a language and other words that contain no meaning such as "a, the, an, of." The use of stemming and stop word elimination can broaden the number of items returned by a query. These techniques drastically reduce storage requirements and increase speed since there are so few word stems compared to all variations of all words.

In addition, words in English can be trimmed down to their root stems by simple, mechanical rules of suffix removal. Stemming allows a search system to find matches for various forms of the same word such as "matches, matched, matching" etc. Unfortunately it can also introduce error and imprecision since "neutron, neutralize, neutral" will reduce to the same stem and have different meanings. And, of course, identical words in different contexts have widely varying meanings. Good search products can use parts of speech and word phrases to increase precision.

Search systems can also expand a query by adding known synonyms to the terms in the query when searching. These synonym lists can be useful in large systems where the authors of documents used different words with identical or nearly identical meaning.

Relevance Ranking Search Becomes Mainstream

Recently search systems have evolved past the nearly-useless 'count the hits' systems that were naively installed in many software applications. Most full-text systems have some form of relevance ranking algorithm that calculates a predicted relevance score between the text of a query and the text contained in the information items (web pages, products, documents) being searched.

In addition, for information systems that contain more than a few tens of thousands of items, structured data (Size, SKU, Manufacturer, Price) can be used for even more precise matching. Structure can almost always be derived from collections of information items. There is, after all, some structure in natural language.

It is also useful to assign a categorization scheme or semantic index (lexicon, controlled vocabulary, concept space) to a collection for precise matching and also for hyperlink navigation as an alternative to entering text in a text box. Examples of public Web systems that display a hierarchical ordering of web pages on the public Internet are Yahoo! and The Open Directory Project.

Relevance, Usefulness, Cogency

It is sometimes possible for a person to establish a 'yes/no' relevance of an information item to a query. That is, people can often say, "yes, that item is relevant and no, that one is not."

To have automated systems predict relevance, we construct models where relevance decisions such as relevance ranking or yes/no, relevant, non-relevant judgments are quantified. One major problem with automated search systems developed thus far is the fundamental assumptions of a universal, absolute meaning for relevance.

Relevance in search systems is in fact a complicated concept. It is much harder to model a user's momentary information need and true intent for entering a search system than the assumptions made by most systems. These assumptions include that there is an absolute, time-independent, yes/no relevance for each item available in the system and in the user's the query.

However there is certainly a time dependence of the true value and usefulness of information items retrieved. In addition, users who type a query are unaware of the information items contained within the system and they are often unclear about their true information need or 'intent.'

Search Systems Evolve

Tools, methods, and new search systems are now becoming available to improve search results.

Recent developments in search include the use of relevance feedback, where users can indicate they want "more like this one," and the use of link density or a web page's "popularity" to increase its predicted relevance. Link density and collaborative filtering methods have become mainstream. They are currently in use at Google and Amazon.com.

The most significant advance in search for the purpose of our discussion is the development of economic and business measures for the success of search. We can now evaluate the performance of a search system according to how well it meets the intent of its owners and users instead of relying on the flawed relevance prediction performance measures developed in the 1960's.

Just as the link density relevance ranking algorithm in Google eclipsed the simple search of previous search engines, match methodologies and intent measurements will replace the current search implementations that frustrate users and webmasters alike.

Future Trends in Search

In the immediate future, search systems will continue to splinter and specialize. Users who become adept at using one or another search system will over-use it to perform other inappropriate tasks because it will be much more convenient than trying to learn the idiosyncrasies and arcane features of other systems.

Because of the growth of the information available, search systems will continue to become more popular. In spite of their shortcomings, search systems are 'doomed to success' and great pressure will be brought to bear to improve them.

From Search to Match - Intent

Euphorion has created a match methodology that centers on intent.

Intent Matching incorporates opportunities for a user to clarify their momentary information need and intent. This is often a process of informing the user of what is possible. Search systems can then use database and full text system technology to fulfill the user's intent.

Success is measured not by relevance but by the actions taken by a user. The intent synthesis process is successful if the user acts on the information retrieved. A cogent interaction as one that results in action.

Why Intent Matching is Better

As more first-time Internet users discover retail sites, they discover that navigating through the complex information space is overwhelming. They lack time, patience, vocabulary or ability to find what they need. In many cases they simply give up clicking and look somewhere else to fulfill their intent.

For the web system owner, the result is a combination of increased cost, lost opportunity, and reduced customer satisfaction.

Creating an environment in which users rapidly and easily convert from visitors to buyers is the most direct and effective way to ensure that the money spent on attracting visitors is fully leveraged. A well-implemented search solution has a huge impact on this conversion. It acts as an online sales person, immediately taking users to what they need to fulfill their intent. It can suggest related products, improving up-sell and cross-sell features of a retail web site. It can even build clarifying choices, providing personalized recommendations where appropriate. In essence, a good search system instills within the user the confidence that they have found all and only what they need and leads the user to act upon the information.

Search systems are now used in many, mainstream business settings and their success can be measured by the actions taken by users of these systems. Performance tuning of almost any search system is possible after instrumenting it for these measures and collecting data on its baseline performance.

The Value of Instrumentation

Euphorion has developed and deployed a comprehensive set of tools that instrument web search systems on retail sites and then provide a detailed analysis of the search system's performance in terms of user intent, measured by user action. The reporting system within the instrumentation tools also provide economic analyses of the search system's performance in terms of the web site owner's intent.

Using the data gathered by these instrumentation tools and techniques, Euphorion's search system experts then augment, tailor, tune, and repair the site's search system to increase revenues, customer satisfaction, and the site's "stickiness." These ongoing efforts enable the simultaneous intent fulfillment of both web site users the web site owner.

Monitoring search usage and analyzing carefully how it is used also enable these features to be tuned to improve the search system's economic ROI and to increase end-user satisfaction..

For example one can analyze searches that produce no results. Depending on the search terms used and reasons for the null-set results, specific actions can be taken. The search system owner can add more synonyms, turn on n-gram indexing and other methods that provide immunity to misspellings. The search system can be tuned to use more aggressive word stemming in its indexing and retrieval.

The result of such ongoing tuning is continual improvement in capturing sales lost to bad search, and more repeat traffic to the web site. Other search features include stored searches and agent-based search.

The Euphorion Advantage

It is important to understand the type of search problem deeply in order to select the best tool or suite of tools from the growing arsenal available to the web site owner.

Euphorion does not sell yet another shrink wrapped software search package. Euphorion partners with world-class search software vendors such as Verity and offers a much more flexible approach to implementation.

The Euphorion methodology begins with an analysis of a system's current search solution. The current system is instrumented to capture data about the types of searches performed, the results and the subsequent user behavior.

Then Euphorion recommends ways to optimize how users interact with a search system. This includes optimizations for traditional search capabilities such as handling misspelled words, condensing words to root stems, etc.

For even better search performance, Euphorion will integrate domain-specific lexicons, an inference engine, advanced user profile data, and customized user interfaces.

For new systems, Euphorion assists clients to identify, assess, select and integrate search technologies that are most appropriate to their needs.

Euphorion then instruments the search system to capture data and analyse the search system. The instrumentation enables Euphorion clients to track performance of the search components of their systems in real-time and to undertake continual search system optimizations.

Euphorion also assists in deploying comprehensive, intent-centered search solutions that reduce costs, increase revenue and improve customer satisfaction.

The Euphorion approach, including the selection of the most appropriate search software, expert search strategy and intrumentation and implementation services adds unique value that other search services cannot provide.

About Euphorion

Search has always been a core practice area within the company. Euphorion implemented one of the first-ever database/full-text hybrid search systems on the public Internet at Intellimatch.

Euphorion can perform deep research on a client's search capability, analyze search performance, and provide on-going, real-time feedback on the impact of improvements and other change. If there are benefits to be gained through improved search, Euphorion can identify them and rapidly implement a solution optimal to the client's need.

If your
customers
can't find it,
they won't
buy it.

Euphorion, Inc.
1900 McCarthy Blvd.
Suite 204
Milpitas, CA 95035

Telephone:
USA-408-894-6800

Fax:
USA-408-894-6899

Email:
info@euphorion.com

Web Site:
www.euphorion.com