The Under-served High Value Search Market

After less than a decade of competition, market giant Google has emerged triumphant over all major search engines.  Once preeminent Yahoo is expected to be absorbed into Microsoft, while Microsoft’s own MSN Search has been unable to retain market share.  Ask.com has announced its departure from the general search arena, and AOL’s market share barely registers.  However, despite appearances, the story isn’t over.  All these players – even Google – are leaving a pile of money on the table.  They are leaving money on the table because they can not handle N-dimensional Query.

“Simple Query” vs. “Complex Query”

A search query is a type of question.  When a search user enters a simple phrase – such as “green tea” – the user is asking a simple question.  If that simple question is asked enough times by enough people, current search works great.  This type of current search capability is “limited search”, and limited search works great for 84 – 85% of all searches submitted to major search engines.  But what happens when the search user asks a complex question, or a question that’s never been asked before, or a question asked only infrequently?  What happens if answers to that question are spread piecemeal over many documents instead of one “popular document”?  What happens if the “most relevant” documents shown the user are written for domain-specific expertise the user doesn’t possess?  What happens if the answers are in documents that are part of the “hidden” or “deep” web?  Then, the major search engines results are often very unsatisfactory.

The search industry has made many efforts to provide answers to these “complex queries”.  Google has both “organic” as well as page-ranked results.  Other players are deploying advanced techniques such as latent semantic analysis, topic detection, concept extraction, user profiling, inter-active dialog with users, GUI enhancements, or search integrated portals.  Still other initiatives rely upon imposing Semantic Web constructs upon all web content.  In spite of these efforts, “complex queries” continue to challenge search engine capabilities.  As a result, the market for “complex queries” remains un-served or under-served - and this market is big.  Estimated to be 16% of all current search queries, this market presents a major new opportunity to search advertisers.  Right now, search advertisers are paying for ads which are inappropriately placed in front of complex query searchers “out of context“ due to unsatisfactory search results.  Right now, search advertisers do not get their ads placed in front of complex query searchers who would be valuable potential consumers.  Complex Queries demand a new and “more complete search” – satisfied only by fundamentally extending search engine capabilities.  An “N-dimensional Search” is required to exploit this opportunity.

As defined by Correlation Concepts, an N-dimensional Query (NDQ) must be semantically complex and/or compound.  An N-dimensional Query must have a minimum of two terms, phrases or concepts.  An NDQ must contain “content words” which exhibit lexical and/or semantic overlap or disjunction.  Finally, an N-dimensional Query is usually comprised of five, six, or seven terms, phrases or concepts.  N-dimensional Search Queries are often “one-offs” or have very low frequency of occurrence. 

The Opportunity

The monetization opportunity from the high value demographics of N-dimensional Search users is very significant.  To begin with, N-dimensional search query users are among the most focused, most likely to click thru and “stick”, and most likely to “take the next step” such as submit contact information or make a purchase.  Because of these characteristics, N-dimensional search query users create a market for premium ad fees.  Yet right now, thirty-two million N-dimensional search query users a day get no useful answers and are presented with no useful advertising.  There is also a latent market for N-dimensional Search that consists of 7–8% of existing search query volume that is available for capture.  Capturing this latent demand alone would add revenue from fifteen million additional high value search customers.

There are other very good reasons to capture the N-dimensional Search market.  The first is that a significant percentage of search queries go “abandoned” – meaning the search user examines the top results page returned by a search engine and decides that none of the result links or the accompanying advertisements are worth a click through.  Nobody in the search industry wins from an abandoned search.  N-dimensional Search has real potential to reduce the percentage of abandoned search queries.  Also, a search engine that provides N-dimensional Search capability has a unique advantage in gaining and retaining user loyalty.  After all, if a high value search consumer is confident that a given search engine will return a great answer to even the most complex query, why would that consumer go to any other search engine for answers to simple queries, local queries, spoken queries or product purchase queries?

Perhaps the most significant incentive for a search provider to provision N-dimensional Search is that N-dimensional Search solves a major problem for search engine advertisers – the problem of acquiring user profiles.   A well known fact is that the more specific the user profile information about an individual consumer, the more valuable a sales lead becomes.  N-dimensional Search takes even an anonymous user’s input and determines the user’s actual intent for the search.  For example, if a search consumer enters the terms “green tea”, “life quality”, and “metastasize”, almost any 12 year old would immediately understand that the search consumer was concerned about “cancer”, and any search consumer should expect to see advertisements from healthcare providers, pharmaceutical companies, holistic and nutritional supplement vendors, and related businesses.  Yet, existing search engines do not understand this simple user query, and can not present appropriate advertisements with results from that same query.  One major search engine returned no advertisements at all for this query, and another major search engine returned an advertisement for automobile purchase discounts!  Because N-dimensional Search “connects-the-dots”, and exhibits an approximation of human comprehension, N-dimensional Search users can get the answers they are “really” after – even if the user doesn’t “know how to ask the question”.  And with N-dimensional Search, advertisers can get the same results as with expensive customer capture methods.

The Research

There is substantial research supporting these estimates for the size and value of the existing and potential market for N-dimensional Search.  To model existing demand, Correlation Concepts allocated the approximately 16% of searches (five, six, seven or more terms) as evidence of demand for N-dimensional search.  This is achieved by means of statistical analysis and phrase composition analysis on such queries.

Demand for search should be characterized as a derived demand; i.e., households only demand search in the course of carrying out other economic activities and not for the pleasure of search in and of itself.  That means that household demand for search is only determined by demand for exogenous economic activities.  In some views, the cost of search is considered virtually irrelevant, but search costs have a documented effect on consumer behavior.  Search time is the major component of variable costs experienced by those using major search engines to find information on the internet.  Because of these characteristics, latent demand for search can therefore be modeled in the much same manner as latent demand for transportation, called “induced travel”.

The underlying theory behind induced travel is based upon the simple economic theory of supply and demand. Any increase in highway capacity or “supply” results in a reduction in the time cost of travel.  Like travel, search - in a characteristic shared by all internet applications - is highly sensitive to time cost and those costs have been calculated.  In search, increased capacity is construed to mean reduction in time to reach a relevant search result.  When any “good” such as travel or search is reduced in cost, demand for that good increases.  Induced travel that represents new trips and longer trips is called “generative”.  On average, every 10% reduction in travel time “cost” produces an 8% increase in travel.  Reduction in search costs have been found to produce increases in effective demand elasticity for consumer goods.  Although it would be fair to estimate all increase in “capacity” to be attributable to N-dimensional Search, to be conservative, a price elasticity of only minus point five (-.5) is estimated.  Further, 80% to 100% of all new capacity in a transportation system is absorbed by induced travel over time, and the relative near and long term effect of improved search has been shown to exhibit similar characteristics.

Conclusion

Most of the major search engines are expanding into mobile search and local search, competing with content providers, or branching out into lines of business that have little to do with search.  Most of the major search engines have not said much lately about solving users’ complex search query needs.  They know well that general search engine customers are not satisfied with the quality of the results they get from what are in fact N-dimensional Queries.  In certain domains, such as Enterprise Search, where a much higher percentage of search queries are N-dimensional, the level of dissatisfaction is very high and very public.  The N-dimensional Search market is very large and very valuable.  That market is greater than the current market share of all search engines except for Google and Yahoo.  Any search engine that can meet the N-dimensional Search needs of this under-served, high value search market will dramatically alter the balance of power in the search engine industry.

Mark Bobick is CTO of Correlation Concepts, a research and development company currently building products based upon the Correlation Technology Platform.  Previous domains of interest include the Semantic Web, multi-agent systems, edge computing, server virtualization, and very high transaction rate object databases