From the early days of search engine optimization (SEO) websites tried to differentiate themselves through keywords. So, while not all keywords were created equal, new websites utilized less competitive keywords to rank high at least for some search queries. However, as the space of single-word queries became more and more crowded, people started to target longer queries (i.e. key-phrases). This made the craft of keyword research subtler.
Today I'm going to examine some of the core concepts of keyword research in attempt to throw some light on its diverse practices.
Although it may sound unrealistic today, back in those days web-pages could rank high for nearly every keyword, regardless of its relevance page content. However, using these techniques nowadays, will almost certainly lead Google to penalize one's site. So, unless he wants to specialize in black-hat SEO, it's better to stay off these practices. Moreover, as the web become more and more competitive, targeting irrelevant keywords may waste a lot of resources on keywords that won't convert. Anyway, the issue of relevance is not black and white, since a keywords or a key-phrases may convey different meanings. Moreover, people may formulate a query that is unsuitable to their intent.
For example, suppose someone is Googling for [long tail]. While she may refer to 'long-tail traffic', she may also refer to the more general term 'Long-tail distribution'. In addition, she may refer to 'long-tail boat' or 'long-tail bicycle'. This means that some percentage of potential keyword traffic will always be wasted on queries with different intent than that of the web-page or websites that has been optimized for it. ( That before we take into consideration factors like the title attractiveness.)
The term 'keyword difficulty' or 'keyword competition' represents the required resources for a link or advertisement to appear "above the fold" for a given query. While the later term has a clear definition in the context of printed newspapers, its definition in the context of the world wide web is depended on the search engine results page (SERP) layout and on the size of the screen of the device on which it is displayed (For desktops it is usually refer to the top ad unit and the first three organic search results .) To make things even more complicated, there is no one-SERP for a given query. In an era where search engines using extensive strategies to personalize their results (including their ad units), different people may get slightly different results even if they send the same query to the same search engine at the same time.
Now, when we speak about keywords or key-phrases difficulty in the context of pay per click (PPC) advertisement the resources needed are mainly financial. And since these phrases are priced through a bidding system, the resources may be estimated easily. The organic search rating system, on the other hand, is much less transparent so it may be harder to predict the resources that will be needed rank among the to three SERP in advance. Indeed, there are some thumb rules to estimate the competition of a given keyword or key-phrase, however they may supply only first approximation.
Some tools, like SEM Rush, use Google AdWords keyword tools to estimate the difficulty of a given keyword. The assumption is that the free market can price correctly the needed resources to rank above the fold for a given keyword, and thus the cost of paid results should be similar to that of the organic search results. However, this assumption ignores the possibility of market failure. Moreover, there are keywords that are rarely used by marketers, although someone who may try to target these keywords for organic results would face great difficulty. For example, the word Plato has low competition on Google AdWords even though the query [Plato] has a bunch of .edu links and the respective wikipedia entry on its first page.
Another issue with keyword research for organic SEO is the dynamics of the web. Even if we use the most reliable tool, it may only supply us with the current competition of a given keyword. Changes in search engines algorithms, keywords traffic, the number of competitors, and the intensity of their efforts may change keywords difficulty over time. And while we may quickly switch to alternative keywords on a PPC campaign, changes on organic SEO campaign may take effect in a few months and in some cases even years.
Like many other phenomenons, the distribution graph of search queries traffic may be divided into 'head' and 'tail'. The head represents the most frequent queries, and the tail represents the rest of the queries (usually uncommon words and long phrases). Now, although a query in the head has much more potential than its counterpart in the long tail, in total the long tail queries have more potential than the queries of the head, since the tail is theoretically infinite. Therefore, we may combine long tail key-phrases as we wish in order to get the same traffic as the one head keyword. Long tail key-phrases are important especially for new websites that can't rank high for a head terms.
While some experts have lamented the moves Google had made to push keywords from the long tail to the head, in my opinion they help to distribute the keywords traffic more equally. Even from the PPC perspective, auto-completion and other incentives for long queries, did not change the supply of ads impressions but only change the distribution between the different queries. Moreover, we may argue that these developments even expand the opportunities for ads that can be more relevant but not too specific.