"He uses statistics as a drunken man uses lampposts - for support rather than for illumination." ~ Andrew Lang
How to use statistics for illumination and not for support in SEO:
1. Be Neutral
We all have an interest of some sort in a particular answer -- whether it is because providing one answer is expected to more likely earn one a raise or continued employment, or because one variety of answer reinforces one's earlier beliefs. When one has an ulterior motive in the gathering or analysis of data, the search for truth can more easily be corrupted by the search for supporting evidence and the exclusion of contrary evidence. In such cases, psychologically speaking, one would feel they are protecting themselves; when in reality, one would just be jeopardizing themselves and their organization.
If one is truly after the truth, is it perhaps easier to be neutral than to not be? It's when we start thinking we absolutely know truth a posteriori when fallacy overcomes reason and the quality of work is no longer optimized toward truth. That's why I advocate to gather, analyze, and report data with a beginner's mind, maintaining an attitude of openness, transparency, and eagerness to understand the truth -- whatever that truth might be. Reserve preconceptions when studying a subject, even when studying that subject at an advanced level.
SEO Application: Rather than balk at doubling the amount of non-paid keywords sending search visits to a domain in a quarter because such a thing has never happened for your domain, learn from your data about what things you've done to earn more non-paid keywords sending search visits to your domain from the past and apply that experience and analysis to a constructive strategy (probably some sort of link building, or resource development for your site...).
2. Avoid Pressure
Use a neutral source for the data collection. I suggest use of common tracking and reporting tools. SEOmoz tools, Google Analytic, Crazy Egg, and GetClicky are my personal favorite tools, and, it seems to me, a lot of people have used them, so the reports don't require as much explanation as reports from other, perhaps more esoteric, perhaps hand-made tools and filters.
The problem is often that (many) people often want a particular answer to confirm to their earlier beliefs or other's expectations. If those people are filtering the way data is gathered or analyzed, others in their organization would be right to be suspicious of the data being reported. The potential for the hidden hand of an subconscious motive may actually cause (or allow) for errors to occur or for manipulation of the data. Further, SEO's, like so many who work with statistics, sometimes face pressure to get the 'right' answer. From others, or from oneself.
Therefore, to reduce inherent bias from these sources when gathering or analyzing data, you should specify in advance what you're going to do. Be exact. Specify your goals and methods in a so called "SMART" method: ensure specificity, measurability, attainability, relevancy, and timeliness.
After you explain what you did in your gathering or your analysis, always release the data (if permitted) for anyone that wants to see it, to analyze it themselves. They might come up with a different answer, in which case you can argue about the assumptions that were made. This usually means you'll be "educating your audience" at first, but eventually should evolve into a very nice relationship.
Don't Miss: