>> Perhaps there are domains of activity where it is often true.
>> But it's certainly not my experience as both a producer and
>> user of non-academic indicators-based research, each in both
>> commercial settings and in government.
> Sounds like a researchable question, no? Where do 'actual decision
> makers' get their information sources from? Why do they turn to it?
> What's the role of 'names that you know' and expectations of how
> their 'audiences/bosses' will respond to the source?
Sure. But it's one thing for Source A to beat Source B. It's another to
turn to Source A because there is no Source B. My experience is that most
'audiences/bosses' are pretty free and easy with the range of acceptable
Source As, as long as it provides answers to the question that they think
is important -- I have seen really any name brand do, which ranges from
IDC to eMarketer to just about anything with the word "university" or
"college" attached.
Now, where data are available from both, say, Research Service Inc. and
Local Research University, the issue you are talking about would probably
come up. I would offer only two caveats. First, it comes up much more
rarely than one would think. Even on questions where multiple attempts at
the answer exist, people often stop at the first reasonable one they find,
rather than evaluating the range of options -- in other words, "names that
you know", but as a search strategy rather than a weighing of alternative
findings. That's why things like citation in the media and search engine
placement are so important to industry researchers: if you can be the
first stop, you may well be the only stop. And, second, even where it
does come up, other criteria are more important: depending on who the
'audience/bosses' are, a more recent/conservative/aggressive answer may be
preferred. My experience has almost always been that criteria such as
these -- i.e. if both Source A and Source B exist, which better fits the
story I am trying to tell -- outweigh any reputational issues.
But, for sure, researchable: till then, anecdotes are all we've got. ;-)
> Certainly there are failings in the review system
> for academic research, but that one can point to
> them is evidence for the point I was, perhaps clumsily,
> trying to make: at least there is some form of public
> review of academic (and government) research, some ethic of
> open review that one can rest some weight of quality upon.
>
> There doesn't seem to be any equivalent system for commercial
> research, revealing methods seems optional, at best.
I think I know what you mean. And, intuitively, it makes sense: academic
research has a relatively uniform institutional structure, so there should
be less variability in research quality than in research conducted in, for
instance, commercial or government settings that are outside the academic
world.
And, variability-wise, maybe that's true. But my sense has also been
that, on the one hand, academic review standards in some journals are very
limited because of anything from disciplinary insularity to very lax
oversight of journals' own review processes -- the opposite is true in
other locations of course, but nonetheless -- and that, on the other hand,
the reputational risks facing industry researchers who know that their
primary audience may well be the exact industry participants best
positioned to review their work, may spur them to be very rigorous indeed.
Putting all that together, then, my sense is that the primary harms of
lousy non-academic research (and especially industry research which is
sold and whose producers are therefore incented to have the media talk
about it) derive from their reporting in popular media. I believe that
lousy industry research that pertains to the telecom and IT domains just
doesn't cut it among its primary consumers, who will know if it is way
wrong. On the other hand, the same research may be very widely reported
in the news media, who not only have few ways of knowing whether the study
is way wrong or not, but rarely consult -- nor really have the time to
consult -- potential sober-second-thought providers who might provide a
reality check.
In fact, there is something similar which occurred with the telecom
meltdown of the late 1990s. The 100-day traffic-doubling myth derived not
from any industry research, but from a media pick-up and amplification and
distortion of something a single spokesperson at an Internet backbone
provider claimed. The absorption of that myth into an environment with a
proclivity to believe it and without competing evidence to infirm it was
one of the more significant bad-"research" consequences around. Except
that there was no "research" at all.
That's why I would agree:
> Now if the commerical firms got together a review system,
> that would be a very interesting 'industry self regulation'
> development :)
A review system which functioned in the form of a certification standard,
and which was publicized and adopted widely enough that it caused
reporters to regard certified studies differently than non-certified
studies, would probably do more good than harm. And, you know, it is
probably the sort of thing where an academic-led effort could successfully
play the role of neutral arbiter.
cheers
Bram
