Re: [Air-l] 988 exabytes of info created in 2010
The results of commercial research firms are always so nicely packaged and illustrated---and maybe that's one of the reasons that actual real world decisions are based on them. (50 Rhode Islands? That sounds like a lot, I'd better buy more ;)
Which, of course, makes them 'important' but for reasons other than their 'truth'. Their importance seems self-reinforcing
Oh, I don't know. The latter thesis -- that, faced with different answers to a similar question, "actual real world decision" makers prefer the truthiness of name-brand commercial research -- is no doubt possible. Perhaps there are domains of activity where it is often true. But it's certainly not my experience as both a producer and user of non-academic indicators-based research, each in both commercial settings and in government. Rather, I've found that a less sneering version of the former thesis fits a lot better. But it's not just packaging and illustration: it's choice of topic matter. Any time I have seen non-academic research I produced used "out there", it's been because there were very few alternative sources for answers to the question being asked. For that matter, when there did exist alternative sources, they tended if not to converge, at least to take each other into account. In this instance, the few-alternatives thesis seems to be the case. The EMC-sponsored studies, first at Berkeley and now at IDC, have gotten a lot of press. I have not seen a lot of other comprehensive attempts to answer the "how much information" question -- a rather different proposition, obviously, then what one thinks of that question. For what it's worth, an IDC narrative of how they counted things, as opposed to a third-party nwes story about it, is at: http://www.emc.com/about/destination/digital_universe/pdf/Expanding_Digital_...
research huh? Does that qualify as an 'A Journal'? ;)
I wouldn't think so. Academic and other forms of research each have their own norms and control mechanisms. They differ from each other, and they differ internally. I would wager that those who place all weight on how they differ from each other, and no weight on how they differ internally, will end up with trust mechanisms that disappoint them in the long run. I have seen very careful and properly-reviewed non-academic research -- even shepherded some through reviewing myself -- and, by the same token, some very inaccurate and poorly-reviewed academic journal articles. So perhaps the journal/not-journal binary is not a good totalising lens through which to view the wide world of doing research and other words for figuring stuff out.
Is anyone aware of research that asks the question, 'how reliable are the predictions of commercial research firms?'.
... or that asks that question of any form of institutional research. I wonder how useful such studies would really be. In the telecom and IT sectors there have, from time to time, been scorecards in industry magazines, which would fairly up front about how they did their scoring. I suppose this could get you started. (Parenthetically: one firm, eMarketer, does something like this. More specifically, and at least the last time I checked, their business model was that, rather than undertaking primary research of their own, they bought or borrowed, etc. other primary research and conducted meta-research -- that is, showed what all the other research firms were saying, then sold that as a competing product. I suppose there are issues of shooting one's golden goose here. But that's another story.) More helpful from a moving-things-forward standpoint, I would think, would be to investigate the methodologies being used. The standard by which predictions should be judged is a bit of a moving target -- ensuring that a prediction is based on sound reasoning and data is more do-able, and probably more reasonable. cheers Bram
Hi Bram, thanks for the thoughtful discussion. On Mar 7, 2007, at 11:10 AM, Bram Dov Abramson wrote:
The results of commercial research firms are always so nicely packaged and illustrated---and maybe that's one of the reasons that actual real world decisions are based on them. (50 Rhode Islands? That sounds like a lot, I'd better buy more ;)
Which, of course, makes them 'important' but for reasons other than their 'truth'. Their importance seems self-reinforcing
Oh, I don't know. The latter thesis -- that, faced with different answers to a similar question, "actual real world decision" makers prefer the truthiness of name-brand commercial research -- is no doubt possible. Perhaps there are domains of activity where it is often true. But it's certainly not my experience as both a producer and user of non- academic indicators-based research, each in both commercial settings and in government.
Sounds like a researchable question, no? Where do 'actual decision makers' get their information sources from? Why do they turn to it? What's the role of 'names that you know' and expectations of how their 'audiences/bosses' will respond to the source?
Rather, I've found that a less sneering version of the former thesis fits a lot better. But it's not just packaging and illustration: it's choice of topic matter. Any time I have seen non-academic research I produced used "out there", it's been because there were very few alternative sources for answers to the question being asked. For that matter, when there did exist alternative sources, they tended if not to converge, at least to take each other into account.
I think you are right, the questions are closer to what the 'market' wants. But there's quite a bit wrapped up in that concept, that research that hopes to be 'relevant' can learn from. Such as why such questions are being asked? By whom and for what purpose?
In this instance, the few-alternatives thesis seems to be the case. The EMC-sponsored studies, first at Berkeley and now at IDC, have gotten a lot of press. I have not seen a lot of other comprehensive attempts to answer the "how much information" question -- a rather different proposition, obviously, then what one thinks of that question.
For what it's worth, an IDC narrative of how they counted things, as opposed to a third-party nwes story about it, is at:
http://www.emc.com/about/destination/digital_universe/pdf/ Expanding_Digital_Universe_IDC_WhitePaper_022507.pdf
Thanks, that's useful.
research huh? Does that qualify as an 'A Journal'? ;)
I wouldn't think so. Academic and other forms of research each have their own norms and control mechanisms. They differ from each other, and they differ internally. I would wager that those who place all weight on how they differ from each other, and no weight on how they differ internally, will end up with trust mechanisms that disappoint them in the long run. I have seen very careful and properly-reviewed non-academic research -- even shepherded some through reviewing myself -- and, by the same token, some very inaccurate and poorly-reviewed academic journal articles.
So perhaps the journal/not-journal binary is not a good totalising lens through which to view the wide world of doing research and other words for figuring stuff out.
Maybe it was a semi-joke? ;)
Is anyone aware of research that asks the question, 'how reliable are the predictions of commercial research firms?'.
... or that asks that question of any form of institutional research. I wonder how useful such studies would really be.
Certainly there are failings in the review system for academic research, but that one can point to them is evidence for the point I was, perhaps clumsily, trying to make: at least there is some form of public review of academic (and government) research, some ethic of open review that one can rest some weight of quality upon. There doesn't seem to be any equivalent system for commercial research, revealing methods seems optional, at best. The market reputation of the company, or maybe the author, could function in that way, but if that's confounded by the sort of effects we discussed above then how can anyone have confidence in it? You mention 'different norms and control systems' for commercial research above (and internal variance), what do you have in mind there? Having worked in consulting I can imagine internal reviews up the chain of researchers/supervisors, but are there other forms reviews? external even? Maybe an ISO 9000 style certification for internal quality procedures? The recent work on Wikipedia has brought 'standards of knowledge review and reliability' to the fore, yet we haven't heard much about commercial research in that regard.
In the telecom and IT sectors there have, from time to time, been scorecards in industry magazines, which would fairly up front about how they did their scoring. I suppose this could get you started.
(Parenthetically: one firm, eMarketer, does something like this. More specifically, and at least the last time I checked, their business model was that, rather than undertaking primary research of their own, they bought or borrowed, etc. other primary research and conducted meta-research -- that is, showed what all the other research firms were saying, then sold that as a competing product. I suppose there are issues of shooting one's golden goose here. But that's another story.)
Thanks, interesting.
More helpful from a moving-things-forward standpoint, I would think, would be to investigate the methodologies being used. The standard by which predictions should be judged is a bit of a moving target -- ensuring that a prediction is based on sound reasoning and data is more do-able, and probably more reasonable.
Makes sense. At least then users of research would have some yardstick to judge commercial research, but there still wouldn't be an institutional quality system analogous to the academic review process (or the role of journalists and FOI requests in government research), or even the edit-war-until-we-drop of Wikipedia. Now if the commerical firms got together a review system, that would be a very interesting 'industry self regulation' development :) Cheers, James
participants (2)
-
Bram Dov Abramson -
James Howison