Do Models Matter? How Parent, Offspring, and Sibling Relationships Shape the Diffusion of AI Innovations

Comments

“Do models matter?” -> I usually equates “models” with “AI research” in my mind. When I hear this question, I’m thinking “obviously?” because models ~ AI in my mind. Models are what matter in AI research.. How about “pedigree” or “lineage” of models? 

Citing & using just Uzzi (2013) and White (2001) seems a bit too narrow to motivate the science of science aspect of this study. I’d go with a slightly more comprehensive list, starting from the author’s status, team characteristics, international collaboration, journal prestige, knowledge combination patterns, hot streaks, and then other misc. characteristics like title, number of references. And then we have accessibility and discoverability (open access, open code, etc.). 

To go for more general science journals, introduction can be compressed quite a bit. I think there are currently some repetitions and long elaboration about theories, which can be compressed into a few sentences with references and then the paper can go directly to research questions and hypotheses. 

The connection from “trading zone” to the ‘model network structure’ is not clear to me. Why do we need to look at the lineage of the models? How is it tied to trading zone theory? When I think about the model lineage network, I’m thinking more about Kuhn’s paradigm shift and Schumpeter’s creative destruction because I believe that the typical trajectory is to have a new idea/architecture (paradigm) and then subsequent refinement, which is punctuated with major departure from the state of the art models. 

This also makes me think about not just the average, but variance in outcomes. Working on old models without lots of attention (centrality) can lead to both marginal work and huge innovation—larger variance, compared with the case of more popular, state-of-the-art models. 

I think it may be useful to be much more explicit about exactly what the Eigenvector centrality captures (and what that means) and what’s the potential mechanisms that led to higher citations. For instance, using a model with a large centrality in a paper can mean that the paper is using a very well-established, but somewhat outdated, rather than at the cutting edge of the method development. There may be lots of benefits of using somewhat proven models, but if it’s too outdated, then the benefit of “provenness” may vanish. By contrast, being at the cutting edge can make it more risky. Similarly, there can be an intuitive mechanism behind the association with the parent’s centrality; e.g., high centrality of the parent may mean that there are way more competition in the space and it is less likely to get the attention. I think spelling out these potential mechanisms early can be useful to orient readers. 

But these mechanisms can also drastically differ when we talk about the AI/CS where models are actively developed vs. application domains where the models are applied to problems. 

Usage of graph neural network is cool, but may introduce a lot of obscurity in interpreting the results. Also lots of things that the GNN does may have a much simpler equivalent (even if it’s not exactly matching the performance). Can there be a simpler, interpretable model that can shed more lights on the exact mechanisms in addition to GNN based model?

“specifically those papers published between January 2000 and December 2022” -> This is before ChatGPT. Also this period includes the inflection point of 2009-2010 where the deep learning really took off. I think before & after this point may be somewhat different because before the deep learning revolution, you needed good feature engineering so “models” may not matter as much, but then deep learning began to dominate as a paradigm and “models” start to mean an end-to-end system that includes feature extraction. So, there are three potential periods that may be distinct and interesting to compare: pre-deep learning, deep learning and transformers, post-GPT.

“We restrict our analysis to papers categorized under the machine learning portal, excluding those related to astronomy, physics, computer science, statistics, and mathematics.” -> this limits the scope to the field of ‘model development’ and exclude the application areas. Would that limit the applicability of “trading zone” idea?

There are quite a few missing data due to the issue of OpenAlex and so on. What kinds of impact can they have on the analysis?

“GNN” can mean a variety of network configuration… “converging to a consistent Mean Absolute Error (MAE) of approximately 4.07.” -> unclear what was the exact loss function; MAE of what?