Correlation

Correlations between many elements in a system inevitably contain indirect correlations from other elements.

You should also be very careful when you calculate a correlation between Time series.

Topics

Handling indirect correlation

The indirect correlation leads us to inaccurate prediction. There are several ways to filter out those indirect edges with spurious correlation:

• Correlation of correlations
• Partial correlation
  • e.g. Genetic Co-Occurrence Network across Sequenced Microbes
• Mutual information based approach (citation needed).
• Methods for the Inverse Ising problem
• network deconvolution: http://www.nature.com/nbt/journal/v31/n8/abs/nbt.2635.html
• Network link prediction by global silencing of indirect correlations
  • Lior Patcher: The network nonsense of Albert-László Barabási

Aggregating correlation

Sometimes, we want to average correlation values. A standard method is performing Fisher transformation and average the z values then transform it back to correlation coefficient.

• Averaging Correlations: Expected Values and Bias in Combined Pearson rs and Fisher’s z Transformations

Constructing networks from correlation matrices

• Constructing networks by filtering correlation matrices: A null model approach

Correlation Matrix

• Correlation Matrix Spectra: A Tool for Detecting Non-apparent Correlations?

Correlation and causation

• http://xkcd.com/552/
• Spurious Correlations
• http://www.google.com/trends/correlate

Generalized ways of inferring associations

• Reshef et al. Detecting Novel Associations in Large Data Sets
• Kinney and Atwal, Equitability, mutual information, and the maximal information coefficient

Articles

• http://www.johndcook.com/blog/2008/11/05/how-to-calculate-pearson-correlation-accurately/ - don’t expect that the small difference between two large numbers will be accurate.
• Cosine similarity, Pearson correlation, and OLS coefficients

References

• http://en.wikipedia.org/wiki/Partial correlation
• Graphical interaction models for multivariate time series by Rainer Dahlhaus
• Comparing association network algorithms for reverse engineering of large-scale gene regulatory networks: synthetic versus real data
• A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics

• PLoS biology: Large-Scale Mapping and Validation of Escherichia coli Transcriptional Regulation from a Compendium of Expression Profiles - Mutual information based method

• Genetic Co-Occurrence Network across Sequenced Microbes

• Removing spurious interactions in complex networks

• Unbiased community detection for correlation matrices