Disease-specific data

[annaritz]

From @ntalluri: For the real data idea, instead of focusing on data tied to a specific molecule or pathway, another approach could be to use disease-specific data. The gold standard nodes could then be molecules or genes already known to be associated with that disease. This idea seems a little different than what is already in the benchmarking repo and what we discussed at the meeting.

[annaritz]

Here, we would have to decide what would be inputs to SPRAS (e.g., node prizes or sources/targets), and what would be held out for evaluation. Many methods assume that disease genes will be near each other (hence disease module identification tasks), so we could do a cross-fold validation approach where we hide some percent of the data, use SPRAS to connect the remaining data, and then check to see how much of the hidden data appear in the output.

[agitter]

We started to talk about whether a cancer-specific evaluation could be a good summer project in this direction for the Madison team. There are many prior pathway reconstruction applications in cancer to use for inspiration and precedent. There are also may public cancer omics datasets, including some data types that could be used to generate pathways and others that could be use to evaluate pathways even if they are not actually ground truth.

[annaritz]

One possible source is DisGeNet, which has lots of different diseases. We just checked and the database also includes evidence type (mutation, expression, fusion gene, etc.). We could reconstruct pathways for one evidence type and see if we recover genes in the other evidence type(s). Here's an example for melanoma) - unregistered users see only the top 6 genes but we could get access to the whole thing.