Decomposed characters and implied weights analyses

[RussellGarwood]

Hi Martin - I've run into an error which has me a little stumped, apologies if I am missing something obvious. I've created a minimal example on the basis that I think this relates to decomposing characters with polymorphisms, and then running any form of implied weights analysis on them. Here is a (zipped to allow upload) two taxon matrix:

Garwood_Morphy_error.zip

This script loads it (assuming the above file is in the wd, and TreeTools and TreeSearch packages are loaded) and orders the characters from a dataframe column (implemented as such as the other column, "characterStates" , is used to calculate weightings for reweighting downstream - not included here). It does a quick check to ensure the change in character number, and then creates an EW addition tree with no error, but fails under implied weights (at least, this is true on 3 of my systems, all running Linux and the development version of TreeTools to implement the handy Reweight function you created, but otherwise the versions of all required packages from CRAN):

# Load required libraries
library("TreeTools")
library("TreeSearch")

#Character number function
NumberOfChars <- function(x) sum(attr(x, "weight"))

#Load data from matrix
datasetInapplicables <- ReadAsPhyDat("Garwood_Morphy_error.nex")
NumberOfChars(datasetInapplicables)

#Decompose ordered characters
orderedCharacters<-data.frame("characterNumber"=c(155,151,148,48,42,12,5,1),"characterStates"=c(3,5,9,8,3,3,3,10))
datasetInapplicablesDecomposed<-Decompose(datasetInapplicables,orderedCharacters$characterNumber)
NumberOfChars(datasetInapplicablesDecomposed)

#No error
startTree <- AdditionTree(datasets[[i]], concavity = Inf)

#Error
startTree <- AdditionTree(datasets[[i]], concavity = 10)

From what I can see it fails when preparing the data for the IW analysis, specifically in the function MorphyErrorCheck, when it calls mpl_attach_rawdata, however I can't actually find the error code returned (ERR_DIMENS_OVER) in the morphy source code. I guess this implies that the reweighting has messed with the character numbers, and perhaps that this originates in TreeSearch, or that I don't have access to the latest version of the required Morphy code?

Any assistance much appreciated...

[ms609]

Thanks for the report; just back from a few days' leave. Will take a look anon.

[RussellGarwood]

Super, many thanks. FWIW the decomposed Equal Weights analyses, whilst running, are also giving me highly unexpected results - well supported clades such as the spiders (with a Bremer support in analyses with no ordered characters of >10) end up distributed across the tree. As such I'm suspicious that whilst not throwing an error, there may be something up with that too.

Hope you had a good break!

[ms609]

Thanks for the heads up. I've started encountering crashes in TreeDist recently, so worry that I might have broken something with tree structures in a recent change to TreeTools that's somehow evaded my unit tests.

[RussellGarwood]

Oh no - good luck! If it helps, I'm not getting any crashes running TreeSearch, and having written this script, I have now run 5 different matrices through it in recent days. For the analysis where I know the terminals well (some were other people's), with both Fitch, and Inapplicables aware Morphy parsimony but without any ordering, the trees appear relatively consistent in terms of topology. They are also behaving as I would expect from first principles: implied weights consensus is more resolved than equal weights; my taxon of interest is jumping around all over the place (it has a weird morphology with a combo of plesiomorphic and apomorphic traits), and strongly supported clades are consistent across recovered topologies.

[ms609]

I've replaced the undefined datasets[[i]] in your code with datasetInapplicablesDecomposed, and AdditionTree succeeds each time.

[RussellGarwood]

Oh dear - apologies, that was clearly an error in setting up the minimal example. Let me see if I can sort it out and post a new one.

[RussellGarwood]

Here we go, a slightly less, but still pretty minimal, example:

Garwood_Morphy.zip

It only seems to occur when I have more than one spider in, if this helps track it down. Based on my fiddling, with any one of those spiders in a two taxon tree, it is fine (I assume this is because it doesn't actually need to do any computation to get an addition tree with two terminals...)

[ms609]

Great, thanks, I've now reproduced the error and pinpointed it to an issue with the decomposed contrast matrix. I'll start work on a fix.

[ms609]

OK, I seem to have fixed the underpinning bug in TreeTools v1.15.0.9002, and will implement some rigorous testing of the problematic aspect (multistate ambiguity). I'd be interested to hear whether the results start looking sensible – if not, I may need to take a deeper dive.

[RussellGarwood]

Sure thing - I'll set off a run now and report back. Many thanks for looking into it!

[ms609]

Hmm, I've spotted your {24} ambiguities. That's tricky to decompose without permitting 3 as well. I'd be interested to know how you can rule out the middle state but not the others. Are cases where this coding can be justified likely to be common?

[RussellGarwood]

Good catch. The character in question is: "Median eyes (0 = four; 1 = three; 2 = two; 3 = one; 4 = absent)." One is present as a state thanks to insects, so clearly having a state 3 is, broadly, possible (although I don't recall the presumbled plesiomorphic state of this in insects off the top of my head). Within arachnids, though, where the median eyes are reliably laterally symmetrical, I think we'd expect a loss to go straight from two eyes to none, without going via one. Perhaps not common - and definitely here a result of long branches between the clade of interest and its outgroup - but defensible, I think.

P.S. Initial EW searches with decomposed characters are looking promising, even with a short search time. Waiting for IW now.

[ms609]

Then I'm still not clear what evidence would result in {24}, i.e. "either there are two eyes or none, but certainly no other number".

It sounds like there's a mismatch between the model of evolution you've described (i.e. two eyes → absent = one step) and the model implicit in your character (two eyes → absent = two steps, as it must pass through one eye). Your description also makes it sound like there's a further potential homology, 'symmetrical arrangement of median eyes', which is not captured by the ordered character. I'd also be nervous about treating 'absent' as a state of a transformational character.

Perhaps a richer set of characters would capture the nuance more effectively: from your description I might imagine something along the lines of

(i) median eyes: (0), absent; (1), present;
(ii) median eyes: arrangement: (-), inapplicable, eyes absent; (1), laterally symmetrical; (2), arranged along midline;
(iii) median eyes: number of lateral pairs: (-), no lateral pairs; (1), one lateral pair; (2), two lateral pairs

The most appropriate configuration would depend on whether you conceived the single medial eye as representing a fusion of lateral pairs (and hence requiring a preexisting pair of eyes); a first step towards two eyes (through duplication and lateralization – which doesn't sit well with my uninformed idea of how arthropod heads organize themselves); or an almost independent feature of patterning genes.

[RussellGarwood]

I was thinking about this on my walk into the Museum fur Naturkunde, and on reflection, 100% agree with the point regarding choice of ordering, or not. I spend so long thinking about my ingroup where there are repeated reductions of 4 lenses --> 2 --> 0 (a context within which ordering makes sense, and in whch the polymorphism in question reflects a loss in some, but not all species, in a genus), that I didn't then twig on the later addition of the molecular outgroups that this was no longer a sensible choice. It is also within this light that "absent" does make sense as a transformative state - indeed, even with the outgroups there are no taxa within this group where we are not confident that the loss is a derived condition (although I guess you could argue about that statement from a purist standpoint). I shall reformulate!

[RussellGarwood]

Oh, and all searches with decomposed and decomposed & weighted ordered characters, with both a Fitch formulation and not for inapplicables, are giving believeable results now!