Differentiate incomplete from inconsistent results

[WilcoFiers]

Currently, incomplete can mean an implementation is missing data, (untested), or that its results are different (expect pass, actual fail). As discussed in act-rules/act-rules.github.io#850, we'd like to distinguish between these two.

We want two flags:

1. Incomplete: Some of the "actuals" are "untested"
2. Inconsistent: Some of the "actuals" aren't in the list of allowed expected results.

We'd also have to add "untested" to all of the allowed expected results. Something is not inconsistent if the result is untested. For an implementation to be "correct", it has can neither be incomplete nor inconsistent.

[WilcoFiers]

Some notes and test cases:

enum Consistency { Consistent, PartialConsistent, Inconsistent }
type Outcome: "passed" | "failed" | "inapplicable" | "cantTell" | "untested"
type Mode: "automatic" | "semiAuto" | "manual"

type Assertion {
  expected: Outcome,
  actual: Outcome,
  correct: boolean,
  mode: Mode,
  url: string
}

type Implementation {
  actRuleId: string,
  consistency: Consistency,
  complete: boolean,
  implementationId: string,
  mode: Mode,
  assertions: Assertion[]
}

type ImplementationSet {
  actRuleId: string,
  consistency: Consistency,
  complete: boolean,
  mode: Mode,
  implementations: Implementation[]
}

An implementation is consistent if

1. the outcome for all passed test cases is passed, inapplicable, cantTell or untested
2. the outcome for all inapplicable test cases is passed, inapplicable, cantTell or untested
3. The outcome for all expected failures is failed, cantTell or untested
4. Of each category (passed, failed, inapplicable), at least one outcome is untested
5. At least one one of the outcomes is neither untested nor cantTell

An implementation is partially consistent if

1. the outcome for all passed test cases is passed, inapplicable, cantTell or untested
2. the outcome for all inapplicable test cases is passed, inapplicable, cantTell or untested
3. The outcome for at least one, but not all failed test cases is failed, cantTell or untested
4. For each test case type (passed, failed, inapplicable), at least one outcome neither cantTell nor untested
5. At least one one of the outcomes is neither untested nor cantTell

An implementation is inconsistent if it is neither consistent nor partially consistent

A set of partially consistent implementations is a consistent implementation if

1. For each failed test case, at least one of the implementations has an outcome that is failed, cantTell or untested

An implementation is incomplete if

1. one or more of the outcomes is untested

A complete and consistent implementation

Expected	Rule 1 out
passed	inapplicable
failed	failed
inapplicable	passed

An incomplete and consistent implementation

Expected	Rule 1 out
passed	passed
passed	untested
failed	failed
inapplicable	inapplicable

A complete and partial consistent implementation

Expected	Rule 1 out
passed	passed
failed	failed
failed	passed
inapplicable	inapplicable

An incomplete and partial consistent implementation

Expected	Rule 1 out
passed	passed
passed	untested
failed	failed
failed	passed
inapplicable	inapplicable

A complete and consistent set of implementations

Expected	Rule 1 out	Rule 2 out
passed	passed	inapplicable
failed	failed	inapplicable
failed	passed	failed
inapplicable	inapplicable	passed

An incomplete and consistent set of implementations

Expected	Rule 1 out	Rule 2 out
passed	passed	inapplicable
failed	failed	inapplicable
failed	passed	failed
failed	passed	untested
inapplicable	inapplicable	passed

A complete and inconsistent set of implementations

Expected	Rule 1 out	Rule 2 out
passed	passed	inapplicable
failed	failed	inapplicable
failed	passed	failed
failed	passed	inapplicable
inapplicable	inapplicable	passed

An incomplete and inconsistent set of implementations

Expected	Rule 1 out	Rule 2 out
passed	passed	inapplicable
passed	passed	untested
failed	failed	inapplicable
failed	passed	failed
failed	passed	inapplicable
inapplicable	inapplicable	passed

Inconsistent implementations:

Passed should be not failed

Expected	Rule 1 out
passed	failed
failed	failed
inapplicable	inapplicable

Inapplicable should be not failed

Expected	Rule 1 out
passed	passed
failed	failed
inapplicable	failed

At least one passed test case must not be untested

Expected	Rule 1 out
passed	untested
failed	failed
inapplicable	inapplicable

At least one failed test case must not be untested

Expected	Rule 1 out
passed	passed
failed	untested
inapplicable	inapplicable

At least one inapplicable test case must not be untested

Expected	Rule 1 out
passed	passed
failed	failed
inapplicable	untested

At least one outcome must be neither untested nor cantTell

Expected	Rule 1 out
passed	untested
failed	untested
inapplicable	cantTell