bug: printed shrink example is not actual input

[peter-jerry-ye]

Example

///|
fn prop_deque(deque : (Deque, Int)) -> Bool {
  println("input: \{deque}")
  let (deque, n) = deque
  deque._.truncate(n)
  guard n < 0 || deque._.length() <= n else { return false }
  guard n > 0 || deque._.is_empty() else { return false }
  true
}

///|
type Deque @deque.T[Int] derive(Show)

///|
impl @quickcheck.Arbitrary for Deque with arbitrary(i, randomstate) {
  @deque.from_array(Array::arbitrary(i, randomstate))
}

test {
  @lib.quick_check_fn!(prop_deque)
}

Test with Core 532f2e8e4c5e041ba695c24ca69a50cc0005d7bd (where truncate has a bug)

Expected Output

*** [5/0/100] Failed! Falsified.
(Deque(@deque.of([])), -2)

Actual Output

*** [5/0/100] Failed! Falsified.
(Deque(@deque.of([])), 0)

[CAIMEOX]

Thanks for reporting.
The function prop_deque here is not pure because truncate will change the state of parameter deque. Impure function can not be tested properly by QuickCheck. There are two solutions can be adapted:

• implement a copy function for Deque to eliminate the state effect inside queue: deque.copy()._.truncate(n)
• (QuickCheck): Use the same RandomState to generate two deques, one for reporting and one for testing. However, this solution may have a 2x performance loss, which is obvious when generating large test samples, so I haven't decided whether to do this yet.
  Maybe we should have a Copy trait in core and support derive(Copy)?

[peter-jerry-ye]

Still one other possibility is to keep the output in some global ref, which also may be used in #23 . All of these solutions have some kind of performance penalty for non pure data structures.

Another possibility may be adding a verbose and seed option, so that the user can jump to the seed when the failure started and then print every shrinked test case before testing each of them

[peter-jerry-ye]

It seems that the issue is caused by this line:

quickcheck/src/lib/testable.mbt

Line 180 in ded134c

counterexample(f(x), s)

which should've been f(a).

[CAIMEOX]

It seems that the issue is caused by this line:

quickcheck/src/lib/testable.mbt

Line 180 in ded134c

counterexample(f(x), s)

which should've been f(a).

Fixed in this commit.

[illusory0x0]

@peter-jerry-ye your prop_deque would panic, quickcheck now can't handle panic,

Deque::truncate run failed when len < 0

result:

+++ [100/0/100] Ok, passed!

test code

type Deque @deque.T[Int] derive(Show)

///|
impl @quickcheck.Arbitrary for Deque with arbitrary(i, randomstate) {
  @deque.from_array(Array::arbitrary(i, randomstate))
}

///|
fn prop_deque(deque : (Deque, Int)) -> Bool {
  let (deque, n) = deque
  if n < 0 {
    // Deque::truncate, assert len >= 0 
    true
  } else {
    deque._.truncate(n)
    deque._.length() <= n
  }
}
test {
  @qc.quick_check_fn!(prop_deque)
}