Rapid scale up and scale down of a MCD result in a cordoned machine

[aaronfern]

How to categorize this issue?

/area auto-scaling
/kind bug
/priority 1

What happened:
We found a bug recently where a rapid scale up and scale down can result in a cordoned machine in the cluster.
This is due to the scale down logic we have with CA, and the recent changes to manageReplicas() in the machineSet controller

The case happens when CA scales down a MCD followed quickly with scaling up the same MCD before the MCD controller notices the replica change and start reconciliation. The result is the MCD ending up with the same number of replicas that it started off with, causing the MCD controller not to have any work to do since from it's point of view, the MCD replicas have not changed.
As part of the scale down operation, CA will cordon an underutilised node. This node just remains part of the cluster as MCM makes no further attempt to remove it

Problem example

T0:
	We have a mcd with 6 replicas and machines M1…M6
	MCD rep=6

T1:
	CA finds M1 unneeded. Reduces mcd replicas to 5, adds a mc (MC1) to trigger-for-deletion annotation on MCD, and cordons this machine
	MCD rep=5; trigger-for-deletion-annotation=M1
	M1 cordoned by CA
	(MCD controller does not run - for whatever reason)

T2: 
	CA increasesMCD rep=6 
	(MCD controller still does not run - for whatever reason)

T3:
	MCD controller start here
	sees that rep=6 (no change). No work done

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

[aaronfern]

SOLUTION:
We introduce a new annotation called LastReplicaChangeAnnotation (Name subject to change.)
This annotation is supposed to denote the timestamp of when the last change to replicas were made. This will help order operations and help ensure that operations are not executed with stale data
For an MCD, this is updated by CA denoting the last timestamp when CA changes the replicas
For an MCS, this is updated by the MCD controller containing the last timestamp when the MCD was updated
For an MC, this is updated by the MCD controller denoting the timestamp when the machine was marked for deletion. This is obtained from the trigger-for-deletion annotation

The trigger-for-deletion annotation will now contain a list of pairs [M1, T1] with each pair denoting the machine name and the timestamp when it was marked for deletion. This is in contrast to the current denotation of the annotation which is a set of machine names that have been triggered for deletion.

Solution example below (same case as above)

`LastReplicaChangeAnnotation` annotation denoted as LRCA
`trigger-for-deletion` annotation denoted as TMFDA

T0:
	MCD;LRCA=T0; rep=6
	MCS;LRCA=T0; rep=6
	M1;

T1
	CA reduces MCD rep to 5
	MCD; rep=5; LRCA=T1; TMFDA=[M1,T1]
	(MCD controller does not run - for whatever reason)
	MCS; LRCA=T0; rep=6 (no change)
	M1; (no change)

T2
	CA increases MCD rep=6 
	MCD; rep=6; LRCA=T2; TMFDA=[M1,T1]
	(MCD controller still does not run - for whatever reason)
	MCS; LRCA=T0; rep=6 (no change)
	M1 (no change)

T3
	MCD controller starts
	MCD; rep=6; LRCA=T2; TMFDA=[M1,T1]
	MCS; LRCA=T2; rep=6
	M1;LRCA=T1; P1 annotation set (set by mcd controller from TMFDA)

T4 (>T3)
	MCS controller starts
	Sees M1 has LRCA set
	Verifies that M1 also has P1 annotation 
	If M1.LRCA < MCS.LRCA, add to stale machines (delete machine)