Hello!
Has pHash been considered for finding the closest match?
pHash performance is pretty good because it allows you to precompute all hashes of the images you want to compare with.
You can try their demo here: https://www.phash.org/demo/
Implementation wise it's pretty simple.