Add option for embedding generation upon BioImage Ingestion

[ktsitsi]

This PR:

• Introduces an extra functionality inside the ingestion process of biomedical images. The new ingestion function on the client side as well as the UDF has been enhanced with extra arguments that enable the creation of image embeddings during the ingestion
• The newly added ingestion arguments are the following:

    :param embedding_model: The model to be used for creating embedding. Supported values are of type class EMBEDDINGS
    :param embedding_level: The resolution level to be used for the embedding. This could be different from the ingestion level
    selected with parameter `level`
    :param embedding_grid: A tuple that represents the (num_of_rows, num_of_cols), in which the image will be split in patches
    for the embedding creation. According to this grid internally the image is being split to fit this requirement.

• Currently the supported model is RESNET for the embeddings, but with the introduced types abstraction we are able to enhance this functionality in the future.
• Another type SupportedExtensions has been added concerning the file extensions we support with our ingestion function.
• This additional functionality changes the Data model specification we have set and thus updates the stored image groups to fmt_version = 3

The resulted image is shown below:

Building a local docker image of the UDFs I was able to test it and the results are shown below as well:

This PR should be followed by PR's that will do the following and will be linked upon creation here:

• Update the python-udf-imaging dockerfile in TileDB-REST-UDF-DOCKER-IMAGES with the dependencies needed as tiledb-vector-search and tensorflow
• Also update the tiledb-cloud version in UDFs to match the one after the release of this PR, so functions and modules are being found on the UDF side

[shortcut-integration]

This pull request has been linked to Shortcut Story #33097: Add option for embedding generation upon ingest.

[thetorpedodog]

these helpers don’t seem like they would be public-facing.

[thetorpedodog]

check out os.path.splitext

[ihnorton]

(also seems slow..)

[ihnorton]

This seems like it is going to be very slow?

[NikolaosPapailiou]

Can you explain a bit the use case that you want to support?

From what I understand this is creating one vector index per image, storing the embedding of different patches of the image. Does this mean that you want to find similar patches within the image? Do you want to do any cross image queries?

[ktsitsi]

The idea here is to find similar patches across multiple images. We don't have any specific customer request for any of that. The idea behind the patches though is that in this data environment is quite unlikely to find similarities across whole images. Each image is quite different from one another as a whole based on the depicted cell. Also from my knowledge due to the blank space outside the cells captured by sensor it is quite possible to introduce a lot of noise in the model if fed to it as a whole. So my thought was that a future user would like to search for similarities across images of a specific region of the query image, or in a next step to be able to query a region from the viewer and find similar "abnormalities" - that he can justify and I can't- across the images.

[NikolaosPapailiou]

You also need to pass source_uri of the input vectors (tmp_features_file)