API

[figueroakl]

Description

This component involves creating a web API for the program, which allows other applications and services to interact with the program via HTTP requests. The API should provide endpoints for querying the parsed genomic variations, filtering the variations based on certain criteria, and other relevant operations. You can implement the API using Rust's actix-web crate, which provides a high-performance and easy-to-use web framework.

Acceptance criteria

Tasks

[em-sanger]

I have had success in my previous experimentation with OpenFaaS.

It is an open-core service for serverless functions, aka Function-as-a-Service.

In summary means you define a function handler for an endpoint that:

1. takes input
2. processes it
3. returns the output

Everything else is managed by OpenFaaS.

This usually works nicely if (and only if) our "actions" for this library consist of a few functions, , e.g.:

• parse a file
• render a parsed file
• etc.

This can be deployed to a Kubernetes cluster, and it seems to scale really well!

There are a few drawbacks, most of which are solved by the paid version:

• the stock open-source installation only supports basic-auth with one account, for admin API access. (I normally don't use the admin API, and instead use custom resource files in k8s)
• High Availability isn't guaranteed (not sure of our request volume).

I don't imagine we'll need the paid version, as the open-source core is very good IMO, but just wanted to list the "shortcomings".

Here is the function template for rust, if anybody else wants to have a preliminary look.

[figueroakl]

@em-sanger this sounds great. Would you be willing to present this to the group as an option. So we can get some questions answered.

This does sound like a great feature we could support.

[em-sanger]

@figueroakl sure, I can present about this. In summary, it's like a self-hosted AWS lambda.

More detail can come later, but my suggestion would be to package the concrete implementations of each "action" (e.g. a file parser with associated methods) in a crate on Cargo.

We could then expose each method of the file parser on a different function docker image, and use dead-code elimination to keep them as lightweight as possible.

This also doesn't preclude us from making a single-executable API using the same methods in the library, for simple and low-volume use-cases.

[jeff-k]

This sounds like a cool feature to support.

The idea right now is that we have a separate library and binary crate so that if you just want to use the parser without building the cli that's possible. Both crates live in this repo's workspace: https://github.com/Rust-Wellcome/vcf-parser/blob/main/Cargo.toml. There's also the option to use 'features' in the parser library to make some functions optional if we ever get to a place where we've got optional stuff that effects the binary size. Do you think this plan will cover those needs?

[em-sanger]

The separation of library and application code is a good idea, and very much compatible with my suggestion.

This permits us to eliminate the CLI/other-application code from any API we build (OpenFaaS or otherwise), and really slim down the size.

To be more explicit about structure, the functions should all be within the vcf crate. We could then support both:

• A standalone, single-binary API that has all of the functions on different endpoints
• A series of OpenFaaS docker images, that each install the vcf crate, and that each expose one function ( e.g. parse/validate/insert) from there. Dead-code elimination should hopefully prevent the docker images installing unnecessary code.

I'll explain more at the next Thursday meet-up, but the benefits of OpenFaaS lie in scalability mostly, and in Event-Driven Architecture. It will most likely be simpler to start with a simple actix API, then split that into separate functions images using OpenFaaS if we later want its benefits (e.g. if we have a demo instance).