This repo is an accelerator help to help kick start using MLOps with Microsoft Fabric. Microsoft Fabric supports mlflow and uses the Azure Machine Learning model registry to support managing models inside of a workspace.
-
Create a Development Workspace to contain the artifacts.
-
Within the workspace create a new folder designating the Project for the given model development work (in my case it is MLOps Framework)
-
Clone this repo
-
Import the Notebooks from the folder in this repo (4. Promote Model is work in progress and will be available soon)
-
Open up the 1. Prep Notebook
-
Connect to a Lakehouse
-
Create a folder named iris and upload the iris.csv file
-
Execute the notebook. You should now have a table and a file named transformed_iris.parquet
-
Open up the 2. Model Build Notebook
-
Execute the notebook. Once completed, you should see an XGBoost-Experiement1 artifact in the root of your workspace.
-
Move the file to the Experiments folder (this is to help organize)
-
Open up the 3. Model Validation Notebook
-
Execute the notebook. Once completed, you should see a new model in the root.
Move this model to the Models folder in the same way you move the Experiment artifact.
To configure this for your use case, you will need to take a few steps.
-
Modify the 1. Prep Notebook to reflect the notebook, experiment and model names.
-
Update the 2. Model Build Notebook with your specific model building logic.
-
Update the 3. Model Validation Notebook to reflect the metrics you wish to use to determine the quality.
-
Schedule the Model Build and Validation on a cadence. The Validation is build to be able to be executed over and over and will not register a new model if the metrics are not better.
These are the todos.
-
Add a method to move the models between workspaces and execute.
-
Add logic to call the second notebook from the first once building is complete.
-
Allow for a dynamic test of the metrics vs a single metric to allow more flexibility.