Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 145 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Operator Architecture

The operator reads a Kubernetes Custom Resource (defined in ???) and uses it to instantiate Pachyderm's helm chart (in ???) to get a Kubernetes manifest, which it then creates. It's based on a kubebuilder template, and many of its files are generated.

# Repo Structure

**TODO**: Comment on each of these directories, and any files of interest (remove files where possible. Many of these are generated by Kubebuilder)

.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

surround the repo structure with three back ticks (```) to retain the formatting.

├── api
│   └── v1beta1
│   ├── groupversion_info.go
│   ├── pachydermexport_types.go
│   ├── pachyderm_types.go
│   ├── pachyderm_webhook.go
│   ├── webhook_suite_test.go
│   └── zz_generated.deepcopy.go
├── bundle
│   ├── manifests
│   │   ├── aiml.pachyderm.com_pachydermexports.yaml
│   │   ├── aiml.pachyderm.com_pachyderms.yaml
│   │   ├── pachyderm-operator.clusterserviceversion.yaml
│   │   ├── pachyderm-operator-controller-manager-metrics-service_v1_service.yaml
│   │   ├── pachyderm-operator-manager-config_v1_configmap.yaml
│   │   ├── pachyderm-operator-metrics-reader_rbac.authorization.k8s.io_v1_clusterrole.yaml
│   │   └── pachyderm-operator-webhook-service_v1_service.yaml
│   ├── metadata
│   │   └── annotations.yaml
│   └── tests
│   └── scorecard
│   └── config.yaml
├── bundle.Dockerfile
├── config
│   ├── certmanager
│   │   ├── certificate.yaml
│   │   ├── kustomization.yaml
│   │   └── kustomizeconfig.yaml
│   ├── crd
│   │   ├── bases
│   │   │   ├── aiml.pachyderm.com_pachydermexports.yaml
│   │   │   └── aiml.pachyderm.com_pachyderms.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │   ├── cainjection_in_pachydermexports.yaml
│   │   ├── cainjection_in_pachyderms.yaml
│   │   ├── webhook_in_pachydermexports.yaml
│   │   └── webhook_in_pachyderms.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   ├── manager_config_patch.yaml
│   │   ├── manager_webhook_patch.yaml
│   │   └── webhookcainjection_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── manifests
│   │   ├── bases
│   │   │   └── pachyderm-operator.clusterserviceversion.yaml
│   │   └── kustomization.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── pachyderm_editor_role.yaml
│   │   ├── pachydermexport_editor_role.yaml
│   │   ├── pachydermexport_viewer_role.yaml
│   │   ├── pachyderm_viewer_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   └── service_account.yaml
│   ├── samples
│   │   ├── aiml_v1beta1_pachydermexport.yaml
│   │   ├── aiml_v1beta1_pachyderm.yaml
│   │   └── kustomization.yaml
│   ├── scorecard
│   │   ├── bases
│   │   │   └── config.yaml
│   │   ├── kustomization.yaml
│   │   └── patches
│   │   ├── basic.config.yaml
│   │   └── olm.config.yaml
│   └── webhook
│   ├── kustomization.yaml
│   ├── kustomizeconfig.yaml
│   ├── manifests.yaml
│   └── service.yaml
├── CONTRIBUTING.md
├── controllers
│   ├── backup_job.go
│   ├── diff.go
│   ├── errors.go
│   ├── generators
│   │   ├── images.go
│   │   ├── manifests.go
│   │   └── template.go
│   ├── initialize_database.go
│   ├── pachyderm_controller.go
│   ├── pachydermexport_controller.go
│   └── suite_test.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   ├── boilerplate.go.txt
│   ├── bundle_prep.sh
│   ├── catalogsource.yaml
│   └── charts
│   ├── 2.0.0
│   │   ├── images.json
│   │   ├── pachyderm-2.0.0-rc.1.tgz
│   │   └── values.yaml
│   └── 2.0.2
│   ├── images.json
│   ├── pachyderm-2.0.2.tgz
│   └── values.yaml
├── main.go
└── PROJECT

# Testing the Operator

**TODO**: Fill this in. Open Questions:
- How are operators installed in a running Kubernetes cluster?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This question is very broad. I need clarification to be able to answer conclusively.

- Is it necessary to install OLM or load the operator into OperatorHub somehow?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Installing an operator in Kubernetes or Openshift can be down either through Kubernetes manifests or via OLM (the Operator Lifecycle Manager).

- Does the operator need to be tested in an OpenShift cluster? Does this mean that we need to provision an AWS cluster to run tests or can we use Minikube/KinD somehow, for local testing? Could we run automated tests in CircleCI?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is little to no functionality specific to Openshift that needs to be deployed. As a result, I would think you could test in either a Minikube or KinD cluster.


# Releasing the Operator

**TODO** Fill this in. Specifically, explain how to:
- Upload new Pachyderm and Worker images to our project in the Red Hat Partner Connect portal
- Also explain how to add people to this project, if needed
- Update the operator itself in Red Hat Marketplace (does it need to be updated anywhere else? ODH?)
- Update the Red Hat Marketplace documentation for Pachyderm
- Update the Open Data Hub (ODH) docs. These docs are also used for Pachyderm in RHODS.


3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
The pachyderm operator is an application responsible for installing, watching and managing Pachyderm resources in your Openshift cluster.

Pachyderm is the data foundation for machine learning. Pachyderm provides industry leading data versioning, pipelines and lineage that allow data science teams to automate the machine learning lifecycle and optimize their machine learning operations (MLOps).

To lean about the structure of the operator's architecture, this repo's code, how to test the operator, or how to release the operator, see [CONTRIBUTING.md](CONTRIBUTING.md)

## Usage

**1. Using AWS S3 for Pachd storage**
Expand Down