-
Notifications
You must be signed in to change notification settings - Fork 109
[WIP]Create separate Go module for OTE test extension and migrating OCP-25806 to component repo #563
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[WIP]Create separate Go module for OTE test extension and migrating OCP-25806 to component repo #563
Changes from all commits
f086230
79ffc75
2b0baba
1fc8bac
9aee474
f422500
843aad4
1b5cf32
19a46ad
e74c62a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| /openshift-apiserver | ||
| /_output | ||
| .idea | ||
| /openshift-apiserver-tests-ext | ||
| /test/extended/tests-extension/bin/ |
This file was deleted.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| [ | ||
| { | ||
| "name": "[Jira:openshift-apiserver][sig-api-machinery] sanity test should always pass [Suite:openshift/openshift-apiserver/conformance/parallel]", | ||
| "labels": {}, | ||
| "resources": { | ||
| "isolation": {} | ||
| }, | ||
| "source": "openshift:payload:openshift-apiserver", | ||
| "lifecycle": "blocking", | ||
| "environmentSelector": {} | ||
| }, | ||
| { | ||
| "name": "[Jira:openshift-apiserver][sig-api-machinery][openshift-apiserver][encryption] Force encryption key rotation for etcd datastore should rotate keys and update encryption prefixes [Slow][Serial][Disruptive][Suite:openshift/openshift-apiserver/conformance/serial]", | ||
| "labels": {}, | ||
| "resources": { | ||
| "isolation": {} | ||
| }, | ||
| "source": "openshift:payload:openshift-apiserver", | ||
| "lifecycle": "blocking", | ||
| "environmentSelector": {} | ||
| } | ||
| ] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # Get the directory where this Makefile is, so we can use it below for including | ||
| DIR := $(strip $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))) | ||
|
|
||
| # Definitions for the extended tests | ||
| GO_PKG_NAME := github.com/openshift-eng/openshift-tests-extension | ||
|
|
||
| GIT_COMMIT := $(shell git rev-parse --short HEAD 2>/dev/null || echo 'unknown') | ||
| BUILD_DATE := $(shell date -u +'%Y-%m-%dT%H:%M:%SZ') | ||
| GIT_TREE_STATE := $(shell if git rev-parse --git-dir > /dev/null 2>&1; then if git diff --quiet; then echo clean; else echo dirty; fi; else echo unknown; fi) | ||
|
|
||
| LDFLAGS := -X '$(GO_PKG_NAME)/pkg/version.CommitFromGit=$(GIT_COMMIT)' \ | ||
| -X '$(GO_PKG_NAME)/pkg/version.BuildDate=$(BUILD_DATE)' \ | ||
| -X '$(GO_PKG_NAME)/pkg/version.GitTreeState=$(GIT_TREE_STATE)' | ||
|
|
||
| METADATA := $(shell pwd)/.openshift-tests-extension | ||
|
|
||
| TOOLS_BIN_DIR := $(CURDIR)/bin | ||
| BINARY_NAME := openshift-apiserver-tests-ext | ||
|
|
||
| .PHONY: help | ||
| help: #HELP Display essential help. | ||
| @awk 'BEGIN {FS = ":[^#]*#HELP"; printf "\nUsage:\n make \033[36m<target>\033[0m\n\n"} /^[a-zA-Z_0-9-]+:.*#HELP / { printf " \033[36m%-17s\033[0m %s\n", $$1, $$2 } ' $(MAKEFILE_LIST) | ||
|
|
||
| #SECTION Development | ||
| .PHONY: verify #HELP To verify the code | ||
| verify: tidy fmt vet | ||
|
|
||
| .PHONY: tidy #HELP Run go mod tidy. | ||
| tidy: | ||
| go mod tidy | ||
|
|
||
| .PHONY: fmt | ||
| fmt: #HELP Run go fmt against code. | ||
| go fmt ./... | ||
|
|
||
| .PHONY: vet | ||
| vet: #HELP Run go vet against code. | ||
| go vet ./... | ||
|
|
||
| .PHONY: build | ||
| build: #HELP Build the extended tests binary | ||
| @mkdir -p $(TOOLS_BIN_DIR) | ||
| GO_COMPLIANCE_POLICY="exempt_all" CGO_ENABLED=0 go build -mod=mod -ldflags "$(LDFLAGS)" -o $(TOOLS_BIN_DIR)/$(BINARY_NAME) ./cmd/... | ||
|
|
||
| .PHONY: update-metadata | ||
| update-metadata: build #HELP Build and run 'update-metadata' to generate test metadata | ||
| $(TOOLS_BIN_DIR)/$(BINARY_NAME) update | ||
| $(MAKE) clean-metadata | ||
|
|
||
| .PHONY: build-update | ||
| build-update: build update-metadata #HELP Build and update metadata and sanitize output | ||
|
|
||
| .PHONY: clean | ||
| clean: #HELP Remove build artifacts | ||
| rm -rf $(TOOLS_BIN_DIR) | ||
|
|
||
| #SECTION Metadata | ||
|
|
||
| .PHONY: list-test-names | ||
| list-test-names: build #HELP Show current full test names | ||
| @$(TOOLS_BIN_DIR)/$(BINARY_NAME) list -o names | ||
|
|
||
| .PHONY: clean-metadata | ||
| clean-metadata: #HELP Remove 'codeLocations' from metadata JSON | ||
| @echo "Cleaning metadata (removing codeLocations)..." | ||
| @for f in $(METADATA)/*.json; do \ | ||
| jq 'map(del(.codeLocations))' "$$f" > "$$f.tmp" && mv "$$f.tmp" "$$f"; \ | ||
| done | ||
|
|
||
| .PHONY: verify-metadata #HELP To verify that the metadata was properly updated | ||
| verify-metadata: update-metadata | ||
| @if ! git diff --exit-code $(METADATA); then \ | ||
| echo "ERROR: Metadata is out of date. Please run 'make build-update' and commit the result."; \ | ||
| exit 1; \ | ||
| fi | ||
|
Comment on lines
+70
to
+75
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fix absolute path in git diff check. The Apply this diff to use a relative path: .PHONY: verify-metadata #HELP To verify that the metadata was properly updated
verify-metadata: update-metadata
- @if ! git diff --exit-code $(METADATA); then \
+ @if ! git diff --exit-code .openshift-tests-extension; then \
echo "ERROR: Metadata is out of date. Please run 'make build-update' and commit the result."; \
exit 1; \
fi
🧰 Tools🪛 checkmake (0.2.2)[warning] 70-70: Missing required phony target "all" (minphony) [warning] 70-70: Missing required phony target "test" (minphony) 🤖 Prompt for AI Agents |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,9 +10,9 @@ They use the framework: https://github.com/openshift-eng/openshift-tests-extensi | |
| | Command | Description | | ||
| |----------------------------------------------------------------------------|--------------------------------------------------------------------------| | ||
| | `make tests-ext-build` | Builds the test extension binary. | | ||
| | `./openshift-apiserver-tests-ext list` | Lists all available test cases. | | ||
| | `./openshift-apiserver-tests-ext run-suite <suite-name>` | Runs a test suite. e.g., `openshift/openshift-apiserver/conformance/parallel` | | ||
| | `./openshift-apiserver-tests-ext run <test-name>` | Runs one specific test. | | ||
| | `test/extended/tests-extension/bin/openshift-apiserver-tests-ext list` | Lists all available test cases. | | ||
| | `test/extended/tests-extension/bin/openshift-apiserver-tests-ext run-suite <suite-name>` | Runs a test suite. e.g., `openshift/openshift-apiserver/conformance/parallel` | | ||
| | `test/extended/tests-extension/bin/openshift-apiserver-tests-ext run <test-name>` | Runs one specific test. | | ||
|
|
||
|
|
||
| ## How to Run the Tests Locally | ||
|
|
@@ -22,7 +22,7 @@ Use the environment variable `KUBECONFIG` to point to your cluster configuration | |
|
|
||
| ```shell | ||
| export KUBECONFIG=path/to/kubeconfig | ||
| ./openshift-apiserver-tests-ext run <test-name> | ||
| test/extended/tests-extension/bin/openshift-apiserver-tests-ext run <test-name> | ||
| ``` | ||
|
|
||
| ### Local Test using OCP | ||
|
|
@@ -48,25 +48,50 @@ export KUBECONFIG=~/.kube/cluster-bot.kubeconfig | |
|
|
||
| **Example:** | ||
| ```shell | ||
| ./openshift-apiserver-tests-ext run-suite openshift/openshift-apiserver/all | ||
| test/extended/tests-extension/bin/openshift-apiserver-tests-ext run-suite openshift/openshift-apiserver/all | ||
| ``` | ||
|
|
||
| ## Writing Tests | ||
|
|
||
| You can write tests in the `test/extended/` directory. | ||
| You can write tests in the `test/extended/tests-extension/` directory. | ||
|
|
||
| ### Test Module Structure | ||
|
|
||
| This test extension uses a **separate Go module** to isolate test dependencies from the production operator code: | ||
|
|
||
| ``` | ||
| test/extended/tests-extension/ | ||
| ├── go.mod # Separate module with test dependencies | ||
| ├── go.sum | ||
| ├── Makefile # Build targets for test binary | ||
| ├── bin/ # Built test binaries (gitignored) | ||
| │ └── openshift-apiserver-tests-ext | ||
| ├── cmd/openshift-apiserver-tests-ext/ | ||
| │ └── main.go # Test binary entry point | ||
| ├── apiserver/ # Test packages | ||
| │ ├── encryption.go | ||
| │ └── util.go | ||
| └── main.go # Additional test suites | ||
| ``` | ||
|
|
||
|
Comment on lines
58
to
76
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Specify a language for fenced code block (markdownlint MD040) Add a language hint; for directory trees, use "text". -```
+```text
test/extended/tests-extension/
├── go.mod # Separate module with test dependencies
...In test/extended/tests-extension/README.md around lines 58 to 73, the fenced |
||
| **Key Benefits:** | ||
| - Production `go.mod` remains clean (no ginkgo, gomega, test framework dependencies) | ||
| - Test dependencies are isolated in `test/extended/tests-extension/go.mod` | ||
| - Prevents dependency version conflicts between operator and test framework | ||
|
|
||
| ## Development Workflow | ||
|
|
||
| - Add or update tests in: `test/extended/` | ||
| - Run `make build` to build the operator binary and `make tests-ext-build` for the test binary. | ||
| - You can run the full suite or one test using the commands in the table above. | ||
| - Add or update tests in: `test/extended/tests-extension/` | ||
| - When adding new test dependencies, run `go mod tidy` from within `test/extended/tests-extension/` | ||
| - Run `make build` to build the operator binary and `make tests-ext-build` for the test binary | ||
| - You can run the full suite or one test using the commands in the table above | ||
| - Before committing your changes: | ||
| - Run `make tests-ext-update` | ||
| - Run `make tests-ext-update` to update test metadata | ||
| - Run `make verify` to check formatting, linting, and validation | ||
|
|
||
| ## How to Rename a Test | ||
|
|
||
| 1. Run `./openshift-apiserver-tests-ext list` to see the current test names | ||
| 1. Run `test/extended/tests-extension/bin/openshift-apiserver-tests-ext list` to see the current test names | ||
| 2. Find the name of the test you want to rename | ||
| 3. Add a Ginkgo label with the original name, like this: | ||
|
|
||
|
|
@@ -84,16 +109,16 @@ It("should pass a renamed sanity check", | |
|
|
||
| ## How to Delete a Test | ||
|
|
||
| 1. Run `./openshift-apiserver-tests-ext list` to find the test name | ||
| 2. Add the test name to the `IgnoreObsoleteTests` block in `cmd/openshift-apiserver-tests-ext/main.go`, like this: | ||
| 1. Run `test/extended/tests-extension/bin/openshift-apiserver-tests-ext list` to find the test name | ||
| 2. Add the test name to the `IgnoreObsoleteTests` block in `test/extended/tests-extension/cmd/openshift-apiserver-tests-ext/main.go`, like this: | ||
|
|
||
| ```go | ||
| ext.IgnoreObsoleteTests( | ||
| "[sig-openshift-apiserver] My removed test name", | ||
| ) | ||
| ``` | ||
|
|
||
| 3. Delete the test code from your suite. | ||
| 3. Delete the test code from your suite | ||
| 4. Run `make tests-ext-update` to clean the metadata | ||
|
|
||
| **WARNING**: Deleting a test may cause issues with Sippy https://sippy.dptools.openshift.org/sippy-ng/ | ||
|
|
@@ -136,11 +161,11 @@ More info: https://docs.ci.openshift.org/docs/architecture/ci-operator/#testing- | |
| | Target | Description | | ||
| |--------------------------|------------------------------------------------------------------------------| | ||
| | `make build` | Builds the operator binary. | | ||
| | `make tests-ext-build` | Builds the test extension binary. | | ||
| | `make tests-ext-build` | Builds the test extension binary into `test/extended/tests-extension/bin/`. | | ||
| | `make tests-ext-update` | Updates the metadata JSON file and cleans machine-specific codeLocations. | | ||
| | `make verify` | Runs formatting, vet, and linter. | | ||
|
|
||
| **Note:** Metadata is stored in: `.openshift-tests-extension/openshift_payload_openshift-apiserver.json` | ||
| **Note:** Metadata is stored in: `test/extended/tests-extension/.openshift-tests-extension/openshift_payload_openshift-apiserver.json` | ||
|
|
||
| ## FAQ | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Handle empty metadata glob to avoid jq errors.
The loop over
$(METADATA)/*.jsonwill fail if no JSON files exist—the glob expands to a literal string when there are no matches, causing jq to fail. Add a guard to skip non-existent files.Apply this diff to handle the empty glob case:
📝 Committable suggestion
🤖 Prompt for AI Agents