Testing
VersityGW is battle-tested and production-ready. Every pull request must pass our comprehensive test suite before it can be reviewed or merged. All code reviews are done by at least one human in the loop. LLMs may be used to augment the review process, but are never the sole reviewer or decision maker.
Our multi-layered testing strategy includes:
- Go Unit Test Files - Extensive unit tests with race detection and code coverage analysis covering core functionality, edge cases, and error handling.
- Integration Test Scripts - Real-world scenario testing across multiple backends (POSIX, S3, Azure) and configurations.
- Functional/Regression Tests - End-to-end SDK tests validating complete workflows including full-flow operations, POSIX-specific behavior, and IAM functionality populated with regression tests as issues are addressed.
- Static Analysis - Static Analysis checks using staticcheck.
-
System Tests - Protocol-level validation using industry-standard S3 clients:
- AWS CLI - Official AWS command-line tools
- s3cmd - Popular S3 client
- MinIO mc - Modern S3-compatible client
- Direct REST API testing with curl for request/response validation
- Security Testing - Both HTTP and HTTPS configurations tested. Vulnerability scanning with govulncheck. And regular dependency updates with dependabot.
- Compatibility Testing - Multiple backends, versioning scenarios, static bucket modes, and various authentication methods.
Unit tests live along side the code they are testing in a file ending with _test.go. This tells the go complier that these should only be built when running tests. The can be invoked in the top level repo dir with go test ./.... See the official Go docs/tutorial for more info on unit testing within Go.
The rest of the system and integration tests live in the tests/ top level repo directory.
Static analysis checks are run automatically on pull requests using staticcheck as part of the static workflow, and GitHub's CodeQL using the codeql workflow.
These tests live in the tests/integration directory. These are all written in Go, and are built into the versitygw binary. These can be run using the versitygw test sub-command:
% ./versitygw test -h
NAME:
versitygw test - Client side testing command for the gateway
USAGE:
versitygw test [command options]
DESCRIPTION:
The testing CLI is used to test group of versitygw actions.
It also includes some performance and stress testing
COMMANDS:
full-flow Tests the full flow of gateway.
<full test case listing output here>
help, h Shows a list of commands or help for one command
OPTIONS:
--access value, -a value aws user access key [$AWS_ACCESS_KEY_ID, $AWS_ACCESS_KEY]
--secret value, -s value aws user secret access key [$AWS_SECRET_ACCESS_KEY, $AWS_SECRET_KEY]
--endpoint value, -e value s3 server endpoint
--host-style Use host-style bucket addressing (default: false)
--debug, -d enable debug mode (default: false)
--allow-insecure, --ai skip tls verification (default: false)
--help, -h show help
The full-flow runs all of the tests against the specified endpoint. If only one test case is desired, then the test name can be substituted in for full-flow.
Most of these test cases are regression tests that are added as issues are fixed. This ensures the fix was completed, and doesn't get re-introduced with any future changes. The full-flow is run automatically on pull requests in the functional workflow.
The system tests use a bash unit test framework called bats. This allows running cli commands in a way that is easy for automated testing to run setup/test/teardown with common testing output.
The strategy for these tests is to use common cli applications for real world client testing. There is test coverage from aws cli, s3cmd, and mc for general s3 client test coverage. The system tests also have extensive test coverage of request/response interactions using curl. This allows controlled testing of requests and responses that might get ignored by most client applications. We can also validated response structures are similar to AWS with the lower level response validation.
These tests are run on all pull requests using the system workflow. The shellcheck workflow also adds static analysis of the bash tests themselves.
The tests/ README has more information on running the system tests suite.
The vulnerability checks use govulncheck to analyze reported vulnerabilities within dependency packages. More info on govulncheck here: https://go.dev/doc/tutorial/govulncheck.
The tests automatically run on all pull requests using GitHub Actions. You can view the results of all of this repo's actions here. More useful, however, are the links within the specific pull request. Navigate to the repo pull requests, find the pull request of interest and click on that. Then scroll down towards the end of the conversations, and there will be a tests results box. Within the box the passing tests will have a green checkmark and the failed tests will have a red x. Click on any test to get the detailed action output. For many tests there can be a lot of output. So if you are looking for a failed test case, it can be useful to search for "FAIL" within the test output.
No pull request will be merged without all tests passing. There can be false positives within tests at times, and GitHub has buttons that can re-trigger the failed jobs if they need to be re-run. This is typically handled by the reviewer or maintainer.
Typically code reviewers will want tests passing before taking time to review the code. This is because failing test cases likely indicate there are problems that need to be resolved with the change before any review can to be started. It is fine to add comments asking for help if the failing test cases are not clear or need to be changed due to incorrect test assumptions.
