exporters: support reading keys from env vars (fixes #97)#134
exporters: support reading keys from env vars (fixes #97)#134Kunal-Darekar wants to merge 2 commits intom-lab:mainfrom
Conversation
GCSExporter and SCPExporter previously required a private key file to exist on disk, making them impractical in containerised deployments where baking credentials into an image is undesirable. - GCSExporter.__init__: if MURAKAMI_GCS_KEY_CONTENT is set, treat it as a base64-encoded service-account JSON string, decode it, and use storage.Client.from_service_account_info() instead of the file-path path. The existing 'key' config option is unchanged (backward compat). - SCPExporter.__init__: read MURAKAMI_SCP_KEY_CONTENT once at init. SCPExporter._push_single: if the env var is set, decode the base64 PEM, write it to a NamedTemporaryFile, pass that path to key_filename=, and delete the temp file in the finally block after ssh.close(). The existing 'key' config option is unchanged. - utilities/encode_key.sh: helper script that base64-encodes any key file (-w 0, no newlines) and prints the env var instructions. - tests/test_gcs_exporter.py, tests/test_scp_exporter.py: cover both the new env-var path and the original file-path path for each exporter.
robertodauria
left a comment
There was a problem hiding this comment.
Thanks for the PR. The feature itself is reasonable and matches what was discussed in #97. A few things before we can merge:
- Coordination: As noted in #135, please coordinate before opening PRs. This happens to align with what we want, but the process matters - especially when you're new to a project.
- AI disclosure: your other PR disclosed Copilot usage, this one doesn't. If AI tools were used, please disclose that per our policy. If they weren't no action needed.
- Fundamental design issue. Murakami already has a generic mechanism for setting any config key via environment variables. The load_env() function in
__main__.pyparsesMURAKAMI_*env vars into a nested config dict. For example,MURAKAMI_EXPORTERS_SCP_KEY=/path/to/keyautomatically populatesconfig["key"]in the SCP exporter - no special code needed. This PR bypasses that mechanism entirely by hardcodingos.environ.get("MURAKAMI_SCP_KEY_CONTENT")andos.environ.get("MURAKAMI_GCS_KEY_CONTENT")directly in the exporters. This creates a parallel config path that only works via env vars and is invisible to the TOML config and the rest of the config system.
I recommend adding a config option to the exporters and let the existing env var mechanism handle it. This way it works via TOML config, env vars, or CLI, just like every other config option in the project.
Adding more technical feedback inline.
| key_filename = self.private_key | ||
| if self._key_content is not None: | ||
| logger.debug("SCP: loading key from MURAKAMI_SCP_KEY_CONTENT") | ||
| tmp_key_file = tempfile.NamedTemporaryFile(delete=False) |
There was a problem hiding this comment.
Please explicitly set the permissions to something like 0o600
|
Thanks for review. Sorry for not coordinating before opening the PR. I understand that I should discuss design decisions first, especially since I’m new to the project. I’ll make sure to follow the process properly going forward. I’ve addressed the requested changes: change code to use the existing config mechanism instead of directly reading os.environ Added error handling for base64 and JSON decoding Set explicit 0o600 permissions for the temporary key file Pushed the updated changes Please let me know if anything else needs to be improved. Also, if there are other issues you think I can work on, I’d be happy to take a look. once again thank you |
Fixes #97
Both GCSExporter and SCPExporter currently require a key file to exist
on disk. This is a problem in containerised setups where you don't want
to bake credentials into the image or deal with volume mounts just for
a key file.
The fix is straightforward: if an environment variable is set, use that
instead of the file path. The original file-path behaviour is untouched
so nothing breaks for existing deployments.
GCSExporter — reads
MURAKAMI_GCS_KEY_CONTENT. If set, it'sexpected to be the service account JSON, base64-encoded with no
newlines. The exporter decodes it and calls
from_service_account_info()instead offrom_service_account_json().If the variable isn't set, it falls back to the
keyconfig value asbefore.
SCPExporter — reads
MURAKAMI_SCP_KEY_CONTENT. If set, it's thePEM private key, base64-encoded. Since paramiko's
key_filename=expects a file path (not raw bytes), the decoded key is written to a
temp file, passed to
ssh.connect(), and deleted in thefinallyblock once the connection closes — even if the transfer fails.
I also added
utilities/encode_key.shwhich just runsbase64 -w 0 "$1"on a key file and prints the result along with theenv var name to set. This matches what was suggested in the issue
comments.
Testing — added two test files covering both code paths for each
exporter (env var present, env var absent). All network and GCS calls
are mocked.
To use:
This change is