Skip to content

exporters: support reading keys from env vars (fixes #97)#134

Open
Kunal-Darekar wants to merge 2 commits intom-lab:mainfrom
Kunal-Darekar:fix/issue-97-env-var-exporter-keys
Open

exporters: support reading keys from env vars (fixes #97)#134
Kunal-Darekar wants to merge 2 commits intom-lab:mainfrom
Kunal-Darekar:fix/issue-97-env-var-exporter-keys

Conversation

@Kunal-Darekar
Copy link

@Kunal-Darekar Kunal-Darekar commented Mar 1, 2026

Fixes #97

Both GCSExporter and SCPExporter currently require a key file to exist
on disk. This is a problem in containerised setups where you don't want
to bake credentials into the image or deal with volume mounts just for
a key file.

The fix is straightforward: if an environment variable is set, use that
instead of the file path. The original file-path behaviour is untouched
so nothing breaks for existing deployments.

GCSExporter — reads MURAKAMI_GCS_KEY_CONTENT. If set, it's
expected to be the service account JSON, base64-encoded with no
newlines. The exporter decodes it and calls
from_service_account_info() instead of from_service_account_json().
If the variable isn't set, it falls back to the key config value as
before.

SCPExporter — reads MURAKAMI_SCP_KEY_CONTENT. If set, it's the
PEM private key, base64-encoded. Since paramiko's key_filename=
expects a file path (not raw bytes), the decoded key is written to a
temp file, passed to ssh.connect(), and deleted in the finally
block once the connection closes — even if the transfer fails.

I also added utilities/encode_key.sh which just runs
base64 -w 0 "$1" on a key file and prints the result along with the
env var name to set. This matches what was suggested in the issue
comments.

Testing — added two test files covering both code paths for each
exporter (env var present, env var absent). All network and GCS calls
are mocked.

To use:

# GCS
export MURAKAMI_GCS_KEY_CONTENT=$(base64 -w 0 /path/to/sa.json)

# SCP
export MURAKAMI_SCP_KEY_CONTENT=$(base64 -w 0 /path/to/id_rsa)
image

This change is Reviewable

GCSExporter and SCPExporter previously required a private key file to
exist on disk, making them impractical in containerised deployments
where baking credentials into an image is undesirable.

- GCSExporter.__init__: if MURAKAMI_GCS_KEY_CONTENT is set, treat it
  as a base64-encoded service-account JSON string, decode it, and use
  storage.Client.from_service_account_info() instead of the file-path
  path.  The existing 'key' config option is unchanged (backward compat).

- SCPExporter.__init__: read MURAKAMI_SCP_KEY_CONTENT once at init.
  SCPExporter._push_single: if the env var is set, decode the base64
  PEM, write it to a NamedTemporaryFile, pass that path to
  key_filename=, and delete the temp file in the finally block after
  ssh.close().  The existing 'key' config option is unchanged.

- utilities/encode_key.sh: helper script that base64-encodes any key
  file (-w 0, no newlines) and prints the env var instructions.

- tests/test_gcs_exporter.py, tests/test_scp_exporter.py: cover both
  the new env-var path and the original file-path path for each exporter.
@Kunal-Darekar Kunal-Darekar changed the title exporters: read key from env var for GCS and SCP (issue #97) exporters: support reading keys from env vars (fixes #97) Mar 1, 2026
Copy link
Collaborator

@robertodauria robertodauria left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. The feature itself is reasonable and matches what was discussed in #97. A few things before we can merge:

  • Coordination: As noted in #135, please coordinate before opening PRs. This happens to align with what we want, but the process matters - especially when you're new to a project.
  • AI disclosure: your other PR disclosed Copilot usage, this one doesn't. If AI tools were used, please disclose that per our policy. If they weren't no action needed.
  • Fundamental design issue. Murakami already has a generic mechanism for setting any config key via environment variables. The load_env() function in __main__.py parses MURAKAMI_* env vars into a nested config dict. For example, MURAKAMI_EXPORTERS_SCP_KEY=/path/to/key automatically populates config["key"] in the SCP exporter - no special code needed. This PR bypasses that mechanism entirely by hardcoding os.environ.get("MURAKAMI_SCP_KEY_CONTENT") and os.environ.get("MURAKAMI_GCS_KEY_CONTENT") directly in the exporters. This creates a parallel config path that only works via env vars and is invisible to the TOML config and the rest of the config system.

I recommend adding a config option to the exporters and let the existing env var mechanism handle it. This way it works via TOML config, env vars, or CLI, just like every other config option in the project.

Adding more technical feedback inline.

key_filename = self.private_key
if self._key_content is not None:
logger.debug("SCP: loading key from MURAKAMI_SCP_KEY_CONTENT")
tmp_key_file = tempfile.NamedTemporaryFile(delete=False)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explicitly set the permissions to something like 0o600

@Kunal-Darekar
Copy link
Author

Thanks for review.

Sorry for not coordinating before opening the PR. I understand that I should discuss design decisions first, especially since I’m new to the project. I’ll make sure to follow the process properly going forward.

I’ve addressed the requested changes:

change code to use the existing config mechanism instead of directly reading os.environ

Added error handling for base64 and JSON decoding

Set explicit 0o600 permissions for the temporary key file

Pushed the updated changes

Please let me know if anything else needs to be improved.

Also, if there are other issues you think I can work on, I’d be happy to take a look.

once again thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Read env variables for exporter keys

2 participants