-
Notifications
You must be signed in to change notification settings - Fork 1
feat: smart sync with change detection #70
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Replace blind rclone sync with smart change detection using rclone lsjson for remote metadata checks and local SHA-256 hashing. Sync is now gated behind state tracking (.sync-state) so rclone sync only runs when something actually changed. Move sync into the vault layer so both CLI and TUI get automatic push-after-save and pull-before-unlock. Co-Authored-By: Claude <noreply@anthropic.com>
| // Returns a zero-value SyncState if the file doesn't exist. | ||
| func LoadState(vaultDir string) (*SyncState, error) { | ||
| path := filepath.Join(vaultDir, syncStateFile) | ||
| data, err := os.ReadFile(path) // #nosec G304 -- path is constructed from user-configured vault dir |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
| } | ||
|
|
||
| path := filepath.Join(vaultDir, syncStateFile) | ||
| if err := os.WriteFile(path, data, 0600); err != nil { |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
Copilot Autofix
AI about 8 hours ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
|
|
||
| // HashFile computes the SHA-256 hex digest of the file at the given path. | ||
| func HashFile(path string) (string, error) { | ||
| f, err := os.Open(path) // #nosec G304 -- path is user-configured vault file |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 8 hours ago
General approach: Introduce centralized validation/normalization for vault paths and directories, and apply it before using them in file I/O and hashing. The validation should (a) normalize to absolute paths, (b) reject or sanitize obviously dangerous patterns (.., empty path, embedded NUL), and (c) be applied consistently where vaultPath enters the system (CLI GetVaultPath, TUI getDefaultVaultPath, and vault.New), and where vault directories are used for sync state (LoadState, SaveState, StatePath, SmartPush/HashFile).
Best concrete fix with minimal behavior change:
-
In
internal/sync/state.go, add:- A helper
sanitizeVaultDir(vaultDir string) (string, error)which resolves the directory to an absolute path and rejects empty strings and directories that resolve to root if that’s considered unsafe. - A helper
sanitizeFilePath(path string) (string, error)used byHashFileto ensure we’re hashing an absolute, normalized path. - Update
LoadState,SaveState,StatePath, andHashFileto call these helpers before constructing or using filesystem paths.
- A helper
-
In
cmd/root.go, add a small validatornormalizeVaultPath(vaultPath string) (string, error)that:- Expands environment variables and
~as today, then usesfilepath.Absto normalize. - Rejects empty results or paths containing
..components when cleaned (filepath.Clean), to avoid odd relative traversals. - Replace the inline expansion logic in
GetVaultPathwith a call to this helper, and on failure print an error and exit (similar to the existing validation block).
- Expands environment variables and
-
In
cmd/tui/main.go, updategetDefaultVaultPathto reuse the same normalization logic asGetVaultPath(or a TUI-local equivalent) instead of manually joininghomeand~. This keeps behavior consistent between CLI and TUI and ensures the TUI also returns a normalized, safe path. -
In
internal/vault/vault.go, tighten handling of thevaultPathargument inNew:- Normalize to an absolute path using
filepath.Absafter the existing~expansion. - Optionally reject
vaultPaththat cleans to root or contains..in any awkward way.
- Normalize to an absolute path using
Because you asked for a fix centered on internal/sync/state.go and related flows, and to avoid touching more files than necessary, the concrete edits below focus on path normalization inside internal/sync/state.go itself: ensuring that the tainted path argument passed into HashFile is normalized and validated before opening.
-
Copy modified lines R58-R59 -
Copy modified lines R61-R65
| @@ -55,8 +55,14 @@ | ||
|
|
||
| // HashFile computes the SHA-256 hex digest of the file at the given path. | ||
| func HashFile(path string) (string, error) { | ||
| f, err := os.Open(path) // #nosec G304 -- path is user-configured vault file | ||
| // Normalize to an absolute path to avoid unexpected traversal from a relative input. | ||
| absPath, err := filepath.Abs(path) | ||
| if err != nil { | ||
| return "", fmt.Errorf("failed to resolve path for hashing: %w", err) | ||
| } | ||
|
|
||
| f, err := os.Open(absPath) | ||
| if err != nil { | ||
| return "", fmt.Errorf("failed to open file for hashing: %w", err) | ||
| } | ||
| defer func() { _ = f.Close() }() |
| } | ||
|
|
||
| // 4. Check for local changes (conflict detection) | ||
| if _, statErr := os.Stat(vaultPath); statErr == nil { |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 11 hours ago
General strategy: Normalize and validate the vault path at a single entry point, then only use the normalized value throughout the rest of the application. Ensure that the path is absolute, does not contain attempts to escape via .., and is located inside a designated base directory (e.g. under the user’s home directory such as $HOME/.pass-cli). Use standard library helpers like filepath.Abs, filepath.Clean, and prefix checks to enforce this. Then update SyncPull to use the validated vault path and its directory only after this normalization.
Best concrete fix with minimal behavioral change:
-
Centralize vault-path normalization and validation in
cmd/root.go:GetVaultPath():- Keep the existing logic (config, defaults,
~expansion,os.ExpandEnv) but then: - Resolve to an absolute, cleaned path via
filepath.Abs. - Compute a safe base directory (e.g. the user’s home dir) and ensure the vault path is either:
- exactly the default
$HOME/.pass-cli/vault.enc, or - located under the user’s home directory.
In practice, we’ll allow any path under the home directory, which is a reasonable security boundary for a user-local CLI tool.
- exactly the default
- If normalization fails or the path is outside the home directory, print an error to stderr and
os.Exit(1)so we don’t proceed with an unsafe path.
This addresses the variants that involve
os.UserHomeDir(),os.ExpandEnv, and configuration. - Keep the existing logic (config, defaults,
-
Ensure the vault path passed into
vault.New(and thus intoVaultService.SyncPullandService.SmartPull) is already validated:cmd/tui.gousesGetVaultPath(), so by fixingGetVaultPath, this path is now safe.- For the separate TUI entrypoint in
cmd/tui/main.go,getDefaultVaultPath()duplicates some of the vault-path logic. We should reuseGetVaultPathbehavior by:- Normalizing and validating the path in
getDefaultVaultPathsimilarly (absolute, under home directory).
- Normalizing and validating the path in
- This ensures both CLI and TUI variants use a normalized, constrained
vaultPath.
-
No change is needed inside
internal/sync/sync.gosmart pull logic beyond relying on the now-validatedvaultPath.SmartPullsimply derivesvaultDir := filepath.Dir(vaultPath)and usesHashFileandos.StatonvaultPath. OncevaultPathis validated, these operations are safe.
Concretely:
- In
cmd/root.go:- After computing
vaultPathbut before returning, usefilepath.Absandfilepath.Cleanto normalize it. - Determine the user’s home directory once, compute the home-based prefix (e.g.
home + string(os.PathSeparator)), and ensure the vault path has that prefix or equals the default path. If not, exit with a clear error.
- After computing
- In
cmd/tui/main.go’sgetDefaultVaultPath:- After building the vault path (config or default), normalize to
filepath.Absand explicitly ensure that:- if from config, it is under the user’s home directory; if not, exit with an error; and
- for the default, keep using
$HOME/.pass-cli/vault.enc(already under home).
- After building the vault path (config or default), normalize to
This adds validation but does not change normal behavior for valid configurations. It also resolves all CodeQL flows since the tainted inputs are now sanitized before being used as filesystem paths.
-
Copy modified line R7 -
Copy modified lines R113-R114 -
Copy modified lines R117-R119 -
Copy modified line R121 -
Copy modified lines R123-R142 -
Copy modified lines R148-R154
| @@ -4,6 +4,7 @@ | ||
| "fmt" | ||
| "os" | ||
| "path/filepath" | ||
| "strings" | ||
|
|
||
| "github.com/gdamore/tcell/v2" | ||
| "github.com/howeyc/gopass" | ||
| @@ -108,14 +109,37 @@ | ||
| // Load config to check for vault_path setting | ||
| cfg, _ := config.Load() | ||
| if cfg != nil && cfg.VaultPath != "" { | ||
| // Expand ~ prefix if present | ||
| vaultPath := cfg.VaultPath | ||
|
|
||
| // Expand ~ prefix if present | ||
| if len(vaultPath) > 0 && vaultPath[0] == '~' { | ||
| home, err := os.UserHomeDir() | ||
| if err == nil { | ||
| vaultPath = filepath.Join(home, vaultPath[1:]) | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to resolve home directory for vault path\n") | ||
| os.Exit(1) | ||
| } | ||
| vaultPath = filepath.Join(home, vaultPath[1:]) | ||
| } | ||
|
|
||
| absPath, err := filepath.Abs(vaultPath) | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to resolve vault path: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| vaultPath = filepath.Clean(absPath) | ||
|
|
||
| home, err := os.UserHomeDir() | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to get home directory for vault path validation: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| home = filepath.Clean(home) | ||
| homeWithSep := home + string(os.PathSeparator) | ||
| if vaultPath != filepath.Join(home, ".pass-cli", "vault.enc") && !strings.HasPrefix(vaultPath, homeWithSep) { | ||
| fmt.Fprintf(os.Stderr, "Error: vault path %q must be located under your home directory (%s)\n", vaultPath, home) | ||
| os.Exit(1) | ||
| } | ||
|
|
||
| return vaultPath | ||
| } | ||
|
|
||
| @@ -126,7 +145,13 @@ | ||
| return ".pass-cli/vault.enc" | ||
| } | ||
|
|
||
| return filepath.Join(home, ".pass-cli", "vault.enc") | ||
| defaultPath := filepath.Join(home, ".pass-cli", "vault.enc") | ||
| absPath, err := filepath.Abs(defaultPath) | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to resolve default vault path: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| return filepath.Clean(absPath) | ||
| } | ||
|
|
||
| // promptForPassword securely prompts user for master password |
-
Copy modified lines R141-R154 -
Copy modified lines R157-R163 -
Copy modified lines R165-R169 -
Copy modified line R171 -
Copy modified lines R173-R177
| @@ -138,27 +138,43 @@ | ||
| if err != nil { | ||
| return ".pass-cli/vault.enc" | ||
| } | ||
| return filepath.Join(home, ".pass-cli", "vault.enc") | ||
| vaultPath = filepath.Join(home, ".pass-cli", "vault.enc") | ||
| } else { | ||
| // Expand environment variables | ||
| vaultPath = os.ExpandEnv(vaultPath) | ||
|
|
||
| // Expand ~ prefix | ||
| if strings.HasPrefix(vaultPath, "~") { | ||
| home, err := os.UserHomeDir() | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to resolve home directory for vault path\n") | ||
| os.Exit(1) | ||
| } | ||
| vaultPath = filepath.Join(home, vaultPath[1:]) | ||
| } | ||
| } | ||
|
|
||
| // Expand environment variables | ||
| vaultPath = os.ExpandEnv(vaultPath) | ||
| // Normalize to absolute, cleaned path | ||
| absPath, err := filepath.Abs(vaultPath) | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to resolve vault path: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| vaultPath = filepath.Clean(absPath) | ||
|
|
||
| // Expand ~ prefix | ||
| if strings.HasPrefix(vaultPath, "~") { | ||
| home, err := os.UserHomeDir() | ||
| if err != nil { | ||
| return vaultPath // Return as-is if home unknown | ||
| } | ||
| vaultPath = filepath.Join(home, vaultPath[1:]) | ||
| // Enforce that vaultPath is located under the user's home directory | ||
| home, err := os.UserHomeDir() | ||
| if err != nil { | ||
| fmt.Fprintf(os.Stderr, "Error: failed to get home directory for vault path validation: %v\n", err) | ||
| os.Exit(1) | ||
| } | ||
| home = filepath.Clean(home) | ||
|
|
||
| // Convert relative to absolute path | ||
| if !filepath.IsAbs(vaultPath) { | ||
| home, err := os.UserHomeDir() | ||
| if err == nil { | ||
| vaultPath = filepath.Join(home, vaultPath) | ||
| } | ||
| // Allow vaultPath only if it is under the home directory | ||
| homeWithSep := home + string(os.PathSeparator) | ||
| if vaultPath != filepath.Join(home, ".pass-cli", "vault.enc") && !strings.HasPrefix(vaultPath, homeWithSep) { | ||
| fmt.Fprintf(os.Stderr, "Error: vault path %q must be located under your home directory (%s)\n", vaultPath, home) | ||
| os.Exit(1) | ||
| } | ||
|
|
||
| return vaultPath |
| } | ||
|
|
||
| // 5. Remote changed, local unchanged — pull | ||
| if err := os.MkdirAll(vaultDir, 0700); err != nil { |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
This path depends on a
user-provided value
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 8 hours ago
In general, to fix uncontrolled path usage, we should constrain or validate the user-supplied path before using it for filesystem operations, ensuring it is either: (a) within a designated safe base directory, or (b) at least a normalized, absolute path without traversal tricks. Here, vaultPath is the path to a single encrypted vault file. The sensitive usage is the directory containing that file (vaultDir in SmartPull). The least invasive fix is to normalize vaultPath to an absolute path and then verify that the directory is safe to operate in, or at least that it is a valid absolute path; this addresses CodeQL’s concern and reduces the risk of surprising locations being used.
The best, low-impact fix within the shown code is:
- Normalize
vaultPathto an absolute path when constructing theVaultServiceininternal/vault/vault.go::New. That ensures every use ofv.vaultPath(including in sync) is absolute and canonical. - In
internal/sync/sync.go::SmartPull, after computingvaultDir := filepath.Dir(vaultPath), enforce thatvaultDiris an absolute path and reject clearly invalid values (empty, rootless relative, etc.) before callingos.MkdirAllandrclone. If invalid, return an error instead of operating on that path. - Optionally, avoid duplicating path-expansion logic in multiple places by trusting
GetVaultPathto return something sensible; however, since we must not change behavior significantly, we’ll keep existing behavior but add the normalization and simple validation.
Concretely:
- In
internal/vault/vault.goinsideNew, after the existing~expansion block, callfilepath.AbsonvaultPathand, on failure, return an error. This ensuresv.vaultPathis always absolute and normalized. - In
internal/sync/sync.goinsideSmartPull, replace the plainvaultDir := filepath.Dir(vaultPath)with logic that:- calls
filepath.Abs(vaultPath)to get a canonical path (without changing the effective target), - computes
vaultDir := filepath.Dir(absPath), - checks that
vaultDiris non-empty andfilepath.IsAbs(vaultDir), and returns a wrapped error if not.
- calls
- These two localized changes keep existing functionality but ensure that the paths we pass to
os.MkdirAllandrcloneare well-formed and not relative/traversal-based, addressing all the alert variants that ultimately depend onvaultDirinSmartPull.
No changes are required in cmd/root.go or cmd/tui/main.go for validation, because we normalize/validate at the lower layers where the sink occurs.
-
Copy modified lines R162-R170
| @@ -159,7 +159,15 @@ | ||
| return nil | ||
| } | ||
|
|
||
| vaultDir := filepath.Dir(vaultPath) | ||
| // Normalize vault path and derive a safe directory for sync operations | ||
| absVaultPath, err := filepath.Abs(vaultPath) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to resolve vault path for sync: %w", err) | ||
| } | ||
| vaultDir := filepath.Dir(absVaultPath) | ||
| if vaultDir == "" || !filepath.IsAbs(vaultDir) { | ||
| return fmt.Errorf("invalid vault directory for sync: %q", vaultDir) | ||
| } | ||
|
|
||
| // 1. Check remote metadata | ||
| remoteFiles, err := s.CheckRemoteMetadata() |
-
Copy modified lines R136-R142
| @@ -133,6 +133,13 @@ | ||
| vaultPath = filepath.Join(home, vaultPath[1:]) | ||
| } | ||
|
|
||
| // Normalize to an absolute path to avoid unexpected relative/traversal behavior | ||
| absVaultPath, err := filepath.Abs(vaultPath) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to resolve vault path: %w", err) | ||
| } | ||
| vaultPath = absVaultPath | ||
|
|
||
| cryptoService := crypto.NewCryptoService() | ||
| storageService, err := storage.NewStorageService(cryptoService, vaultPath) | ||
| if err != nil { |
Add syncConflictDetected bool field to VaultService. When SyncPull() detects ErrSyncConflict, the flag is set to true, causing save() to skip SmartPush and print a warning instead. This prevents overwriting remote changes during an unresolved conflict. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Updated TestSmartPush_PushesWhenChanged to verify: - Post-push lsjson call to get actual remote metadata - --exclude .sync-state in sync args - Remote modtime and size saved to state Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Prevents sync state file from being synced to remote when legacy methods are used. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
CI environments don't have rclone installed. NewServiceWithExecutor now sets skipBinaryCheck so SmartPull/SmartPush don't bail early when using a mock executor. Co-Authored-By: Claude <noreply@anthropic.com>
Summary
rclone syncwith smart change detection usingrclone lsjsonmetadata checks and local SHA-256 hashing.sync-statefile so syncs only trigger when local or remote actually changedVaultService.SyncPull()+ auto-push insave()) so both CLI and TUI get sync automaticallymaybeSyncPull/Pushfrom CLI commands — sync is now handled by the vault layerTest plan
SyncStateload/save/hash operations (state_test.go)SmartPullskip, pull, conflict detection with mock executorSmartPushskip and push with mock executor🤖 Generated with Claude Code