anthony.wen
fa97ce5ad0
- change ATVM status formatting to the approved Markdown-table template with SUMMARY:, HOSTS:, TIMING:, and NOTES: - document that normal status requests print locally only unless explicitly asked to send to Mattermost - document Mattermost defaults and posting rules, including only sending after full run completion - document the controller-side systemd watcher design for future automation - add the secrets migration/cleanup review doc - ignore .env.credentials.local in git and reflect the move toward using that local credentials file instead of hardcoded secrets
189 lines
6.6 KiB
Markdown
189 lines
6.6 KiB
Markdown
# Secrets Migration And Cleanup
|
|
|
|
## Purpose
|
|
This document explains:
|
|
- whether the workspace can be cleaned up to stop storing credentials and tokens in tracked files
|
|
- how `.env.credentials.local` should be used
|
|
- what has to happen to remove already-committed secrets from git history and the remote repository
|
|
|
|
## 1. Can the workspace be cleaned up to stop referencing raw secrets in tracked files?
|
|
Yes.
|
|
|
|
The intended cleanup is:
|
|
- remove hardcoded credentials, API tokens, webhook URLs, and similar secrets from tracked docs and files
|
|
- replace those values with references to `/home/aw/code/cds/.env.credentials.local`
|
|
- keep only non-secret metadata in tracked files, such as:
|
|
- hostnames
|
|
- IPs
|
|
- usernames when acceptable
|
|
- variable names
|
|
- usage instructions
|
|
|
|
Examples of what tracked docs should say instead of storing raw values:
|
|
- `Use ATVM_CONTROLLER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
|
- `Use VCENTER_USER and VCENTER_PASSWORD from /home/aw/code/cds/.env.credentials.local`
|
|
- `Use MATTERMOST_ATVM_WEBHOOK from /home/aw/code/cds/.env.credentials.local`
|
|
|
|
Recommended scope of cleanup:
|
|
- `atvm/inventory/accounts-and-credentials.md`
|
|
- `atvm/inventory/infrastructure.md`
|
|
- any other tracked docs or scripts that contain:
|
|
- passwords
|
|
- API tokens
|
|
- TOTP secrets
|
|
- webhook URLs
|
|
- install codes or secrets that should not remain in git
|
|
|
|
## 2. What do I need to do for the assistant to use `.env.credentials.local`?
|
|
The file exists on disk, but the assistant does not automatically import shell environment files unless one of the following is done:
|
|
|
|
### Option A: Explicitly source it in the shell session
|
|
Example:
|
|
|
|
```bash
|
|
source /home/aw/code/cds/.env.credentials.local
|
|
```
|
|
|
|
This is the simplest and most reliable option for interactive terminal work.
|
|
|
|
### Option B: Scripts explicitly read it
|
|
A script can do:
|
|
|
|
```bash
|
|
source /home/aw/code/cds/.env.credentials.local
|
|
```
|
|
|
|
before using any secret-backed variables.
|
|
|
|
### Option C: The workflow documentation tells the assistant to load it
|
|
The workspace docs can instruct the assistant to use `/home/aw/code/cds/.env.credentials.local` when credentials are required, but the assistant still needs an execution path that actually loads those variables into the shell or reads them directly from the file.
|
|
|
|
## Practical rule
|
|
If you want the assistant to reliably use these values during execution, the safest approach is:
|
|
- either explicitly source the file first
|
|
- or instruct the assistant to source it as part of the command/script it runs
|
|
|
|
## Important limitation
|
|
The existence of `.env.credentials.local` does not automatically make every shell command aware of those variables.
|
|
|
|
The assistant needs one of these:
|
|
- the current shell environment already contains the exported variables
|
|
- the command explicitly sources the file
|
|
- the script being executed explicitly sources the file
|
|
|
|
## 3. What do I need to do if secrets were already committed and pushed to the remote repository?
|
|
If secrets were already committed to git history and pushed, `.gitignore` does not fix that.
|
|
|
|
You need to treat those secrets as exposed.
|
|
|
|
## Required response
|
|
Do these in this order:
|
|
|
|
### Step 1: Rotate or revoke the exposed secrets
|
|
This is the most important step.
|
|
|
|
Examples:
|
|
- regenerate Mattermost webhook URLs
|
|
- replace API tokens
|
|
- rotate passwords
|
|
- regenerate TOTP/shared secrets if applicable
|
|
- replace any service registration or install tokens that should be considered exposed
|
|
|
|
Even if you later remove them from git history, assume they were already copied.
|
|
|
|
### Step 2: Remove secrets from the current tracked files
|
|
Edit the tracked docs and scripts so they no longer contain raw secrets.
|
|
|
|
Replace them with:
|
|
- references to `.env.credentials.local`
|
|
- redacted placeholders
|
|
- variable names
|
|
|
|
### Step 3: Rewrite git history to remove the secrets from all commits
|
|
This is a history-rewrite operation.
|
|
|
|
Typical tools:
|
|
- `git filter-repo` (preferred)
|
|
- BFG Repo-Cleaner
|
|
|
|
High-level workflow:
|
|
1. identify all tracked files and literal secrets that must be removed
|
|
2. rewrite repository history to remove or replace them
|
|
3. verify the secrets no longer exist in any commit
|
|
4. force-push the rewritten history to the remote
|
|
|
|
### Step 4: Force-push the cleaned history
|
|
After rewriting history, the remote must be updated with a force push.
|
|
|
|
That usually means:
|
|
- `git push --force-with-lease origin <branch>`
|
|
|
|
### Step 5: Coordinate with anyone else using the repo
|
|
Anyone with an old clone will still have the old history unless they reset or reclone.
|
|
|
|
They need instructions to:
|
|
- stop using the old history
|
|
- fetch the rewritten branch
|
|
- hard reset or reclone as appropriate
|
|
|
|
## Important caution about remote cleanup
|
|
Cleaning the git remote history does not guarantee that every copy is gone.
|
|
|
|
Secrets may still exist in:
|
|
- old clones
|
|
- forks
|
|
- CI logs
|
|
- code review systems
|
|
- backups
|
|
- screenshots or pasted chat logs
|
|
|
|
That is why secret rotation must happen first.
|
|
|
|
## Recommended cleanup policy for this workspace
|
|
For this workspace, the correct policy should be:
|
|
- keep real secrets only in `/home/aw/code/cds/.env.credentials.local`
|
|
- keep that file gitignored
|
|
- remove raw secrets from tracked docs
|
|
- document variable names and usage instead of values
|
|
- rotate any secrets that were ever committed
|
|
- rewrite history if the repository should no longer retain those secret values
|
|
|
|
## Proposed next implementation work
|
|
When approved, the cleanup work would likely be:
|
|
1. inventory all tracked files containing secrets
|
|
2. patch those files to reference `.env.credentials.local`
|
|
3. update docs so the credential source is explicit
|
|
4. prepare a history-rewrite plan
|
|
5. prepare exact git commands for review before any destructive git action
|
|
|
|
## Git-history cleanup note
|
|
History rewriting is disruptive and should not be done casually.
|
|
|
|
Before doing it, prepare:
|
|
- the list of files and secrets to purge
|
|
- the exact rewrite tool and command
|
|
- the exact verification commands
|
|
- the exact force-push command
|
|
- the operator communication plan for other users of the repo
|
|
|
|
## Summary
|
|
Answers to the three direct questions:
|
|
|
|
### Question 1
|
|
Yes, the workspace can be cleaned up to stop storing secrets in tracked files and instead reference `/home/aw/code/cds/.env.credentials.local`.
|
|
|
|
### Question 2
|
|
To have the assistant reliably use `.env.credentials.local`, either:
|
|
- explicitly source it
|
|
- or ensure the script/command being run sources it
|
|
|
|
The assistant does not automatically inherit its contents just because the file exists.
|
|
|
|
### Question 3
|
|
If secrets were already committed and pushed:
|
|
- rotate them first
|
|
- remove them from current files
|
|
- rewrite git history
|
|
- force-push the cleaned history
|
|
- coordinate with anyone else who has a clone
|