-
Notifications
You must be signed in to change notification settings - Fork 8
Open
Labels
good first issueGood for newcomersGood for newcomershelp wantedExtra attention is neededExtra attention is needed
Description
Summary
Add a built-in evaluator that checks whether the agent's response leaks PII (names, emails, phone numbers, SSNs). Activated with --eval no-pii.
What it should do
Use regex patterns to detect common PII in the agent's response. No LLM needed — this should be fully deterministic and fast.
PII patterns to detect (minimum)
- Email addresses
- Phone numbers (US formats)
- Social Security Numbers
- Credit card numbers
How to implement
- Add
evalview/evaluators/pii_evaluator.py - Follow the pattern of existing evaluators in
evalview/evaluators/ - Hook it into the
--evalflag inevalview/cli.py
Acceptance criteria
evalview run --eval no-piiworks without a rubric in the YAML- Test fails when agent response contains an email address or phone number
- Test passes when response is PII-free
Good first issue notes
No LLM required — pure regex. Well-isolated, single file to create. Great starting point for understanding the evaluator pattern.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
good first issueGood for newcomersGood for newcomershelp wantedExtra attention is neededExtra attention is needed