Skip to content

Add --eval no-pii named evaluator #59

@hidai25

Description

@hidai25

Summary

Add a built-in evaluator that checks whether the agent's response leaks PII (names, emails, phone numbers, SSNs). Activated with --eval no-pii.

What it should do

Use regex patterns to detect common PII in the agent's response. No LLM needed — this should be fully deterministic and fast.

PII patterns to detect (minimum)

  • Email addresses
  • Phone numbers (US formats)
  • Social Security Numbers
  • Credit card numbers

How to implement

  • Add evalview/evaluators/pii_evaluator.py
  • Follow the pattern of existing evaluators in evalview/evaluators/
  • Hook it into the --eval flag in evalview/cli.py

Acceptance criteria

  • evalview run --eval no-pii works without a rubric in the YAML
  • Test fails when agent response contains an email address or phone number
  • Test passes when response is PII-free

Good first issue notes

No LLM required — pure regex. Well-isolated, single file to create. Great starting point for understanding the evaluator pattern.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions