Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

detector: StringDetector matchtype fullmatch #1084

Open
leondz opened this issue Jan 17, 2025 · 0 comments
Open

detector: StringDetector matchtype fullmatch #1084

leondz opened this issue Jan 17, 2025 · 0 comments
Labels
detectors work on code that inherits from or manages Detector new plugin Describes an entirely new probe, detector, generator or harness

Comments

@leondz
Copy link
Collaborator

leondz commented Jan 17, 2025

Summary

Detector for if generator output is a specific string

Add configurable option to strip() generator output first

Add configurable option to still match even if output truncates before entire match is found (to handle max_tokens clipping and similar)

Useful for e.g. shield models, or probes that specify a complete expectedoutput string
@leondz leondz added detectors work on code that inherits from or manages Detector new plugin Describes an entirely new probe, detector, generator or harness labels Jan 17, 2025
@leondz leondz changed the title detector: fullmatch detector: StringDetector matchtype fullmatch Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
detectors work on code that inherits from or manages Detector new plugin Describes an entirely new probe, detector, generator or harness
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@leondz