aquin.compute.steer_eval Namespace Reference

Functions
list[str]|None	_as_str_list (Any raw)
float	_rate (int n, int total)
dict[str, Any]	_behavior_summary (list[dict[str, Any]] rows, *, str key)
dict[str, Any]	_custom_summary (list[dict[str, Any]] rows, *, str key)
tuple[list[BehaviorProbe], dict[str, Any]]|None	_resolve_behavior_probes (Any prompts)
dict[str, Any]	run_steer_probe_eval (*, Callable[[str], str] generate_baseline, Callable[[str], str] generate_steered, Any prompts=None, Any reference_answers=None, float threshold=DEFAULT_THRESHOLD, int max_probes=MAX_PROBES)

Variables
int	MAX_PROBES = 50
float	DEFAULT_THRESHOLD = 0.5

Function Documentation

_as_str_list()

list[str] | None _as_str_list ( Any raw )

protected

Definition at line 24 of file steer_eval.py.

Referenced by run_steer_probe_eval().

_behavior_summary()

dict[str, Any] _behavior_summary ( list[dict[str, Any]] rows, * , str key )

protected

Definition at line 82 of file steer_eval.py.

References _rate().

Referenced by run_steer_probe_eval().

_custom_summary()

dict[str, Any] _custom_summary ( list[dict[str, Any]] rows, * , str key )

protected

Definition at line 101 of file steer_eval.py.

References _rate().

Referenced by run_steer_probe_eval().

_rate()

float _rate ( int n, int total )

protected

Definition at line 78 of file steer_eval.py.

Referenced by _behavior_summary(), and _custom_summary().

_resolve_behavior_probes()

tuple[list[BehaviorProbe], dict[str, Any]] | None _resolve_behavior_probes ( Any prompts )

protected

Try deception/behavior probe loading; return None if not applicable.

Definition at line 114 of file steer_eval.py.

Referenced by run_steer_probe_eval().

run_steer_probe_eval()

dict[str, Any] run_steer_probe_eval	(	*	,
		Callable[[str], str]	generate_baseline,
		Callable[[str], str]	generate_steered,
		Any	prompts = None,
		Any	reference_answers = None,
		float	threshold = DEFAULT_THRESHOLD,
		int	max_probes = MAX_PROBES )

Score the same probe set under baseline vs steered generation.

Modes:
  - behavior (default): deception-style probes with honest/deceptive references;
    pass = classified truthful.
  - custom: prompts + reference_answers; pass = keyword overlap >= threshold.

Definition at line 133 of file steer_eval.py.

References _as_str_list(), _behavior_summary(), _custom_summary(), and _resolve_behavior_probes().

Variable Documentation

DEFAULT_THRESHOLD

float aquin.compute.steer_eval.DEFAULT_THRESHOLD = 0.5

Definition at line 21 of file steer_eval.py.

MAX_PROBES

int aquin.compute.steer_eval.MAX_PROBES = 50

Definition at line 20 of file steer_eval.py.