aquin.compute.confidence_analysis Namespace Reference

Functions
dict[str, Any]	_prompt_confidence_metrics (Any model, str text)
dict[str, Any]	_llm_sae_join_row (Any model, str model_id, int layer, str text)
list[dict[str, Any]]	_stressor_summary (list[dict[str, Any]] rows, *, str baseline=_BASELINE_TAG)
dict[str, Any]	_finalize_payload (*, str mode, str model_id, list[dict[str, Any]] probes, list[dict[str, Any]] per_probe, float threshold, bool join_sae, int|None sae_layer)
dict[str, Any]	run_confidence_analysis (Any model, str model_id, list[dict[str, Any]] probes, *, float threshold=DEFAULT_THRESHOLD, bool join_sae=False, int|None sae_layer=None)
dict[str, Any]	run_confidence_analysis_from_args (dict[str, Any] args)
list[dict[str, Any]]	_coerce_probes (Any prompts_raw)
list[dict[str, Any]]	_rows_to_probes (list[Any] rows)

Variables
tuple	_META_KEYS = ("id", "stressor", "lang", "quant_run_id", "condition", "label", "group")
float	DEFAULT_THRESHOLD = 0.40
str	_BASELINE_TAG = "baseline"

Function Documentation

_coerce_probes()

list[dict[str, Any]] _coerce_probes ( Any prompts_raw )

protected

Accept file path, JSON array string, list of strings, or list of probe objects.

Definition at line 228 of file confidence_analysis.py.

References _coerce_probes(), and _rows_to_probes().

Referenced by _coerce_probes(), and run_confidence_analysis_from_args().

_finalize_payload()

dict[str, Any] _finalize_payload ( * , str mode, str model_id, list[dict[str, Any]] probes, list[dict[str, Any]] per_probe, float threshold, bool join_sae, int | None sae_layer )

protected

Definition at line 99 of file confidence_analysis.py.

References _stressor_summary().

Referenced by run_confidence_analysis().

_llm_sae_join_row()

dict[str, Any] _llm_sae_join_row ( Any model, str model_id, int layer, str text )

protected

Definition at line 44 of file confidence_analysis.py.

Referenced by run_confidence_analysis().

_prompt_confidence_metrics()

dict[str, Any] _prompt_confidence_metrics ( Any model, str text )

protected

Definition at line 23 of file confidence_analysis.py.

Referenced by run_confidence_analysis().

_rows_to_probes()

list[dict[str, Any]] _rows_to_probes ( list[Any] rows )

protected

Definition at line 272 of file confidence_analysis.py.

Referenced by _coerce_probes().

_stressor_summary()

list[dict[str, Any]] _stressor_summary ( list[dict[str, Any]] rows, * , str baseline = _BASELINE_TAG )

protected

Definition at line 63 of file confidence_analysis.py.

Referenced by _finalize_payload().

run_confidence_analysis()

dict[str, Any] run_confidence_analysis	(	Any	model,
		str	model_id,
		list[dict[str, Any]]	probes,
		*	,
		float	threshold = DEFAULT_THRESHOLD,
		bool	join_sae = False,
		int | None	sae_layer = None )

Definition at line 139 of file confidence_analysis.py.

References _finalize_payload(), _llm_sae_join_row(), and _prompt_confidence_metrics().

Referenced by run_confidence_analysis_from_args().

run_confidence_analysis_from_args()

dict[str, Any] run_confidence_analysis_from_args

(

dict[str, Any]

args

)

Definition at line 178 of file confidence_analysis.py.

References _coerce_probes(), and run_confidence_analysis().

Variable Documentation

_BASELINE_TAG

str aquin.compute.confidence_analysis._BASELINE_TAG = "baseline"

protected

Definition at line 20 of file confidence_analysis.py.

_META_KEYS

tuple aquin.compute.confidence_analysis._META_KEYS = ("id", "stressor", "lang", "quant_run_id", "condition", "label", "group")

protected

Definition at line 18 of file confidence_analysis.py.

DEFAULT_THRESHOLD

float aquin.compute.confidence_analysis.DEFAULT_THRESHOLD = 0.40

Definition at line 19 of file confidence_analysis.py.