aquin.compute.calibration Namespace Reference

Functions
float	ece_proxy (list[float] confs)
list[float]	mean_token_confidence (Any model, Any tokenizer, str device, list[str] prompts)
dict	run_calibration_for_models (*, Any base_model, Any ft_model, Any tokenizer, str device, list[str] eval_prompts, list[str]|None eval_categories=None, list[str]|None ft_outputs=None, float threshold=0.40)

Function Documentation

ece_proxy()

float ece_proxy

(

list[float]

confs

)

Scalar ECE proxy: mean |confidence - 0.5| in [0, 0.5]. Lower = less peaked.

Definition at line 16 of file calibration.py.

Referenced by run_calibration_for_models().

mean_token_confidence()

list[float] mean_token_confidence	(	Any	model,
		Any	tokenizer,
		str	device,
		list[str]	prompts )

Definition at line 23 of file calibration.py.

Referenced by run_calibration_for_models().

run_calibration_for_models()

dict run_calibration_for_models	(	*	,
		Any	base_model,
		Any	ft_model,
		Any	tokenizer,
		str	device,
		list[str]	eval_prompts,
		list[str] | None	eval_categories = None,
		list[str] | None	ft_outputs = None,
		float	threshold = 0.40 )

Return calibration payload matching web CalibrationPayload / simulate event.

Definition at line 38 of file calibration.py.

References ece_proxy(), and mean_token_confidence().