Score

The Score dataclass is the return type required by all optimizer metrics. It provides structured feedback that optimizers use to learn from successes and failures.

from vizpy.optimizers.core import Score

Definition

@dataclass
class Score:
    value: float          # 0-1 normalized score
    is_success: bool      # Meets success threshold
    feedback: str = ""    # Human-readable explanation
    error_type: str = ""  # Categorized error type
    metadata: dict = {}   # Extra data (not used by optimizer)

Fields

Field	Type	Description
`value`	`float`	Normalized score between 0 and 1.
`is_success`	`bool`	Whether this prediction meets your success threshold.
`feedback`	`str`	Human-readable explanation of the score. More specific feedback produces better optimization results. Instead of `"Wrong"`, prefer `"Expected a 3-digit number but got a sentence."`
`error_type`	`str`	Optional categorization of the error (e.g., `"wrong_format"`, `"missing_info"`). Enables diversity-aware optimization features like stratified sampling and error clustering.
`metadata`	`dict`	Arbitrary extra data. Not used by the optimizer but available for your own tracking.

Example

from vizpy.optimizers.core import Score
 
def my_metric(example: dict, prediction) -> Score:
    correct = prediction.answer.strip() == example["gold"].strip()
    return Score(
        value=1.0 if correct else 0.0,
        is_success=correct,
        feedback="Correct" if correct else f"Expected '{example['gold']}', got '{prediction.answer}'",
        error_type="" if correct else "wrong_answer",
    )

Score

Definition

Fields

Example

On this page