dmx.compressor.utils.benchmark.measure_model_error

dmx.compressor.utils.benchmark.measure_model_error(model_maker: Callable[[], Tuple[Module, Callable, Callable, device]], modes: List[EVALUATION_MODE], reference_mode: EVALUATION_MODE = EVALUATION_MODE.BASELINE)

Entry function for measuring the error at the output of each layer of the model for each mode relative that layer’s output in the reference mode

Parameters:

• model_maker (Callable[[], Tuple[torch.nn.Module, Callable, Callable, torch.device]]) – A callable that returns the model to be measured, togetherwith some callables to run a sample input through the model or to evaluate the model’s accuracy

• modes (List[EVALUATION_MODE]) – The modes for which we want to evaluate the errors in the layer’s activations

• reference_mode (EVALUATION_MODE) – The reference mode whose layer activations serve as the ground truth used to evaluate the per-layer errors for the other modes