> For the complete documentation index, see [llms.txt](https://zjunlp.gitbook.io/easyedit/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://zjunlp.gitbook.io/easyedit/easyeditor/evaluate.md).

# Evaluate

> Contain evaluation utilities for pytorch-based rewriting methods. To use, simply call `compute_edit_quality` with the appropriate arguments, which returns a dictionary containing them.

#### compute\_rewrite\_or\_rephrase\_quality() -> Dict

> evaluation method for `Reliability` and `Generalization`

* Conduct a computation to identify the token that possesses the highest probability at each respective token position. 
* Subsequently, compare this calculated result with the established ground truth to ascertain the mean accuracy of the probabilistic model.

```python
def compute_rewrite_or_rephrase_quality(
    model,
    model_name,
    hparams: HyperParams,
    tok: AutoTokenizer,
    prompt: str,
    target_new: str,
    device,
    test_rephrase: bool = False
) -> typing.Dict:
```

* **Paramters**
  * model(<mark style="color:purple;">PreTrainedModel</mark>): model to be edited
  * modle\_name(<mark style="color:purple;">Str</mark>): model\_name\_or\_path
  * hparams(<mark style="color:purple;">Hyperparams</mark>): hyperparameters for editing method
  * tok(<mark style="color:purple;">PreTrainedTokenizer</mark>): tokenizer for inputs
  * prompt(<mark style="color:purple;">Str</mark>): the edit descriptor
  * target\_new(<mark style="color:purple;">Str</mark>): the edit target
  * test\_rephrase(<mark style="color:purple;">bool</mark>): whether to evalute the rephrase prompt(For Generalization)
* **Return Type**
  * metrics(<mark style="color:purple;">Dict</mark>): model weights after editing

#### compute\_locality\_quality() -> Dict

> the input-ouput format is same as `compute_rewrite_or_rephrase_quality`

* Conduct a computation to identify the token that possesses the highest probability at each respective token position.&#x20;
* Compare whether the output tokens before and after editing are the same, and calculate the average accuracy rate

### Example

* metrics

```json
{
    "post": {
        "rewrite_acc": ,
        "rephrase_acc": ,
        "locality": {
            "YOUR_LOCALITY_KEY": ,
            //...
        },
        "portablility": {
            "YOUR_PORTABILITY_KEY": ,
            //...
        },
    },
    "pre": {
        "rewrite_acc": ,
        "rephrase_acc": ,
        "portablility": {
            "YOUR_PORTABILITY_KEY": ,
            //...
        },
    }
}
```