# CmdCaliper: 
## CmdDataset
The similar command-line dataset is stored in the `./data` directory, and it has the following structure:
```
- data\
    - train_data.json # 28,520 similar command-line pairs in total.
    - test_data.json # 2,807 similar command-line pairs in total.
```

The structure of the training file is a list of list, which contains contextual similar command-line pair as:
```
[
    [similar_command_1, similar_command_2],
    [similar_command_1, similar_command_2],
    [similar_command_1, similar_command_2],
]
```
The structure of the testing file is a list of dict, which contains contextual similar command-line pair and a negative sample index list as:
```
[
    {
        "query_cmd": similar_command_1, 
        "positive_cmd": similar_command_2,
        "negative_index_list": [negative_index_1, negative_index_2,....]
    },
]
```

## Model Zoo
- Download the checkpoints from Google Drive ([here](https://drive.google.com/drive/folders/1nx0MeaH5iDqhrbtfoFohu7KRyz5AZIkf?usp=sharing))

|       Model       | Params (B) | MRR@10 | Top@10 |
|:-----------------:|:----------:|:------:|:-------:|
| CmdCaliper-Small|    0.03   |  87.78 | 94.76 | 
| CmdCaliper-Base |    0.11   |  88.47 | 95.26 | 
| CmdCaliper-Large|    0.335   |  89.11 | 95.65 |


## Requirements
```
pip3 install -r requirements.txt
```

## Getting Started
### Training
The CmdCaliper was trained using an internal training framework based on PyTorch. Due to certain constraints, we are currently unable to make this framework publicly available. However, we are pleased to provide the training script, entirely written in PyTorch, in the final camera-ready version of this paper.

### Evaluating
Execute the following command line to reproduce the performance of `gte-small`, `cmdcaliper-small`, `cmdcaliper-base`, and `cmdcaliper-large` evaluated on the testing set of our proposed dataset reported in paper.
#### Evaluate CmdCaliper
```
python3 evaluate.py --path-to-test-data ./data/test_data.json \
                    --model-name ["cmdcaliper-small"/"cmdcaliper-base"/"cmdcaliper-large"] \
                    --path-to-checkpoint [Path to checkpoint downloaded from Model Zoo] \
                    --batch-size 16
```
#### Evaluate Benchmark Baselines
```
python3 evaluate.py --path-to-test-data ./data/test_data.json --model-name "thenlper/gte-small" --batch-size 16
```


