Benchmarks
Compare zcls2 and apex training
Requirements
* Model: Torchvision Pretrained ResNet18
* Dataset: CIFAR10/CIFAR100/FashionMNIST
* Train:
* Epoch: 90
* Num gpus: 4
* Batch size: 256 / one gpu
* Loss: CrossEntropyLoss
* Optimizer: SGD (initial lr 0.01)
* Lr_scheduler: Warmup (5) + MultiStepLR(30/60/80)
* apex: commit `727a6452c9b781930acee5e24e09efe9360b4890`
* zcls2: commit `193b3995c37578222957ac9e72f6ad9d49438b70`
Prepare data
Go to the ./demo/benchmarks/datasets
folder, run
bash run.sh
This script will download torchvision data and extract to ./data/
folder.
ZCls2/data$ tree -L 2
.
├── cifar10
│ ├── cifar-10-batches-py
│ ├── cifar-10-python.tar.gz
│ ├── train
│ └── val
├── cifar100
│ ├── cifar-100-python
│ ├── cifar-100-python.tar.gz
│ ├── train
│ └── val
├── cifar-10-python.tar.gz
└── fashionmnist
├── FashionMNIST
│ ├── processed
│ └── raw
├── train
│ ├── Ankle boot
│ ├── Bag
│ ├── Coat
│ ├── Dress
│ ├── Pullover
│ ├── Sandal
│ ├── Shirt
│ ├── Sneaker
│ ├── Trouser
│ └── T-shirt
└── val
├── Ankle boot
├── Bag
├── Coat
├── Dress
├── Pullover
├── Sandal
├── Shirt
├── Sneaker
├── Trouser
└── T-shirt
Results
repos | arch | dataset | top1 | top5 |
---|---|---|---|---|
apex | resnet18 | cifar10 | 92.910 | 99.800 |
zcls2 | resnet18 | cifar10 | 92.490 | 99.800 |
apex | resnet18 | cifar100 | 73.400 | 93.000 |
zcls2 | resnet18 | cifar100 | 73.260 | 92.910 |
apex | resnet18 | fashionmnist | 94.230 | 99.950 |
zcls2 | resnet18 | fashionmnist | 94.250 | 99.940 |
apex | mobilenetv2 | cifar10 | 92.800 | 99.850 |
zcls2 | mobilenetv2 | cifar10 | 92.830 | 99.800 |
apex | mobilenetv2 | cifar100 | 73.560 | 93.140 |
zcls2 | mobilenetv2 | cifar100 | 73.610 | 93.320 |
apex | mobilenetv2 | fashionmnist | 93.870 | 99.980 |
zcls2 | mobilenetv2 | fashionmnist | 93.860 | 99.960 |
I don't set cudnn.deterministic = True
and cudnn.benchmark = False
, so each time the best_prec@1/best_prec@5
is different, may be big diff. For example,
CIFAR10
- ResNet18
- Apex
92.910 | 99.800
92.390 | 99.780
- ZCls2
92.410 | 99.770
92.490 | 99.800
- MobileNetV2
- Apex
92.800 | 99.850
- ZCls2
92.830 | 99.800
CIFAR100
- ResNet18
- Apex
73.250 | 92.890
73.400 | 93.000
- ZCls2
73.190 | 93.080
73.260 | 92.910
- MobileNetV2
- Apex
73.560 | 93.140
- ZCls2
73.610 | 93.320
FashionMNIST
- ResNet18
- Apex
94.230 | 99.950
94.060 | 99.950
- ZCls2
93.920 | 99.930
94.250 | 99.940
- MobileNetV2
- Apex
93.870 | 99.980
- ZCls2
93.860 | 99.960