How to benchmark a GPU box with docker

* Clone the git repo for benchmarking tools to /tmp or your favourite locations

$ git clone https://github.com/tensorflow/benchmarks.git

* Change into the benchmarking folder and enable SELINUX ( This is only if your using RHEL or Centos)

$ cd /tmp/benchmarks
$ chcon -Rt svirt_sandbox_file_t /tmp/benchmarks


* Start your interactive docker container with the tensorflow image. I used this because someone already did the hard work of creating an image with the stress tools

$ docker run -u $(id -u):$(id -g) -it -v /tmp/benchmarks/scripts/tf_cnn_benchmarks:/benchmarks tensorflow/tensorflow:nightly-gpu bash

* Once the docker container is ready. You will be in the container. Now cd to /benchmarks

tf-docker / > cd benchmarks/

* Now run the python tool for single GPU benchmarking

python tf_cnn_benchmarks.py –num_gpus=1 –batch_size=32 –model=resnet50 –variable_update=parameter_server
…….
You will see the results like

Done warm up
Step Img/sec total_loss
1 images/sec: 322.8 +/- 0.0 (jitter = 0.0) 8.169
10 images/sec: 322.5 +/- 0.1 (jitter = 0.5) 7.593
20 images/sec: 322.5 +/- 0.1 (jitter = 0.4) 7.696
30 images/sec: 322.5 +/- 0.1 (jitter = 0.4) 7.753
40 images/sec: 322.5 +/- 0.1 (jitter = 0.3) 8.007
50 images/sec: 322.6 +/- 0.1 (jitter = 0.4) 7.520
60 images/sec: 322.6 +/- 0.1 (jitter = 0.3) 7.989
70 images/sec: 322.6 +/- 0.1 (jitter = 0.4) 8.028
80 images/sec: 322.6 +/- 0.0 (jitter = 0.4) 7.931
90 images/sec: 322.6 +/- 0.0 (jitter = 0.4) 7.851
100 images/sec: 322.6 +/- 0.0 (jitter = 0.4) 7.798
————————————-
total images/sec: 322.49
————————————



How to cleanup docker

#> for id in $( for id in $( docker ps | awk ‘{print $NF}’ | awk -F_ ‘{print $2}’) ; do bjobs $id | grep “not found”; done 2>&1 | awk ‘{print $2}’ | tr -d ‘<>’ ); do docker ps |grep $id | awk ‘{print $1}’ | xargs -n1 docker kill ;done