Setup TensorFlow

Here are things you must do to install tensorflow. In this wiki, we have instructions for an Ubuntu machine and a Mac. The Mac instructions are not complete.

Linux installation

We installed tensorflow on the vistalab virtual machine on GCP. We had to do some manual work.

The main installation work flow was taken from this web-page

The first two sections worked without any real problem.

# For CPU
pip install tensorflow
sudo apt-get install protobuf-compiler python-pil python-lxml python-tk
sudo pip install Cython
sudo pip install jupyter
sudo pip install matplotlib

Then we did the cocoapi installation

git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
make
cp -r pycocotools <path_to_tensorflow>/models/research/

We then had some challenges with the protobuf installation. There was an issues thread we found from the error that told us we needed to have protoc3 (we had 2.5) and how to do the installation. We put the new protoc source code in /usr/local/share, and we replaced the /usr/bin/protoc with links to the protoc3 installation. Then the following worked

# After installing protoc3 properly,
# Execute this line from tensorflow/models/research/
protoc object_detection/protos/*.proto --python_out=.

We added this line to the .bashrc, which is personalized for wandell. You should change the directory to your own TFLOWMODELS

# From tensorflow/models/research/
TFLOWMODELS="/home/wandell/Documents/MATLAB/tensorflow/models/research"
#echo $TFLOWMODELS

# We are worried that there is an extra colon at the front of this.  Maybe we sh
ould check the PYTHONPATH variable
export PYTHONPATH=$PYTHONPATH:$TFLOWMODELS:$TFLOWMODELS'/slim'

This test ran correctly. The first time, though, it built the matplotlib fonts. Subsequent tests do not print out the patplotlib message.

python object_detection/builders/model_builder_test.py
/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
...............
----------------------------------------------------------------------
Ran 15 tests in 0.073s

OK

MAC installation

You may need to install the Mac xcode command line tools. Also, if you did not install python with tk, you will need to uninstall and reinstall.

xcode-select --install

Pip install requirements

Here are the basic pip installs we needed on a Mac

pip install Cython
pip install pillow
pip install lxml
pip install jupyter
pip install matplotlib

We had trouble upgrading the jupyter install because an error with distutils 'traitlets'. So we did the jupyter upgrade using

conda install jupyter

This solved the problem, which we verified by then running the pip install jupyter again, and it was satisfied with us.

We have not yet figured out how to do this equivalent on the mac

sudo apt-get install protobuf-compiler python-pil python-lxml python-tk

Adding the cocoapi calls

For tensorflow v1.8, you need pycocotools to utilize coco metrics on your evaluation process. This should not be needed in the future, but at this time there are known issues caused by tensorflow(v1.8).

To solve this for now, we manually include pycocotools when submitting an evaluation job to cloud. This is down by download ing pycocotools-2.0.tar.gz.

Put the tar.gz in ./models/research/object_detection. It will be copied to the GCP and dealt with as part of the isetcloud process.

Something else

There are two main types of actions. First, we select and train the model with training data. Once the model is trained, we apply it to new data.

Some of these actions we ask you to do by hand. For example, if you are downloading a specific data set and converting it to tensorflow format, we give an you an example but not a universal function.

Other things we help you automate with the gCloud class. For example, creating and executing the command to start training, or the command to evaluate the test data, we do for you.

We also provide some simple utilities of things that we do all the time, like download a tar file locally. Or upload a tarfile to a specific directory in the cloud and then extract it up there.

It is possible to do these steps locally on your machine. But since you are reading about the matlab2cloud implementation, we suppose that you would like to learn about how to run this on the cloud. Just a thought. We do have flags that allow you to test these steps locally before you start charging your Google account real money.

Notes

Protoc issue

We had to have protoc version 3.3.0 installed on gray for the object_detection code to run. The issue and solution is described here.

On gray, we moved the installed protoc to protoc-2.5.0 and made protoc a link to protoc_3.3/bin/protoc.