A Biological Challenge in the Billions
Deep Learning for Nematode Detection and Classification
By: Jiejun Lu, Hongxiang Qiu, Weidong Xu and Zeyu Zhao
Introduction
The organic food industry has been growing rapidly over the past few years. An important task in the organic food industry is to find alternatives to chemical pesticides for protecting crops and plants from pests due to the detrimental consequences of pesticides. An alternative to chemical pesticides is
to
encourage naturally occurring organisms in the soil to destroy pests such as slugs, weevils
and
caterpillars. Nematodes are naturally occurring microscopic worms already present in the
soil
that
actively seek out and destroy pests. Biologicals for pest control is one of the focus areas
within
BASF's Agricultural Solutions business. They develop unique formulations of beneficial
nematodes
and
their storage media to provide optimum stability and product performance. The nematodes in
the
products selectively target problematic insect species, while remaining harmless to
beneficial
insects (e.g. ladybugs) and nearby wildlife. Finding an efficient way to use nematodes to
protect
crops could potentially reduce the need to introduce artificial chemicals in to the food
supply. BASF is the only mass producer of nematodes used for slug protection, utilizing a plant in
Littlehampton, UK equipped with 20 vessels that provide over 190,000 L of fermentation
capacity
and
can hold ~ 38 × 1012 nematodes. Product quality is continually tested and
maintained
through
refrigeration and efficient shipping logistics to all their customers, which is a key BASF
skill.
As shown in the figure above, nematodes have different efficacy at different stages of their life cycles. Automation of identification of whether a nematode is infective juvenile or not enables efficient quality control. This project applied deep learning techniques to microscope image data of nematode populations to determine whether nematodes are at infective juvenile stage (interested) or not (non-interested). We implemented and trained a scale-aware (SA) version of faster region-based convolutional neural network (faster R-CNN) model for the identification and classification of nematodes from microscope images, and built a software package in a virtual machine image for the automation of nematode classification task. The expected input and output is shown in the figure below.
Data
First batch of data
BASF didn't have a digital way to label the nematodes in the microscope images. Initially, we only had some sample microscope images and a slide briefing the characteristics of each life stage of nematodes. A sample image is shown below.
Labeling tool
We implemented a labeling tool and delivered it to BASF. BASF used the labeling tool we provided to generate digitalized labels of nematodes in microscope images. A screen shot of our labeling tool is shown below.
After that, we have received several batches of data from BASF. Currently (May 2018), we have 406 valid labeled microscope images, with unbalanced distribution as shown below.
Challenges
We found many challenges during our data analysis and processing, including bad illumination, noise, overlapping nematodes, nematodes with weired shape, and mislabeled data (in the example image below, all nematodes were labeled as "interested"; however, it is impossible in this case), as shown below:
Data augmentation
Typically, we need much more samples to train a deep neural network such as faster R-CNN. Considering we don't have enough data, we have applied aggressive data augmentation. For each original image, we generate its flipped versions and illumination adjusted versions, which grows our training set by 11 times.
Literature Review
Conventional computer vision methods rely on the extraction of feature vectors from images or image patches by using the feature descriptor. Histogram of oriented gradients (HOG) is a popular feature descriptor, which uses the distribution of directions of gradients as features [1]. Gradients of an image are large around edges and corners, which carry a lot of information about object shape, and are thus useful for classification. The HOG descriptors can be fed into a conventional classification algorithm, such as support vector machine (SVM). An example of HOG feature descriptor is shown below.
Deep learning methods usually outperform conventional methods when there are enough data. Faster R-CNN is the state-of-the-art deep learning algorithm for object detection [2]. The basic idea is first apply convolutional layers to get the feature maps. The region proposal network (RPN) then takes different sizes and positions of anchor boxes from the feature map, and learns to classify bounding boxes into background and foreground as well as to refine the bounding boxes. The foreground boxes from RPN goes through region of interest pooling. It will resize the feature map corresponding to the proposed bounding boxes into same shape, and the classifier will predict the class of each bounding box. The architecture scheme of faster R-CNN is shown below.
Modeling
Conventional Models
From a machine learning point of view, the problem could be divided into two parts: detection and classification. Detection is to find individual nematodes in the microscope image. Classification is to predict whether each nematode is in interested stage.
The steps of our baseline model is as follows:
- Extract images of individual nematodes from a microscope image. Our segmentation uses conventional computer vision techniques. We set area threshold and average color threshold to distinguish large noise contours from nematodes.
- Apply HOG feature extraction algorithm to obtain feature vectors for extracted nematodes.
- Apply classification on the feature vectors. The classifier could be anything like support vector machine or random forest.
The scheme of baseline models pipeline is shown below.
Deep Learning (R-CNN)
Our baseline models can not handle situations such as overlapped nematodes. They also seem unable to capture details which could be very important in classification. Deep learning models usually outperform conventional methods when there are enough data. We applied an end-to-end faster R-CNN model for nematode detection and classification.
We applied a pre-trained RES-152 [3] network to get the feature maps. The region proposal network (RPN) takes different sizes and positions of anchor boxes from the feature map, and learns to classify bounding boxes into background and foreground; it also learns to refine the bounding boxes. The output of RPN goes through region of interest (RoI) pooling. It resizes the feature map corresponding to the proposed bounding boxes into same shape, and the classifier will predict the class of each bounding box. The architecture scheme of the faster R-CNN model is shown below.
Adding Scale Information
The sizes of bounding boxes are not explicitly considered when faster R-CNN does classification, since RoI layer resizes feature maps into same dimensions. This makes sense in most applications because objects appear smaller when they are further away. For example, the cars have different sizes in the example image below. When we do object detection in such cases, size of bounding boxes (or feature maps) shouldn’t be considered.
However, in microscope images, the distance to camera is fixed. And when we classify nematodes, size matters. Nematodes that are either too small or too big are unlikely to be interested. To take the size information into account, we implemented a modified version of faster R-CNN, which we call scale-aware R-CNN (SA-R-CNN). We get the size information from RPN, which is then passed into the classifier together with the results from RoI pooling. The architecture scheme of SA-R-CNN is shown below.
Results
We recieved 4 batches of data, which contained 406 labeled images in total. We trained our models on the first 3 batches and evaluated on the last batch (42 images).
The bounding box detection error for SA-R-CNN is summarized below:
| Detection | Actual label | No Actual Label |
|---|---|---|
| Predicted Label | 85 | 78 (only 1 interested) |
| No Predicted Label | 0 | N/A |
The classification results are summarized below (AUC/accuracy is evaluated by matching each box predicted to each box labeled and ignoring those don’t match):
| Model | AUC | Accuracy |
|---|---|---|
| SVM | 0.66 | 0.75 |
| Random Forest | 0.69 | 0.75 |
| Faster R-CNN (res152) | 0.82 | 0.74 |
| SA-R-CNN (res152) | 0.88 | 0.75 |
From the results, we see our SA-R-CNN is better than the default faster R-CNN and both of them are better than conventional models (since conventional models are worse on detection, the statistics shown for conventional models are higher than the actual values). For detection, our SA-R-CNN predicts many nematode-like objects such as dead nematodes. However, since almost all those predictions are non-interested, we think it’s not an issue (dead nematodes are not interested anyway).
| Predicted as Non-interested | Predicted as Interested | |
|---|---|---|
| Labeled as Non-interested | 17 | 1 |
| Labeled as Interested | 20 | 47 |
From the confusion matrix we can see we have few false positives but many false negatives. As discussed, we think our models are not trained enough due to the lack of training data. With more data, we believe our models can do better.
Deliverable
We built a software that automatically outputs the predictions and generates a CSV for counts of interested/non-interested nematodes in each image. An example output image by our software is shown below.
We delivered a virtual machine with all models and GUI to BASF (screenshots are shown below).
The flowchart for using the software is:
The source code of our models and software can be found here: https://github.com/BASFProject-2018spring
The VM can be downloaded from our Google Drive: https://drive.google.com/file/d/1-VdYoSuK8tQ_kuPHHLNH-PFjvQa4caqY/view?usp=sharing
The video tutorial of our software is here:
In case BASF wants to retrain the model in the future with more available data, we wrote a documented training pipeline.
All code can be found in our project organization (public)
We also made a video tutorial for the training:
In the latest meeting, BASF told us they would like to add more classes of nematodes in the future (such as dead nematode class). So we have included information on places to change for our model working for more classes. This documentation is also in our training pipeline.
Once BASF have new data labled (with more classes), they could make changes to our code accordingly and train a new model.
Empirical Interpretation
BASF is also interested in how the model makes the prediction (i.e. what features the network learns). However, deep learning models are hard to interpret.
One way to do this is to fit a simpler model to simulate the deep learning model, using the desired features (like size, angle and etc.). The fitted simpler model can partially explain the deep neural network. The problem is that some desired features are hard to define mathematically (for example, the angle or the shape) and some features are hard to extract (for example, accurately obtaining size of a nematode is hard). We leave this method as a future work.
Another simpler empirical way is to get an interested nematode, change its features and see whether our model changes the prediction. We tried this method and the result is shown in the following images.
We made the first image based on an interested nematode (top-left corner). We used PhotoShop to rotate the nematode, change its color, resize it and change its shape. All generated fake nematodes were then combined into a single image whose background is from a training image. The second image is generated by passing the first image into our model. We observed the following:
- The model predicts all rotated nematodes as interested. So rotation doesn't affect the model.
- With mild shape change, the nematodes are still predicted as interested. However, if we straighten the nematode, it will become non-interested. So the model does look at the shape of the nematode, but with mild shape change, its prediction won't change.
- Relative darkness (brightness) of nematodes matters a lot. We changed the brightness of the nematode and it becomes non-intersted. For more aggressive change (let the nematode transparent), our model will think the nematode is part of the background.
- Size matters a lot. Letting the nematode larger or smaller makes it non-interested. If the nematode is too small, our model will think it's part of the background.
- Noise does not matter. There're different debris around the rotated nematodes and the original nematodes. But the model does not change the prediction nor the prediction confidence level.
We conducted such experiment on some nematodes and the conclusions are similar. So we hope our conclusions above is generalizable for all nematode images because that means our model works as expected. In order to get insight about how exactly the model uses those features to make decisions, we need to use more sophisticated methods.
Conclusions & Future Work
We have implemented both conventional models and deep learning models for the nematode detection and classification tasks. We have created a new network architecture to make faster R-CNN model scale aware for nematode classification. Our deep learning models outperform baseline models, although our models suffer from overfitting due to the unavailability of enough training data. Also, we have delivered our models and GUI (with tools like CSV generator) within a virtual machine to BASF.
Our models can be improved in the following aspects:
- Deep learning models typically require hugh amount of training data. We believe our model can perform better with more data.
- Currently our models assume the input images are at a fixed scale. For different scale images, users have to use our script to transfer the images into the required scale. We should be able to handle images at different scales in the future.
- More efforts are required to improve the interpretability of our models.