Original picture set

[GeorgeGkinis]

Nice work @jakebruce !

I have been thinking about cell classification of frames for some days now and I came across your article.

I would like to see if using TensorFlow or another neural network framework, if classification could be further refined to make a distinction between worker and drone brood, pollen, capped honey, uncapped honey and egg/larvae (possibly stages of developement).

The Distinction between uncapped honey and egg/larva seems to be the most difficult after seeing the sample images.

A bigger dataset than the provided images in this repo would be needed to train such a neural net.

What do you think?
Do you still have the whole original set of images?

[jakebruce]

Hey @GeorgeGkinis, sounds good! I agree that this could be done with machine learning. The main challenge will be deployment-time variations in things like lighting, framing, and camera properties.

As you say, the solution here is a giant (labeled) dataset. I'd ballpark that you'd want a set of photos in the high thousands for that, with as much diversity as possible. Unfortunately I hand-tuned this algorithm on only a few dozen photos.

The domain experts are likely to have many more you might be able to obtain, however. You might try reaching out to my co-authors and to authors of other papers that cite our work here.

I'm optimistic about the performance of e.g. a SegNet style architecture if you can get a hold of the data to train it.

[jakebruce]

Thinking about it a bit more, you might be able to get away with far less data if you limit the scope to n-way classification of just the individual cells. Maybe this is what you were thinking of all along. You could do a rotation-invariant thing by using a polar representation of the individual cell patches, or make use of data augmentation in the form of rotation and mirroring to squeeze as much juice as you can out of limited data. Maybe do some lighting invariance techniques using the CIELAB channels etc to minimize lighting differences.

Given that there are hundreds of cells per image, maybe you could get that down to a manageable amount of photos and labelling.

[GeorgeGkinis]

I was thinking of individual cell classification, yes.
I can program quite well but I am new to machine learning.
This would be my first endeavor with training neural networks.

The approach I was thinking of would be :

• The frame area for each image would be clipped, ignoring areas outside the frame.
• Keep the yellow/brownish area within the frame
• Following would be the clipping of the individual cells as single images.
• Coordinates (x,y,side of frame) could be retained.
• Image manipulation like altering resolution/color before feeding the algorithm
• Your idea of rotating, mirroring (or maybe slightly deforming) each cell image to enrich the dataset is awesome!
• Build a tool to show each cell and manually classify choosing between a set of labels to make labeling faster/easier. Would not take longer than a couple of days for 1-2 people
• Train a model

I will need to look into the following terms which I do not clearly understand ( or not at all! :)

• SegNet style architecture
• rotation-invariant thing
• polar representation of the individual cell patches
• lighting invariance techniques
• CIELAB channels

[jakebruce]

Sounds good to me!

SegNet [1] is a fully-convolutional neural network for image segmentation. i.e. every input pixel gets an output label, but this isn't necessary if you're doing patchwise classification.

Data augmentation can get you pretty good rotation robustness. I was imagining some kind of rotationally invariant representation of the individual cell patches, like doing a convolution in polar space (angle, distance from center) rather than in (x, y) space. Just spitballing there, not sure if there's established state of the art in this area.

CIELAB is the color space that I'm using in some of this code, it's an alternative to RGB where the first channel ("L") is meant to represent human-perceived brightness, independent of the hue of the color, whereas the "a" and "b" channels represent hue. So it's sometimes used in computer vision to achieve lighting invariance [2].

[1] https://arxiv.org/abs/1511.00561
[2] http://www.cs.sfu.ca/~mark/ftp/Eccv04/

Good luck! I wish I had data for you, but I think contacting the bee experts will be your best bet there.

[GeorgeGkinis]

Thanks for your input!

Gotta read some more :)

Cheers!

[Theotime-Colin]

Hi, very good to hear from you. A machine learning approach would be great. I can provide hundreds of images, would you like me to upload them? You may want to have a look at this project: https://avsthiago.github.io/DeepBee/ But this is based on high quality pictures, very hard to take. Theo

…

On Wed, Jun 3, 2020 at 9:17 AM Jake Bruce ***@***.***> wrote: Sounds good to me! SegNet [1] is a fully-convolutional neural network for image segmentation. i.e. every input pixel gets an output label, but this isn't necessary if you're doing patchwise classification. Data augmentation can get you pretty good rotation robustness. I was imagining some kind of rotationally invariant representation of the individual cell patches, like doing a convolution in polar space (angle, distance from center) rather than in (x, y) space. Just spitballing there, not sure if there's established state of the art in this area. CIELAB is the color space that I'm using in some of this code, it's an alternative to RGB where the first channel ("L") is meant to represent human-perceived brightness, independent of the hue of the color, whereas the "a" and "b" channels represent hue. So it's sometimes used in computer vision to achieve lighting invariance [2]. [1] https://arxiv.org/abs/1511.00561 [2] http://www.cs.sfu.ca/~mark/ftp/Eccv04/ Good luck! I wish I had data for you, but I think contacting the bee experts will be your best bet there. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEKABMBTR77GBWCBCZGVJDRUWCB7ANCNFSM4NRDNDZQ> .

[GeorgeGkinis]

Wow! I feel like a time traveler seeing it already implemented 👍

I see you used keras. Nice :)

I can learn much going through your code.

A machine learning approach would be great.

So you did not use machine learning in your work?
There is a classification and segmentation model under software/model

An upload would be greatly appreciated!

[Theotime-Colin]

No I am not the author of DeepBee, I am a collaborator of Jake. I think DeepBee is a machine learning approach but it seems to still have a very high error rate for some categories (not surprising even for a human it's difficult). I haven't had time to try it yet personally.

…

On Wed, Jun 3, 2020 at 8:45 PM Georgios Gkinis ***@***.***> wrote: Wow! I feel like a time traveler seeing it already implemented 👍 I see you used keras. Nice :) I can learn much going through your code. A machine learning approach would be great. So you did not use machine learning in your work? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEKABLPZBELYVM5BI4VORLRUYSUDANCNFSM4NRDNDZQ> .

[GeorgeGkinis]

They did use ML with keras which uses tensor flow.
Also they have a ton of training images there :
https://github.com/AvsThiago/DeepBee-source/blob/release-0.1/src/data/resources/images_urls.csv