Hierarchical CNN to identify Mollusca

Introduction

In recent years, Convolutional Neural Networks (CNNs) have emerged as a dominant paradigm in machine learning, particularly for tasks involving image recognition, object detection, and classification. Their architecture, inspired by the human visual system, enables them to effectively capture spatial hierarchies in data, making them highly effective for extracting features across varying levels of granularity. While CNNs have demonstrated remarkable success in numerous domains, most applications are designed to produce flat, single-level predictions, without explicitly leveraging the hierarchical relationships inherent in many classification tasks.

Hierarchical classification tasks, such as taxonomic classification of species, inherently involve nested relationships between classes. For example, biological taxonomies organize organisms in a hierarchy of taxonomic ranks (e.g., class, order, family, and species), while diseases are categorized by systems and subtypes. Exploiting these hierarchical structures within a classification framework has the potential to improve both the accuracy and interpretability of machine learning models. Hierarchical CNNs, which either embed hierarchical structures into a single model or use a collection of CNN models working together hierarchically, represent a promising approach to address this challenge.

The concept of hierarchical CNNs encompasses two primary strategies. The first approach integrates hierarchical relationships directly into the architecture or loss function of a single CNN, enabling it to model dependencies among different levels of classification. The second approach involves a collection of CNN models, each specialized to operate at a specific level of the hierarchy, progressively refining predictions from coarse categories (e.g., order, family) to finer ones (e.g., species). This latter method closely mirrors the hierarchical decision-making processes, where general categories are identified first, followed by finer distinctions.
Hierarchical CNNs have shown potential in domains like taxonomy, where large datasets often contain imbalances across levels of the hierarchy. This paper explores the application of hierarchical CNNs in Mollusca. We discuss the advantages of leveraging hierarchical structures and analyze existing methodologies.

Hierarchical Classification in Biological Taxonomy Using CNNs on Images

CNNs are a widely used to detect and classify organisms. It is common to use a traditional “flat” classification, whereby CNN models do not explicitly consider evolutionary relationships between taxa [1]. However, hierarchical classification has been applied in deep learning to manage large datasets with numerous classes effectively. For the Mollusca dataset presented here, which includes over 100,000 species [11], a hierarchical classification approach is particularly well-suited. Hierarchical classification was attempted for many organism such as arthropods [3]. Tresson et al. (2021)[3] used a two-step approach, first classifying arthropods in five super-classes and then using another model to classify cropped images at the species level. Ong SQ & Hamid SA. [4] have shown the each taxonomic level has its optimal algorithm. Moreover each taxon gave a different performance. They demonstrated that a single deep learning architecture was not robust enough to classify different taxonomic levels of specimens. Another approach was used by Wu et al. [6]. They used a hierarchical cross-entropy (HXE) loss function to organize a “flat” network of leaves in a tree. However, results varied widely among different datasets. Gupta et al. [4] used an hierarchical classification together with YOLOv3 object detection in fish. The same approach is used by Iwano et al [14]. They developed a hierarchical framework that combines YOLOv7 and EfficientNetV2, demonstrating its efficacy in plant disease diagnosis and hierarchical classification. While hierarchical classification has already been developed for species classification, it has to our knowledge not been carried out for Mollusca.

Approaches to use CNN for Hierarchical Classification

There are several approaches to create a hierarchical CNN for a taxonomic tree, having many child nodes per parent node [9]. The easiest method is to determine the most specific level, species (the lowest level, the leafs), ignoring the hierarchical structure. This approach behaves like a traditional classifier during training and testing and provides only indirect information about the hierarchy outside of the classifier. Figure 1 illustrates this approach. This "flat" classification might become unwiedly when there are many species.

Another approach is the "Local Classifiers per Parent Node", where separate CNN models for each node in the taxonomic tree is trained. Each classifier is responsible for discriminating between the child nodes of its parent node [13]. It is also known as the top-down approach (see figure 2). In this top-down approach, for each new example in the test set, the system first predicts its first-level (coarse-grained) class, then it uses that predicted class to narrow the choices of classes to be predicted at the second level (the only valid candidate second-level classes are the children of the class predicted at the first level), and so on, recursively, until the most specific (fine-grained) prediction is made. A disadvantage of the top-down class-prediction approach is that an error at one level will be propagated downwards the hierarchy, unless some procedure for avoiding this problem is used (see next chapter). For example, one CNN at the class level to differentiate gastropods from bivalves, then another CNN for gastropods to distinguish orders, and so on
A compromise between the previous two is the Local Classifiers per Level. In this approach one CNN per taxonomic level (one for class, one for order, one for family, etc.) is trained. This means that depending on the higher levels, there are less classes, so the model is trained in the coarse-grained features, while the classifier on the lowest level will be trained on many more classes on the fine-grained features. Figure 3 gives a schematic presentation of this approach. One possible way of classifying test examples using classifiers trained by this approach is when a new test example is presented to the classifier, to get the output of all classifiers (one classifier per level) and use this information as the final classification. This approach was used by S. Kiel on fossil bivalves [20] (a subset of the Mollusca). This research focused on training CNNs at different taxonomic levels (family, order, and subclass) and combining their predictions to address the issue of convergent evolution, where unrelated species develop similar features. The study found that incorporating predictions from higher taxonomic levels improved the accuracy of family-level clustering, particularly for families not included in the training data. This finding supports the concept of hierarchical CNNs being able to leverage broader morphological knowledge to refine fine-grained distinctions. The major drawback of this class-prediction approach is being prone to class-membership inconsistency.






At last, there is the approach to take the hierarchy into account in a global classifier. This gives a single , but complex, classification model built from the training set. For example, the CNN's loss function could incorporate the hierarchical structure. This approach penalize errors more heavily that are further away in the taxonomic tree [12]. Only one CNN is used to train the complete dataset, so this corresponds with the approach described in figure 1, but with the advantage of having the hierarchy included in the model.

Rationale for using a local classifier per node approach

In this article the "Local Classifier per Parent Node (LCPN) approach [9] was selected. In LCPN a multi-class classifier is trained for each parent node to distinguish between its child nodes. For example, a classifier at the family level would be trained to classify all species within that family. Using a model for each node provides much more flexibility during training of the model. The hyperparameters were optimized for each node.

Ong SQ & Hamid SA. [5] have demonstrated that using another model for each taxonomic level gives better results. The hyperparameters were different for each model, which depends on the size of the image dataset, the quality of the images and most important, the morphological features of each taxon (order, family, genus). Some genera have species that are easily to recognize, while other genera have species that are very difficult to identify. Therefore, having specializing classifiers for each node, LCPN can potentially improve overall classification accuracy . Each classifier focuses on a specific subset of the data and learns the distinct characteristics of that taxonomic group. This can be particularly beneficial when dealing with diverse and complex datasets, as is often the case in biological taxonomy.

LCPN has the potential to handle imbalanced datasets at different levels of the hierarchy. In biological taxonomy, certain taxa may have significantly fewer representatives than others. By training separate classifiers for each node, LCPN can address this imbalance by focusing on the specific characteristics of each group, even if some groups have limited data.

This allows working in an interative manner; adding more nodes over time. It enables also more experimentation because the time and IT resources to train a model for a new node, especially the leaves in the tree. At last, because the Mollusca taxonomy changes regularly, only a few nodes need new training.

At last, this approach is cost effective because it requires much less IT infrastructure when compared to training on the complete dataset of images.

A common disadvantage of this approach is the potential for inconsistencies in classification across different levels of the hierarchy . This can occur when a classifier at a lower level assigns an instance to a class that contradicts the classification made by a classifier at a higher level. For example, when classifying mammals, if a classifier at the order level assigns a specimen to the order Primates, but a classifier at the family level assigns it to the family Felidae (cats), this creates an inconsistency. But because an top-down approach is followed during inference time, no inconsistencies will occur. This involves starting at the root of the hierarchy and progressing down, using the predictions at each level to guide the selection of the next classifier. Thresholds can be applied to the classifier outputs to ensure that an instance is only assigned to a child node if the parent node's prediction exceeds a certain confidence level [9]

Another challenge is the increased complexity of managing and training a large number of individual classifiers, especially in a complex hierarchy like biological taxonomy . This requires a careful design of the inference process. [10] Using efficient algorithms and data structures can help reduce the computational burden of training and managing multiple classifiers. This could involve optimizing the code for specific IT architectures or using specialized libraries for hierarchical classification.

Methods

Data collection

Data collection was described previously. All taxonomic information was retrieved from MolluscaBase [11] or Worms [19] To have a reliable classification success a minimum of 25 images per species are needed.

Training and testing the convolutional neural network

Model selection was described here. For each node in the taxonomy, the hyperparameters were optimized. The following table shows starting values. All models were training using Tensorflow and Keras.

Model evaluation

The output layer (softmax) of the CNN consisted of a vector with a confidence value for each class (i.e., taxon) included in the model. The class with the highest value is the predicted class for an image. We assessed if the image was predicted correctly, if the ground truth name class is the same. The python module sklearn.metrics was used to calculate all common metrics: recall, precision, F1 score and accuracy.

Results

Criteria to create the tree structure

The taxonomic tree for Mollusca can have many levels, for example the tree for Callista chione (Linneaeus, 1758), a bivalve from the North Atlantic Ocen and Mediterranean Sea, is given below [11]. There are 12 taxonomic levels in this example between Mollusca and the species, Callista choine in this example. Creating a CNN model for each node would make the collection very complex. A simplified approach will be to have a maximum of only 4 levels, with the most common levels being Phylum > "Order" > "Family" > Genus > Species. This means in the maximal case, 4 separate CNN models are created to classify a species. The classification begins with the root node, the phylum Mollusca and determine the order. The first CNN model determines the order, although "order" is placed in quotes because this level can vary— it may correspond to a higher (e.g., subterclass) or lower taxonomic rank. Is some cases this intermediate level will be skipped, and the family will be directly determined starting of the phylum (refer to the dotted line in figure 6). The input of the next model will be the classification made by the Mollusca model. Depending on the predicted class another CNN model will be activated to determine the child class of the model. For example, consider the hierarchy for Callista chione. The correct class predicted by the Mollusca model will be Euheterodonta (see table 1). A Euheterodonta model will be activated that predicts the family. The correct classification would be Veneridae in the example. The next step will be the activation the Veneridae model to determine the genus. At last a Callista model will determine the species.
If not enough species are available in a particular genus (for example, only one or two species with enough images), it might be more efficient to create a model that classifies the species starting from the family. This is shown in figure 6 with the second dotted line. This hierarchical approach ensures an efficient and targeted classification process, with each CNN model specializing in specific levels of the taxonomy.

The root model: Mollusca

The classes selected for the first taxonomic level, the Mollusca (Phylum) parent node are listed in the next table. It has 22 classes, which classify molluscs at the "order" level (see figure 6). The taxonomic level is given in table I, and can be subterclass, infraclass, order, superfamily or family. When the class has the taxonomic level family, we consider this first level to be skipped (see figure 6). The last column shows the number of images available for the class. There is a large imbalance in the dataset.

Many orders were not included because there were not enough images available (for example, Laevidentaliidae, Neomphalida, Mesoneura, etc.). Also extinct orders were excluded. The amount of images available is a snapshot of January 2025. The criteria to select a particular taxonomic level are the number of classes available for the 2nd level (families) and the amount of images. Having between 5 and 20 classes for a model makes the model manageable and easier to analyse and fine-tune. While the Order taxonomic level is selected is most instances, sometimes a higher taxonomic level (Subterclass) or lower taxonomic level (Superfamily, Family) was selected. Even after carefully selection of the class, an imbalanced dataset was constructed. Including class weights [15], a focal loss function [16], augmentation [17] and/or undersampling [18] in model training alleviated partly the problems occuring with an imbalances among the classes in the dataset. The results show that some classes have a bad performance, most likely due to the large imbalance between classes in the dataset. Because of this large imbalance class weights, focal loss, augmentation of minority classes (rotate, zoom out, change brightness or contrast) and undersampling of the majority classes was tested. Table III shows the result of undersampling. It shows that increasing the maximum number of images per class improves the F1 score for the majority classes. The overall validation accuracy also increases until the maximum reaches 10 000 images per class. Using 25 000 images for the majority classes increases performance marginally. The F1 score for the minority class gives a mixed result, for one class there is an improvement, the other classes deteriorates. Analysis of other classes with low number of images show a similar tendency (not shown). Focal loss, regularization and top layer dropout were kept constant. The last row shows the results without undersampling. While the results are ambiguous, the results still show that undersampling improves the performance slightly.
All tests with augmentation of the minority classes (rotate, change brightness and contrast and zoom-out) gave no better results (not shown).
We have further fine-tuned the model and made more top layers trainable. This increases the overall performance, where 10 trainable layers is optimal. Fine-tuning the model has further improved the model, and again it shows that undersampling improves the performance. The best model has a maximum of 25000 images, splitted in 15000 images for the training dataset, and 5000 for the validation and test dataset. To have more images to train, we have also omitted the test dataset, using the validation dataset for testing. The results are similar (not shown).

Analyzing the models on genus taxonimic level of the Cypraeidae family

The species of the family Cypraeidae have a similar form, with a wide variation in the pattern and colours. For 18 genera we created models. We have compared these models to see if the hyperparameters correlate with the size of the dataset or the number of classes (species) in the dataset. Table V shows the result of all genera with the same (initial) hyperparameters and the final optimized, fine-tuned models.

Using the same parameters for the first training (see default parameters), the results vary widely between a training accuracy of 0.815 for the Mauritia model until 0.962 for the Pseudozonaria model. Optimizing the hyperparameters and further fine-tuning gave always better results, all models having at least a validation accuracy of 0.9. When plotting the amount of images in the dataset, or the number of species (classes) in the dataset against the accuracy, no correlation was found. The size of the dataset is not able to explain the variation in accuracy among the models.

Conclusion

This study has successfully demonstrated the power of hierarchical convolutional neural networks (CNNs) as a robust tool for the identification of Mollusca species. Our results provide compelling evidence for the efficacy of hierarchical classification strategies, which can be implemented through diverse architectures. These range from a single, end-to-end CNN capable of refining predictions from coarse taxonomic levels (e.g., class) down to fine-grained distinctions (e.g., species), to an ensemble of interconnected CNNs, where each model is specialized for a specific node within the taxonomic hierarchy. In this work, we elected to pursue the latter approach, constructing a hierarchical network of independent CNNs. As detailed in previous sections, this architectural decision was driven by the inherent advantages it offers in terms of modularity, optimization, and feature specificity. The strong performance of our model validates this choice and underscores the potential of hierarchical CNNs for accurate and efficient Mollusca classification.

The observed variations in performance across different taxonomic groups, particularly within the family Cypraeidae, offer critical insights into the benefits of our chosen architecture. By utilizing a dedicated model for each node within the hierarchy, we are empowered to fine-tune hyperparameters at each level of classification. This granular level of optimization ensures that each model is ideally suited to the specific challenges posed by its target taxonomic group. Consequently, this strategy leads to demonstrably superior outcomes compared to a single, "flat" model attempting to classify all Mollusca species simultaneously. In such a monolithic approach, optimal hyperparameters for one group might prove detrimental to another, hindering overall accuracy.

Furthermore, the adoption of a hierarchical network of specialized models allows each CNN to focus on learning the distinct morphological characteristics relevant to its assigned genus. Our findings suggest that the presence of these genus-specific features is a more significant determinant of model performance than either the sheer size of the dataset or the total number of classes within a given node. This implies that even within a complex and diverse group like Mollusca, a well-structured hierarchy can leverage subtle, genus-level morphological differences to achieve accurate identification.

The success of our hierarchical CNN approach has significant implications for the field of conchology and biodiversity research. It provides a framework for developing automated, accurate, and efficient tools for species identification, which can be invaluable for tasks such as biodiversity monitoring, ecological surveys, conservation efforts and shell collection. Investigating the interpretability of the learned features within each model could provide further insights into the specific morphological characteristics that distinguish different Mollusca genera and species.