Summary Brain-Inspired Efficient Pruning in Spiking Neural Networks arxiv.org
8,040 words - PDF document - View PDF document
One Line
A brain-inspired pruning method enhances spiking neural networks by extracting crucial information, resulting in improved performance, feature uniformity, and structure selection.
Slides
Slide Presentation (7 slides)
Key Points
- Researchers have developed a brain-inspired pruning method for spiking neural networks (SNNs) that efficiently extracts critical information while reducing computational and storage overhead.
- SNNs are attractive for deployment on devices with limited resources due to their event-driven computing characteristic.
- The proposed method uses a regeneration mechanism based on criticality to obtain critical pruned networks.
- The method achieves higher performance compared to the state-of-the-art methods with the same time overhead and achieves comparable or better performance with significant acceleration.
- The authors provide insights into the underlying mechanisms of their method, highlighting its effectiveness in selecting critical structures, improving feature uniformity, and reducing overfitting.
Summaries
19 word summary
A brain-inspired pruning method efficiently extracts critical information in spiking neural networks, improving performance, feature uniformity, and structure selection.
52 word summary
A brain-inspired pruning method for spiking neural networks has been developed to extract critical information efficiently. The method uses a regeneration mechanism based on criticality to obtain critical pruned networks. It achieves higher performance than state-of-the-art methods with the same time overhead, improves feature uniformity, reduces overfitting, and efficiently selects potential structures.
107 word summary
Researchers have developed a brain-inspired pruning method for spiking neural networks (SNNs) that efficiently extracts critical information while reducing computational and storage overhead. The method proposes a regeneration mechanism based on criticality to obtain critical pruned networks. It defines a low-cost metric for the criticality of pruning structures and re-ranks and regenerates the pruned structures with higher criticality. The method is evaluated on VGG-16 and ResNet-19 models for both unstructured and structured pruning, achieving higher performance compared to state-of-the-art methods with the same time overhead. It also achieves comparable or better performance with significant acceleration. The method improves feature uniformity, reduces overfitting, and efficiently selects potential structures.
498 word summary
Researchers have developed a brain-inspired pruning method for spiking neural networks (SNNs) that efficiently extracts critical information while reducing computational and storage overhead. SNNs are attractive for deployment on devices with limited resources due to their event-driven computing characteristic. However, pruning deep SNNs is challenging due to the binary and non-differentiable nature of spike signals. Existing methods require high time overhead to make pruning decisions.
In this study, the researchers propose a regeneration mechanism based on criticality to obtain critical pruned networks. They first define a low-cost metric for the criticality of pruning structures and then re-rank and regenerate the pruned structures with higher criticality. The method is evaluated using VGG-16 and ResNet-19 for both unstructured and structured pruning. The results show that the proposed method achieves higher performance compared to the state-of-the-art methods with the same time overhead. It also achieves comparable performance, and even better performance on VGG-16, with significant acceleration.
SNNs have gained attention as the third generation of neural networks due to their ability to emulate the behavior of biological neurons. They are particularly suitable for devices with limited computing resources and lower power consumption. However, the limited computing and storage capacity of these devices poses a challenge for implementing deep SNNs with large-scale parameters.
Network pruning has been widely explored as a solution to reduce the computing and storage overhead of SNNs. Some previous works have been inspired by the human brain and modeled synaptic plasticity and spine motility to optimize network structure and connections. Other works have used predefined thresholds to remove weak weights during the learning process. The lottery ticket hypothesis has also been explored in SNN pruning. However, the binary representation and non-differentiable property of spike signals make training deep SNNs challenging.
To address these challenges, the researchers propose a regeneration mechanism based on criticality. They define a metric for neuron criticality in SNNs, which is related to the distance between the membrane potential and the threshold voltage of a neuron. The criticality-based regeneration mechanism selects neurons with higher criticality for reactivation and synapse regeneration after each pruning iteration.
The proposed method is evaluated on VGG-16 and ResNet-19 models for both unstructured and structured pruning. The results show that the method achieves higher performance compared to the state-of-the-art methods with the same time overhead. It also achieves comparable performance, and even better performance on VGG-16, with significant acceleration. The authors investigate the impact and underlying mechanisms of their method and find that it efficiently selects potential structures, improves the uniformity of features, and reduces overfitting during the recovery phase.
In conclusion, the researchers have developed a brain-inspired pruning method for SNNs that efficiently extracts critical information while reducing computational and storage overhead. The method achieves higher performance compared to existing methods with the same time overhead and achieves comparable or better performance with significant acceleration. The authors also provide insights into the underlying mechanisms of their method, highlighting its effectiveness in selecting critical structures, improving feature uniformity, and reducing overfitting.
637 word summary
Researchers have developed a brain-inspired pruning method for spiking neural networks (SNNs) that efficiently extracts critical information while reducing computational and storage overhead. SNNs are attractive for deployment on devices with limited resources due to their event-driven computing characteristic. However, pruning deep SNNs is challenging due to the binary and non-differentiable nature of spike signals. Existing methods require high time overhead to make pruning decisions.
In this study, the researchers propose a regeneration mechanism based on criticality to obtain critical pruned networks. They first define a low-cost metric for the criticality of pruning structures and then re-rank and regenerate the pruned structures with higher criticality. The method is evaluated using VGG-16 and ResNet-19 for both unstructured and structured pruning. The results show that the proposed method achieves higher performance compared to the state-of-the-art methods with the same time overhead. It also achieves comparable performance, and even better performance on VGG-16, with significant acceleration.
SNNs have gained attention as the third generation of neural networks due to their ability to emulate the behavior of biological neurons. They are particularly suitable for devices with limited computing resources and lower power consumption. However, the limited computing and storage capacity of these devices poses a challenge for implementing deep SNNs with large-scale parameters.
Network pruning has been widely explored as a solution to reduce the computing and storage overhead of SNNs. Some previous works have been inspired by the human brain and modeled synaptic plasticity and spine motility to optimize network structure and connections. Other works have used predefined thresholds to remove weak weights during the learning process. The lottery ticket hypothesis has also been explored in SNN pruning. However, the binary representation and non-differentiable property of spike signals make training deep SNNs challenging. Spike signals are easily confused by disturbance and suffer from spike vanish or explosion, resulting in insufficient expression of feature information. The non-differentiable property of spikes necessitates the use of surrogate functions to approximate gradients, leading to gradient vanishing. These challenges become more prominent in pruned SNNs because it is difficult to retain the most critical parameters from the not fully trained network. Current state-of-the-art methods typically require extended training or iteration times to attain pruned networks, resulting in significant pruning costs.
To address these challenges, the researchers propose a regeneration mechanism based on criticality. Inspired by the critical brain hypothesis, which suggests that the brain operates at a critical state highly sensitive to inputs and facilitating information transmission, they define a metric for neuron criticality in SNNs. The criticality score is related to the distance between the membrane potential and the threshold voltage of a neuron. They use the derivative of a surrogate function as a suitable criticality metric, as it reflects the criticality changes of neuron decisions and can be obtained with minimal additional computational cost. The criticality-based regeneration mechanism selects neurons with higher criticality for reactivation and synapse regeneration after each pruning iteration.
The proposed method is evaluated on VGG-16 and ResNet-19 models for both unstructured and structured pruning. The results show that the method achieves higher performance compared to the state-of-the-art methods with the same time overhead. It also achieves comparable performance, and even better performance on VGG-16, with significant acceleration. The authors investigate the impact and underlying mechanisms of their method and find that it efficiently selects potential structures, improves the uniformity of features, and reduces overfitting during the recovery phase.
In conclusion, the researchers have developed a brain-inspired pruning method for SNNs that efficiently extracts critical information while reducing computational and storage overhead. The method achieves higher performance compared to existing methods with the same time overhead and achieves comparable or better performance with significant acceleration. The authors also provide insights into the underlying mechanisms of their method, highlighting its effectiveness in selecting critical structures, improving feature uniformity, and reducing overfitting.