Haoyi Zhou

Triplet Attention

2021

The Transformer model has benefited various real-world applications, where the self-attention mechanism with dot-products shows superior alignment ability on building long dependency. However, the pair-wisely attended self-attention limits further performance improvement on challenging tasks. To …

RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models

2021 · arXiv (Cornell University)

Pre-trained language models achieve outstanding performance in NLP tasks. Various knowledge distillation methods have been proposed to reduce the heavy computation and storage requirements of pre-trained language models. However, from our observations, student models acquired …

Research on transformer internal defect detection based on large scale model

2023 · Journal of Physics Conference Series

Abstract Aiming at the problem that the traditional convolution network can not be used to detect and identify targets in the more complex target detection application scenarios in power grid business, a target detection scheme …