Tuo Zhao

Toward Deeper Understanding of Nonconvex Stochastic Optimization with Momentum using Diffusion Approximations.

2018 · arXiv (Cornell University)

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning. Popular examples include training deep neural networks, dimensionality reduction, and etc. Due to the lack of convexity …

Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective

2020 · arXiv (Cornell University)

Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets). However, the theory behind such a phenomenon is still largely unknown. This paper studies this fundamental problem in deep learning from …

BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision

2020

We study the open-domain named entity recognition (NER) problem under distant supervision. The distant supervision, though does not require large amounts of manual annotations, yields highly incomplete and noisy distant labels via external knowledge bases. …

ARCH: Efficient Adversarial Regularized Training with Caching

2021

Adversarial regularization can improve model generalization in many natural language processing tasks. However, conventional approaches are computationally expensive since they need to generate a perturbation for each sample in each epoch. We propose a new …

Token-wise Curriculum Learning for Neural Machine Translation

2021 · arXiv (Cornell University)

Existing curriculum learning approaches to Neural Machine Translation (NMT) require sampling sufficient amounts of "easy" samples from training data at the early training stage. This is not always achievable for low-resource languages where the amount …

CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing

2022 · arXiv (Cornell University)

Model ensemble is a popular approach to produce a low-variance and well-generalized model. However, it induces large memory and inference costs, which are often not affordable for real-world deployment. Existing work has resorted to sharing …

Robust Reinforcement Learning from Corrupted Human Feedback

2024 · arXiv (Cornell University)

Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect …

BlendFilter: Advancing Retrieval-Augmented Large Language Models via Query Generation Blending and Knowledge Filtering

2024

Haoyu Wang, Ruirui Li, Haoming Jiang, Jinjin Tian, Zhengyang Wang, Chen Luo, Xianfeng Tang, Monica Xiao Cheng, Tuo Zhao, Jing Gao. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.