Quentin Rebjock

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

2019 · arXiv (Cornell University)

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not …