A Sharp Convergence Rate for the Asynchronous Stochastic Gradient Descent

24 Jan 2020 Zhu Yuhua Ying Lexing

We give a sharp convergence rate for the asynchronous stochastic gradient descent (ASGD) algorithms when the loss function is a perturbed quadratic function based on the stochastic modified equations introduced in [An et al. Stochastic modified equations for the asynchronous stochastic gradient descent, arXiv:1805.08244]. We prove that when the number of local workers is larger than the expected staleness, then ASGD is more efficient than stochastic gradient descent... (read more)

PDF Abstract
No code implementations yet. Submit your code now

Categories


  • NUMERICAL ANALYSIS
  • NUMERICAL ANALYSIS
  • OPTIMIZATION AND CONTROL