Algorithms for Smooth Nonconvex Optimization with Worst-case Guarantees PDF Download
Are you looking for read ebook online? Search for your book and save it on your Kindle device, PC, phones or tablets. Download Algorithms for Smooth Nonconvex Optimization with Worst-case Guarantees PDF full book. Access full book title Algorithms for Smooth Nonconvex Optimization with Worst-case Guarantees by Michael John O'Neill. Download full books in PDF and EPUB format.
Author: Michael John O'Neill
Publisher:
ISBN:
Category :
Languages : en
Pages : 0
Get Book Here
Book Description
The nature of global convergence guarantees for nonconvex optimization algorithms has changed significantly in recent years. New results characterize the maximum computational cost required for algorithms to satisfy approximate optimality conditions, instead of focusing on the limiting behavior of the iterates. In many contexts, such as those arising from machine learning, convergence to approximate second order points is desired. Algorithms designed for these problems must avoid saddle points efficiently to achieve optimal worst-case guarantees. In this dissertation, we develop and analyze a number of nonconvex optimization algorithms. First, we focus on accelerated gradient algorithms and provide results related to the avoidance of "strict saddle points''. In addition, the rate of divergence these accelerated gradient algorithms exhibit when in a neighborhood of strict saddle points is proven. Subsequently, we propose three new algorithms for smooth, nonconvex optimization with worst-case complexity guarantees. The first algorithm is developed for unconstrained optimization and is based on the classical Newton Conjugate Gradient method. This approach is then extended to bound constrained optimization by modifying the primal-log barrier method. Finally, we present a method for a special class of ``strict saddle functions'' which does not require knowledge of the parameters defining the optimization landscape. These algorithms converge to approximate second-order points in the best known computational complexity for their respective problem classes.
Author: Michael John O'Neill
Publisher:
ISBN:
Category :
Languages : en
Pages : 0
Get Book Here
Book Description
The nature of global convergence guarantees for nonconvex optimization algorithms has changed significantly in recent years. New results characterize the maximum computational cost required for algorithms to satisfy approximate optimality conditions, instead of focusing on the limiting behavior of the iterates. In many contexts, such as those arising from machine learning, convergence to approximate second order points is desired. Algorithms designed for these problems must avoid saddle points efficiently to achieve optimal worst-case guarantees. In this dissertation, we develop and analyze a number of nonconvex optimization algorithms. First, we focus on accelerated gradient algorithms and provide results related to the avoidance of "strict saddle points''. In addition, the rate of divergence these accelerated gradient algorithms exhibit when in a neighborhood of strict saddle points is proven. Subsequently, we propose three new algorithms for smooth, nonconvex optimization with worst-case complexity guarantees. The first algorithm is developed for unconstrained optimization and is based on the classical Newton Conjugate Gradient method. This approach is then extended to bound constrained optimization by modifying the primal-log barrier method. Finally, we present a method for a special class of ``strict saddle functions'' which does not require knowledge of the parameters defining the optimization landscape. These algorithms converge to approximate second-order points in the best known computational complexity for their respective problem classes.
Author: Courtney Paquette
Publisher:
ISBN:
Category :
Languages : en
Pages : 167
Get Book Here
Book Description
Complexity theory drives much of modern optimization, allowing a fair comparison between competing numerical methods. The subject broadly seeks to both develop efficient algorithms and establish limitations on efficiencies of any algorithm for the problem class. Classical complexity theory based on oracle models targets problems that are both smooth and convex. Without smoothness, methods rely on exploiting the structure of the target function to improve on the worst-case complexity of non-smooth convex optimization. This thesis explores complexity of first-order methods for structured non-smooth and non-convex problems. A central example is the minimization of a composition of a convex function with a smooth map - the so-called convex-composite problem class. Nonlinear least squares formulations in engineering and nonlinear model fitting in statistics fall within this framework. The thesis develops new algorithms for the composite problem class, along with inertial variants that are adaptive to convexity. Acceleration is a widely used term in contemporary optimization. The term is often used to describe methods with efficiency guarantees matching the best possible complexity estimates for a given problem class. This thesis develops methods that interpolate between convex and non-convex settings. In particular, we focus on minimizing large finite sum problems, popular for modeling empirical risk in statistical applications, when the user is unaware of the convexity of the objective function. The scheme we describe has convergence guarantees that adapt to the underlying convexity of the objective function. First-order algorithms for non-smooth problems depend on having access to generalized derivatives of the objective function. We conclude the thesis with a fresh look at variational properties of spectral function. These are the functions on the space of symmetric matrices that depend on the matrix only through its eigenvalues. In particular, our analysis dramatically simplifies currently available derivations of differential formulas of such functions.
Author: Coralia Cartis
Publisher: SIAM
ISBN: 1611976995
Category : Mathematics
Languages : en
Pages : 549
Get Book Here
Book Description
A popular way to assess the “effort” needed to solve a problem is to count how many evaluations of the problem functions (and their derivatives) are required. In many cases, this is often the dominating computational cost. Given an optimization problem satisfying reasonable assumptions—and given access to problem-function values and derivatives of various degrees—how many evaluations might be required to approximately solve the problem? Evaluation Complexity of Algorithms for Nonconvex Optimization: Theory, Computation, and Perspectives addresses this question for nonconvex optimization problems, those that may have local minimizers and appear most often in practice. This is the first book on complexity to cover topics such as composite and constrained optimization, derivative-free optimization, subproblem solution, and optimal (lower and sharpness) bounds for nonconvex problems. It is also the first to address the disadvantages of traditional optimality measures and propose useful surrogates leading to algorithms that compute approximate high-order critical points, and to compare traditional and new methods, highlighting the advantages of the latter from a complexity point of view. This is the go-to book for those interested in solving nonconvex optimization problems. It is suitable for advanced undergraduate and graduate students in courses on advanced numerical analysis, data science, numerical optimization, and approximation theory.
Author: Phillip-Raphael Keldenich
Publisher:
ISBN:
Category :
Languages : en
Pages :
Get Book Here
Book Description
Author: A. R. Conn
Publisher: SIAM
ISBN: 0898714605
Category : Mathematics
Languages : en
Pages : 960
Get Book Here
Book Description
Mathematics of Computing -- General.
Author: Yurii Nesterov
Publisher: Springer
ISBN: 3319915789
Category : Mathematics
Languages : en
Pages : 603
Get Book Here
Book Description
This book provides a comprehensive, modern introduction to convex optimization, a field that is becoming increasingly important in applied mathematics, economics and finance, engineering, and computer science, notably in data science and machine learning. Written by a leading expert in the field, this book includes recent advances in the algorithmic theory of convex optimization, naturally complementing the existing literature. It contains a unified and rigorous presentation of the acceleration techniques for minimization schemes of first- and second-order. It provides readers with a full treatment of the smoothing technique, which has tremendously extended the abilities of gradient-type methods. Several powerful approaches in structural optimization, including optimization in relative scale and polynomial-time interior-point methods, are also discussed in detail. Researchers in theoretical optimization as well as professionals working on optimization problems will find this book very useful. It presents many successful examples of how to develop very fast specialized minimization algorithms. Based on the author’s lectures, it can naturally serve as the basis for introductory and advanced courses in convex optimization for students in engineering, economics, computer science and mathematics.
Author:
Publisher:
ISBN:
Category :
Languages : en
Pages :
Get Book Here
Book Description
Author: John Wright
Publisher: Cambridge University Press
ISBN: 1108489737
Category : Computers
Languages : en
Pages : 717
Get Book Here
Book Description
Connects fundamental mathematical theory with real-world problems, through efficient and scalable optimization algorithms.
Author: Mukhdeep Singh Manshahia
Publisher: John Wiley & Sons
ISBN: 1119791820
Category : Technology & Engineering
Languages : en
Pages : 948
Get Book Here
Book Description
HANDBOOK OF INTELLIGENT COMPUTING AND OPTIMIZATION FOR SUSTAINABLE DEVELOPMENT This book provides a comprehensive overview of the latest breakthroughs and recent progress in sustainable intelligent computing technologies, applications, and optimization techniques across various industries. Optimization has received enormous attention along with the rapidly increasing use of communication technology and the development of user-friendly software and artificial intelligence. In almost all human activities, there is a desire to deliver the highest possible results with the least amount of effort. Moreover, optimization is a very well-known area with a vast number of applications, from route finding problems to medical treatment, construction, finance, accounting, engineering, and maintenance schedules in plants. As far as optimization of real-world problems is concerned, understanding the nature of the problem and grouping it in a proper class may help the designer employ proper techniques which can solve the problem efficiently. Many intelligent optimization techniques can find optimal solutions without the use of objective function and are less prone to local conditions. The 41 chapters comprising the Handbook of Intelligent Computing and Optimization for Sustainable Development by subject specialists, represent diverse disciplines such as mathematics and computer science, electrical and electronics engineering, neuroscience and cognitive sciences, medicine, and social sciences, and provide the reader with an integrated understanding of the importance that intelligent computing has in the sustainable development of current societies. It discusses the emerging research exploring the theoretical and practical aspects of successfully implementing new and innovative intelligent techniques in a variety of sectors, including IoT, manufacturing, optimization, and healthcare. Audience It is a pivotal reference source for IT specialists, industry professionals, managers, executives, researchers, scientists, and engineers seeking current research in emerging perspectives in the field of artificial intelligence in the areas of Internet of Things, renewable energy, optimization, and smart cities.
Author: Suvrit Sra
Publisher: MIT Press
ISBN: 026201646X
Category : Computers
Languages : en
Pages : 509
Get Book Here
Book Description
An up-to-date account of the interplay between optimization and machine learning, accessible to students and researchers in both communities. The interplay between optimization and machine learning is one of the most important developments in modern computational science. Optimization formulations and methods are proving to be vital in designing algorithms to extract essential knowledge from huge volumes of data. Machine learning, however, is not simply a consumer of optimization technology but a rapidly evolving field that is itself generating new optimization ideas. This book captures the state of the art of the interaction between optimization and machine learning in a way that is accessible to researchers in both fields. Optimization approaches have enjoyed prominence in machine learning because of their wide applicability and attractive theoretical properties. The increasing complexity, size, and variety of today's machine learning models call for the reassessment of existing assumptions. This book starts the process of reassessment. It describes the resurgence in novel contexts of established frameworks such as first-order methods, stochastic approximations, convex relaxations, interior-point methods, and proximal methods. It also devotes attention to newer themes such as regularized optimization, robust optimization, gradient and subgradient methods, splitting techniques, and second-order methods. Many of these techniques draw inspiration from other fields, including operations research, theoretical computer science, and subfields of optimization. The book will enrich the ongoing cross-fertilization between the machine learning community and these other fields, and within the broader optimization community.
