Evolutionary Multi-objective Bi-level Optimization for Efficient Deep Neural Network Architecture Design PDF Download
Are you looking for read ebook online? Search for your book and save it on your Kindle device, PC, phones or tablets. Download Evolutionary Multi-objective Bi-level Optimization for Efficient Deep Neural Network Architecture Design PDF full book. Access full book title Evolutionary Multi-objective Bi-level Optimization for Efficient Deep Neural Network Architecture Design by Zhichao Lu. Download full books in PDF and EPUB format.
Author: Zhichao Lu
Publisher:
ISBN:
Category : Electronic dissertations
Languages : en
Pages : 142
Get Book Here
Book Description
Deep convolutional neural networks (CNNs) are the backbones of deep learning (DL) paradigms for numerous vision tasks, including object recognition, detection, segmentation, etc. Early advancements in CNN architectures are primarily driven by human expertise and elaborate design. Recently, neural architecture search (NAS) was proposed with the aim of automating the network design process and generating task-dependent architectures. While existing approaches have achieved competitive performance, they are still impractical to real-world deployment for three reasons: (1) the generated architectures are solely optimized for predictive performance, resulting in inefficiency in utilizing hardware resources---i.e. energy consumption, latency, memory size, etc.; (2) the search processes require vast computational resources in most approaches; (3) most existing approaches require one complete search for each deployment specification of hardware or requirement. In this dissertation, we propose an efficient evolutionary NAS algorithm to address the aforementioned limitations. In particular, we first introduce Pareto-optimization to NAS, leading to a diverse set of architectures, trading-off multiple objectives, being obtained simultaneously in one run. We then improve the algorithm's search efficiency through surrogate models. We finally integrate a transfer learning scheme to the algorithm that allows a new task to leverage previous search efforts that further improves both the performance of the obtained architectures and search efficiency. Therefore, the proposed algorithm enables an automated and streamlined process to efficiently generate task-specific custom neural network models that are competitive under multiple objectives.
Author: Zhichao Lu
Publisher:
ISBN:
Category : Electronic dissertations
Languages : en
Pages : 142
Get Book Here
Book Description
Deep convolutional neural networks (CNNs) are the backbones of deep learning (DL) paradigms for numerous vision tasks, including object recognition, detection, segmentation, etc. Early advancements in CNN architectures are primarily driven by human expertise and elaborate design. Recently, neural architecture search (NAS) was proposed with the aim of automating the network design process and generating task-dependent architectures. While existing approaches have achieved competitive performance, they are still impractical to real-world deployment for three reasons: (1) the generated architectures are solely optimized for predictive performance, resulting in inefficiency in utilizing hardware resources---i.e. energy consumption, latency, memory size, etc.; (2) the search processes require vast computational resources in most approaches; (3) most existing approaches require one complete search for each deployment specification of hardware or requirement. In this dissertation, we propose an efficient evolutionary NAS algorithm to address the aforementioned limitations. In particular, we first introduce Pareto-optimization to NAS, leading to a diverse set of architectures, trading-off multiple objectives, being obtained simultaneously in one run. We then improve the algorithm's search efficiency through surrogate models. We finally integrate a transfer learning scheme to the algorithm that allows a new task to leverage previous search efforts that further improves both the performance of the obtained architectures and search efficiency. Therefore, the proposed algorithm enables an automated and streamlined process to efficiently generate task-specific custom neural network models that are competitive under multiple objectives.
Author: Maria Gabriela Baldeón Calisto
Publisher:
ISBN:
Category : Diagnostic imaging
Languages : en
Pages : 120
Get Book Here
Book Description
Deep neural networks have become very successful at solving many complex tasks such as image classification, image segmentation, and speech recognition. These models are composed of multiple layers that have the capacity to learn increasingly higher-level features, without prior handcrafted specifications. However, the success of a deep neural network relies on finding the proper configuration for the task in hand. Given the vast number of hyperparameters and the massive search space, manually designing or fine-tuning deep learning architectures requires extensive knowledge, time, and computational resources. There is a growing interest in developing methods that automatically design a neural network ́s architecture, known as neural architecture search (NAS). NAS is usually modeled as a single-objective optimization problem where the aim is to find an architecture that maximizes the prediction ́s accuracy. However, most deep learning applications require accurate as well as efficient architectures to reduce memory consumption and enable their use in computationally-limited environments. This has led to the need to model NAS as a multiple objective problem that optimizes both the predictive performance and efficiency of the network. Furthermore, most NAS framework have focused on either optimizing the micro-structure (structure of the basic cell), or macro-structure (optimal number of cells and their connection) of the architecture. Consequently, manual engineering is required to find the topology of the non-optimized structure. Although NAS has demonstrated great potential in automatically designing an architecture, it remains a computationally expensive and time-consuming process because it requires training and evaluating many potential configurations. Recent work has focused on improving the search time of NAS algorithms, but most techniques have been developed and applied only for single-objective optimization problems. Given that optimizing multiple objectives has a higher complexity and requires more iterations to approximate the Pareto Front, it is critical to investigate algorithms that decrease the search time of multiobjective NAS. One critical application of deep learning is medical image segmentation. Segmentation of medical images provides valuable information for various critical tasks such as analyzing anatomical structures, monitoring disease progression, and predicting patient outcomes. Nonetheless, achieving accurate segmentation is challenging due to the inherent variability in appearance, shape, and location of the region of interest (ROI) between patients and the differences in imagining equipment and acquisition protocols. Therefore, neural networks are usually tailored to a specific application, anatomical region, and image modality. Moreover, medical image data is often volumetric requiring expensive 3D operations that result in large and complex architectures. Hence, training and deploying them requires considerable storage and memory bandwidth that makes them less suitable for clinical applications. To overcome these challenges, the main goal of this research is to automatically design accurate and efficient deep neural networks using multiobjective optimization algorithms for medical image segmentation. The proposed research consists of three major objectives: (1) to design a deep neural network that uses a multiobjective evolutionary based algorithm to automatically adapt to different medical image datasets while minimizing the model’s size; (2) to design a self-adaptive 2D-3D Fully Convolutional network (FCN) ensemble that incorporates volumetric information and optimizes both the performance and the size of the architecture; and (3) to design an efficient multiobjective neural architecture search framework that decreases the search time while simultaneously optimizing the micro- and macro-structure of the neural architecture. For the first objective, a multiobjective adaptive convolutional neural network named AdaResU-Net is presented for 2D medical image segmentation. The proposed AdaResU-Net is comprised of a fixed architecture and a learning framework that adjusts the hyperparameters to a particular training dataset using a multiobjective evolutionary based algorithm (MEA algorithm). The MEA algorithm evolves the AdaResU-Net network to optimize both the segmentation accuracy and model size. In the second objective, a self-adaptive ensemble of 2D-3D FCN named AdaEn-Net is proposed for 3D medical image segmentation. The AdaEn-Net is comprised of a 2D FCN that extracts intra-slice and long-range 2D context, and a 3D FCN architecture that exploits inter-slice and volumetric information. The 2D and 3D FCN architectures are automatically fitted for a specific medical image segmentation task by simultaneously optimizing the expected segmentation error and size of the network using the MEA algorithm. Finally, for the third objective, an efficient multiobjective neural architecture search framework named EMONAS is presented for 3D medical image segmentation. EMONAS has two main components, a novel search space that includes the hyperparameters that define the micro- and macro-structure of the architecture, and a Surrogate-assisted multiobjective evolutionary based algorithm (SaMEA algorithm) that efficiently searches for the best hyperparameter values using a Random Forest surrogate and guiding selection probabilities. The broader impact of the proposed research is as follows: (1) automating the design of deep neural networks’ architecture and hyperparameters to improve the performance and efficiency of the models; and (2) increase the accessibility of deep learning to a broader range of organizations and people by reducing the need of expert knowledge and GPU time when automatically designing deep neural networks. In the medical area, the proposed models aim to improve the automatic extraction of data from medical images to potentially enhance diagnosis, treatment planning and survival prediction of various diseases such as cardiac disease and prostate cancer. Although the proposed techniques are applied to medical image segmentation tasks, they can also be implemented in other applications where accurate and resource-efficient deep neural networks are needed such as autonomous navigation, augmented reality and internet-of-things.
Author: Hitoshi Iba
Publisher: Springer Nature
ISBN: 9811536856
Category : Computers
Languages : en
Pages : 437
Get Book Here
Book Description
This book delivers the state of the art in deep learning (DL) methods hybridized with evolutionary computation (EC). Over the last decade, DL has dramatically reformed many domains: computer vision, speech recognition, healthcare, and automatic game playing, to mention only a few. All DL models, using different architectures and algorithms, utilize multiple processing layers for extracting a hierarchy of abstractions of data. Their remarkable successes notwithstanding, these powerful models are facing many challenges, and this book presents the collaborative efforts by researchers in EC to solve some of the problems in DL. EC comprises optimization techniques that are useful when problems are complex or poorly understood, or insufficient information about the problem domain is available. This family of algorithms has proven effective in solving problems with challenging characteristics such as non-convexity, non-linearity, noise, and irregularity, which dampen the performance of most classic optimization schemes. Furthermore, EC has been extensively and successfully applied in artificial neural network (ANN) research —from parameter estimation to structure optimization. Consequently, EC researchers are enthusiastic about applying their arsenal for the design and optimization of deep neural networks (DNN). This book brings together the recent progress in DL research where the focus is particularly on three sub-domains that integrate EC with DL: (1) EC for hyper-parameter optimization in DNN; (2) EC for DNN architecture design; and (3) Deep neuroevolution. The book also presents interesting applications of DL with EC in real-world problems, e.g., malware classification and object detection. Additionally, it covers recent applications of EC in DL, e.g. generative adversarial networks (GAN) training and adversarial attacks. The book aims to prompt and facilitate the research in DL with EC both in theory and in practice.
Author: Ngoc Thanh Nguyen
Publisher: Springer Nature
ISBN: 3031160142
Category : Computers
Languages : en
Pages : 863
Get Book Here
Book Description
This book constitutes the refereed proceedings of the 14th International Conference on Computational Collective Intelligence, ICCCI 2022, held in Hammamet, Tunisia, in September 2022. The 56 full papers and 10 short papers were carefully reviewed and selected from 420 submissions. The papers are grouped in topical sections on collective intelligence and collective decision-making; deep learning techniques; natural language processing; data minning and machine learning; knowledge engineering and semantic web; computer vision techniques; social networks and intelligent systems; cybersecurity and internet of things; cooperative strategies for decision making and optimization; computational intelligence for digital content understanding; applications for industry 4.0.
Author: Frank Hutter
Publisher: Springer
ISBN: 3030053180
Category : Computers
Languages : en
Pages : 223
Get Book Here
Book Description
This open access book presents the first comprehensive overview of general methods in Automated Machine Learning (AutoML), collects descriptions of existing systems based on these methods, and discusses the first series of international challenges of AutoML systems. The recent success of commercial ML applications and the rapid growth of the field has created a high demand for off-the-shelf ML methods that can be used easily and without expert knowledge. However, many of the recent machine learning successes crucially rely on human experts, who manually select appropriate ML architectures (deep learning architectures or more traditional ML workflows) and their hyperparameters. To overcome this problem, the field of AutoML targets a progressive automation of machine learning, based on principles from optimization and machine learning itself. This book serves as a point of entry into this quickly-developing field for researchers and advanced students alike, as well as providing a reference for practitioners aiming to use AutoML in their work.
Author: Hamido Fujita
Publisher: Springer Nature
ISBN: 3031085302
Category : Computers
Languages : en
Pages : 932
Get Book Here
Book Description
This book constitutes the thoroughly refereed proceedings of the 35th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2022, held in Kitakyushu, Japan, in July 2022. The 67 full papers and 11 short papers presented were carefully reviewed and selected from 127 submissions. The IEA/AIE 2022 conference focuses on focuses on applications of applied intelligent systems to solve real-life problems in all areas including business and finance, science, engineering, industry, cyberspace, bioinformatics, automation, robotics, medicine and biomedicine, and human-machine interactions.
Author: Jason Zhi Liang
Publisher:
ISBN:
Category :
Languages : en
Pages : 356
Get Book Here
Book Description
Deep neural networks (DNNs) have produced state-of-the-art results in many benchmarks and problem domains. However, the success of DNNs depends on the proper configuration of its architecture and hyperparameters. DNNs are often not used to their full potential because it is difficult to determine what architectures and hyperparameters should be used. While several approaches have been proposed, computational complexity of searching large design spaces makes them impractical for large modern DNNs. This dissertation introduces an efficient evolutionary algorithm (EA) for simultaneous optimization of DNN architecture and hyperparameters. It builds upon extensive past research of evolutionary optimization of neural network structure. Various improvements to the core algorithm are introduced, including: (1) discovering DNN architectures of arbitrary complexity; (1) generating modular, repetitive modules commonly seen in state-of-the-art DNNs; (3) extending to the multitask learning and multiobjective optimization domains; (4) maximizing performance and reducing wasted computation through asynchronous evaluations. Experimental results in image classification, image captioning, and multialphabet character recognition show that the approach is able to evolve networks that are competitive with or even exceed hand-designed networks. Thus, the method enables an automated and streamlined process to optimize DNN architectures for a given problem and can be widely applied to solve harder tasks
Author: Léo Souquet
Publisher:
ISBN:
Category :
Languages : en
Pages : 0
Get Book Here
Book Description
This last decade the complexity of the problems increased with the increase of the CPUs' power and the decrease of memory costs. The appearance of clouds infrastructures provide the possibility to solve large scale problems. However, most of the exact and stochastic optimization algorithms see their performances go down with the increase of the dimension of the problems. Evolutionary approaches and other bio-inspired approaches were widely used to solve large scale problems without lot of success. Indeed, the complexity of large scale problems non convex functions comes from the fact that local minima (and maxima) are rare.In this thesis, we propose to tackle large scale problems by designing a new approach based on fractal decomposition of the search space using hyperspheres. This geometrical decomposition allows the algorithm to be intrinsically parallel for solving large scale problems. The proposed algorithm called Fractal Decomposition Algorithm (FDA). It is a deterministic algorithm with low complexity and easy to implement. FDA has been tested on several functions, compared with competing metaheuristics and showed good results on problems with dimensions from 50 to 1000. Its structure allows it to be naturally parallelized, which resulted in developing two new versions: PFDA for multi-threaded environments and MA-FDA for multi-nodes environments. Then, the proposed algorithm was adapted to solve multi-objective problems. Two algorithms were proposed: the first one is based on scalarization and has been distributed on multi-node architecture virtual environments known as containers. While the second approach is based on sorting non-dominated solutions.Moreover, we applied FDA to the optimization of the hyperparameters of deep learning architectures with a focus on Convolutional Neural Networks. We present an approach using bi-level optimization separating the architecture search composed of discrete parameters from hyperparameter optimization with the continuous parameters. This is motivated by the fact that automating the construction of deep neural architecture has been an important focus over recent years as doing it manually is very time consuming and prone to error.
Author: Vivienne Sze
Publisher: Springer Nature
ISBN: 3031017668
Category : Technology & Engineering
Languages : en
Pages : 254
Get Book Here
Book Description
This book provides a structured treatment of the key principles and techniques for enabling efficient processing of deep neural networks (DNNs). DNNs are currently widely used for many artificial intelligence (AI) applications, including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Therefore, techniques that enable efficient processing of deep neural networks to improve key metrics—such as energy-efficiency, throughput, and latency—without sacrificing accuracy or increasing hardware costs are critical to enabling the wide deployment of DNNs in AI systems. The book includes background on DNN processing; a description and taxonomy of hardware architectural approaches for designing DNN accelerators; key metrics for evaluating and comparing different designs; features of DNN processing that are amenable to hardware/algorithm co-design to improve energy efficiency and throughput; and opportunities for applying new technologies. Readers will find a structured introduction to the field as well as formalization and organization of key concepts from contemporary work that provide insights that may spark new ideas.
Author: Kalyanmoy Deb
Publisher: John Wiley & Sons
ISBN: 9780471873396
Category : Mathematics
Languages : en
Pages : 540
Get Book Here
Book Description
Optimierung mit mehreren Zielen, evolutionäre Algorithmen: Dieses Buch wendet sich vorrangig an Einsteiger, denn es werden kaum Vorkenntnisse vorausgesetzt. Geboten werden alle notwendigen Grundlagen, um die Theorie auf Probleme der Ingenieurtechnik, der Vorhersage und der Planung anzuwenden. Der Autor gibt auch einen Ausblick auf Forschungsaufgaben der Zukunft.
