More Web Proxy on the site http://driver.im/

research-article

A Comparative Visual Analytics Framework for Evaluating Evolutionary Processes in Multi-Objective Optimization

Authors:

Ran ChengAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 30, Issue 1

Pages 661 - 671

https://doi.org/10.1109/TVCG.2023.3326921

Published: 24 October 2023 Publication History

Abstract

Evolutionary multi-objective optimization (EMO) algorithms have been demonstrated to be effective in solving multi-criteria decision-making problems. In real-world applications, analysts often employ several algorithms concurrently and compare their solution sets to gain insight into the characteristics of different algorithms and explore a broader range of feasible solutions. However, EMO algorithms are typically treated as black boxes, leading to difficulties in performing detailed analysis and comparisons between the internal evolutionary processes. Inspired by the successful application of visual analytics tools in explainable AI, we argue that interactive visualization can significantly enhance the comparative analysis between multiple EMO algorithms. In this paper, we present a visual analytics framework that enables the exploration and comparison of evolutionary processes in EMO algorithms. Guided by a literature review and expert interviews, the proposed framework addresses various analytical tasks and establishes a multi-faceted visualization design to support the comparative analysis of intermediate generations in the evolution as well as solution sets. We demonstrate the effectiveness of our framework through case studies on benchmarking and real-world multi-objective optimization problems to elucidate how analysts can leverage our framework to inspect and compare diverse algorithms.

References

[1]

Geatpy. [Online]. Available: https://github.com/geatpy-dev/geatpy. Accessed: 2023-03-10. 3.

[2]

A. Abdolmaleki, S. Huang, L. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. Martins, N. Heess, R. Hadsell, and M. Riedmiller. A distributional view on multi-objective policy optimization. In Proceedings of the International Conference on Machine Learning, pp. 11–22, 2020. 2.

[3]

N. Andrienko, G. Andrienko, S. Miksch, H. Schumann, and S. Wrobel. A theoretical model for pattern discovery in visual analytics. Visual Informatics, 5 (1): pp. 23–42, 2021. 2.

[4]

B. Bach, C. Shi, N. Heulot, T. Madhyastha, T. Grabowski, and P. Dragicevic. Time Curves: Folding time to visualize patterns of temporal evolution in data. IEEE Transactions on Visualization and Computer Graphics, 22 (1): pp. 559–568, 2016. 6.

Digital Library

[5]

J. Blank and K. Deb. Pymoo: Multi-objective optimization in python. IEEE Access, 8: 89497–89509, 2020. 3.

[6]

M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying density-based local outliers. In Proceedings of the ACM SIGMOD International Conference on Management of Data, pp. 93–104, 2000. 7.

Digital Library

[7]

G. C. Cawley and N. L. Talbot. On over-fitting in model selection and subsequent selection bias in performance evaluation. The Journal of Machine Learning Research, 11: 2079–2107, 2010. 2.

Digital Library

[8]

A. Chatzimparmpas, R. M. Martins, K. Kucher, and A. Kerren. VisEvol: Visual analytics to support hyperparameter search through evolutionary optimization. Computer Graphics Forum, 40 (3): pp. 201–214, 2021. 2.

[9]

S. Chen, D. Amid, O. M. Shir, L. Limonad, D. Boaz, A. Anaby-Tavor, and T. Schreck. Self-organizing maps for multi-objective Pareto frontiers. In Proceedings of the IEEE Pacific Visualization Symposium, pp. 153–160, 2013. 2.

[10]

J. Choo and S. Liu. Visual analytics for explainable deep learning. IEEE Computer Graphics and Applications, 38 (04): pp. 84–92, 2018. 2.

[11]

L. Cibulski, H. Mitterhofer, T. May, and J. Kohlhammer. PAVED: Pareto front visualization for engineering design. Computer Graphics Forum, 39 (3): pp. 405–416, 2020. 2, 9.

[12]

C. A. Coello Coello and M. Reyes Sierra. A study of the parallelization of a coevolutionary multi-objective evolutionary algorithm. In Proceedings of the Mexican International Conference on Artificial Intelligence, pp. 688–697, 2004. 4.

[13]

K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. Scalable multi-objective optimization test problems. In Proceedings of the Congress on Evolutionary Computation, vol. 1, pp. 825–830, 2002. 7.

[14]

A. Endert, W. Ribarsky, C. Turkay, B. W. Wong, I. Nabney, I. D. Blanco, and F. Rossi. The state of the art in integrating machine learning into visual analytics. Computer Graphics Forum, 36 (8): pp. 458–486, dec 2017. 2.

[15]

J. Feydy, T. Séjourné, F.-X. Vialard, S.-I. Amari, A. Trouve, and G. Peyré. Interpolating between optimal transport and MMD using Sinkhorn divergences. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 2681–2690, 2019. 9.

[16]

M. Gleicher. Considerations for visualizing comparison. IEEE Transactions on Visualization and Computer Graphics, 24 (1): pp. 413–423, 2018. 3, 4.

[17]

M. Gleicher, A. Barve, X. Yu, and F. Heimerl. Boxer: Interactive comparison of classifier results. Computer Graphics Forum, 39 (3): pp. 181–193, 2020. 2.

[18]

M. Gong, L. Zhang, L. Jiao, and S. Gou. Solving multiobjective clustering using an immune-inspired algorithm. In Proceedings of the IEEE Congress on Evolutionary Computation, pp. 15–22, 2007. 8.

[19]

R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys, 51 (5), 2018. 2.

[20]

C. He, Y. Tian, H. Wang, and Y. Jin. A repository of real-world datasets for data-driven evolutionary multiobjective optimization. Complex and Intelligent Systems, 6 (1): pp. 189–197, 2020. 8.

[21]

Z. He and G. G. Yen. Visualization and performance metric in many-objective optimization. IEEE Transactions on Evolutionary Computation, 20 (3): pp. 386–402, 2016. 3.

Digital Library

[22]

F. M. Hohman, M. Kahng, R. Pienta, and D. H. Chau. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics, 25 (8): pp. 2674–2693, 2018. 2.

Digital Library

[23]

A. Ibrahim, S. Rahnamayan, M. V. Martin, and K. Deb. 3D-RadVis: Visualization of Pareto front in many-objective optimization. In Proceedings of the IEEE Congress on Evolutionary Computation, pp. 736–745, 2016. 2,9.

[24]

M. Kahng, N. Thorat, D. Horng, P. Chau, F. B. Vi, and M. Wattenberg. GAN Lab: Understanding complex deep generative models using interactive visual experimentation. IEEE Transactions on Visualization and Computer Graphics, 25 (1): pp. 310–320, 2019. 2.

Digital Library

[25]

T. Kamada and S. Kawai. An algorithm for drawing general undirected graphs. Information Processing Letters, 31 (1): pp. 7–15, 1989. 6.

Digital Library

[26]

H. Kim, J. Choo, C. Lee, H. Lee, C. K. Reddy, and H. Park. PIVE: Per-iteration visualization environment for real-time interactions with dimension reduction and clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1001–1009, 2017. 3.

[27]

L. Kotthoff, C. Thornton, H. H. Hoos, F. Hutter, and K. Leyton-Brown. Auto-Weka: Automatic model selection and hyperparameter optimization in WEKA. Automated Machine Learning: Methods, Systems, Challenges, pp. 81–95, 2019. 2.

[28]

B. La Rosa, G. Blasilli, R. Bourqui, D. Auber, G. Santucci, R. Capobianco, E. Bertini, R. Giot, and M. Angelini. State of the art of visual analytics for explainable deep learning. Computer Graphics Forum, 42 (1): pp. 319–355, 2023. 2.

[29]

K. Li, R. Wang, T. Zhang, and H. Ishibuchi. Evolutionary many-objective optimization: A comparative study of the state-of-the-art. IEEE Access, 6: 26194–26214, 2018. 2,3.

[30]

M. Li and X. Yao. Quality evaluation of solution sets in multiobjective optimisation: A survey. ACM Computing Surveys, 52 (2), 2019. 2, 3, 4.

Digital Library

[31]

M. Li, L. Zhen, and X. Yao. How to read many-objective solution sets in parallel coordinates. IEEE Computational Intelligence Magazine, 12 (4): pp. 88–100, 2017. 2.

Digital Library

[32]

D. Liu, W. Cui, K. Jin, Y. Guo, and H. Qu. DeepTracker: Visualizing the training process of convolutional neural networks. ACM Transactions on Intelligent Systems and Technology, 10 (1): pp. 1–25, Nov. 2018. 2.

[33]

M. Liu, J. Shi, K. Cao, J. Zhu, and S. Liu. Analyzing the training processes of deep generative models. IEEE Transactions on Visualization and Computer Graphics, 24 (1): pp. 77–87, 2018. 2.

[34]

S. Liu, J. Xiao, J. Liu, X. Wang, J. Wu, and J. Zhu. Visual diagnosis of tree boosting methods. IEEE Transactions on Visualization and Computer Graphics, 24 (1): pp. 163–173, 2017. 7.

[35]

A. Lotov, V. A. Bushenkov, and G. K. Kamenev. Interactive Decision Maps: Approximation and Visualization of Pareto Frontier. Kluwer Academic Publishers, 2004. 2.

[36]

K. Lu, K. Reda, O. Deussen, and Y. Wang. Interactive context-preserving color highlighting for multiclass scatterplots. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems, 2023. 9.

[37]

Y. Lu, R. Garcia, B. Hansen, M. Gleicher, and R. Maciejewski. The state-of-the-art in predictive visual analytics. Computer Graphics Forum, 36 (3): pp. 539–562, 2017. 2.

Digital Library

[38]

Z. Lu, R. Cheng, Y. Jin, K. C. Tan, and K. Deb. Neural architecture search as multiobjective optimization benchmarks: Problem formulation and performance assessment. IEEE Transactions on Evolutionary Computation, 2022. early access. 2.

[39]

Y. Ma, A. Fan, J. He, A. R. Nelakurthi, and R. Maciejewski. A visual analytics framework for explaining and diagnosing transfer learning processes. IEEE Transactions on Visualization and Computer Graphics, 27 (2): pp. 1385–1395, 2021. 3.

[40]

L. McInnes and J. Healy. Accelerated hierarchical density based clustering. in Proceedings of the IEEE International Conference on Data Mining Workshops, pp. 33–42. IEEE, 2017. 6.

[41]

J.-B. Mouret. Novelty-based multiobjectivization. In New Horizons in Evolutionary Robotics: Extended Contributions from the EvoDeRob Workshop, pp. 139–154. Springer, 2011. 8.

[42]

T. Muhlbacher, H. Piringer, S. Gratzl, M. Sedlmair, and M. Streit. Opening the black box: Strategies for increased user involvement in existing algorithm implementations. IEEE Transactions on Visualization and Computer Graphics, 20 (12): pp. 1643–1652, 2014. 2.

[43]

S. Murugesan, S. Malik, F. Du, E. Koh, and T. M. Lai. DeepCompare: Visual and interactive comparison of deep learning model performance. IEEE Computer Graphics and Applications, 39 (5): pp. 47–59, Sept. 2019. 3.

Digital Library

[44]

D. Nagar, P. Ramu, and K. Deb. Visualization and analysis of Pareto-optimal fronts using interpretable self-organizing map (isom). Swarm and Evolutionary Computation, 76: p. 101202, 2022. 2,9.

[45]

J. P. Ono, S. Castelo, R. Lopez, E. Bertini, J. Freire, and C. Silva. Pipeline-pro filer: A visual analytics tool for the exploration of automl pipelines. IEEE Transactions on Visualization and Computer Graphics, 27 (2): pp. 390–400, 2021. 3.

[46]

A. Pister, P. Buono, J. D. Fekete, C. Plaisant, and P. Valdivia. Integrating prior knowledge in mixed-initiative social network clustering. IEEE Transactions on Visualization and Computer Graphics, 27 (2): 1775–1785, 2021. 2.

[47]

H. Pohlheim. Visualization of evolutionary algorithms - set of standard techniques and multidimensional visualization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 533–540, 1999. 2.

[48]

S. Raschka. Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:, 2018. 2.

[49]

M. J. Reddy and D. N. Kumar. Optimal reservoir operation using multi-objective evolutionary algorithm. Water Resources Management, 20 (6): pp. 861–878, 2006. 1.

[50]

J. R. Schott. Fault tolerant design using single and multicriteria genetic algorithm optimization. Master's thesis, Massachusetts Institute of Technology, 1995. 4.

[51]

O. Sener and V. Koltun. Multi-task learning as multi-objective optimization. Advances in Neural Information Processing Systems, 31: p. 525–536, 2018. 2.

[52]

B. Shneiderman. The eyes have it: a task by data type taxonomy for information visualizations. In Proceedings of the IEEE Symposium on Visual Languages, pp. 336–343, 1996. 4.

[53]

A. K. A. Talukder and K. Deb. PaletteViz: A visualization method for functional understanding of high-dimensional Pareto-optimal data-sets to aid multi-criteria decision making. IEEE Computational Intelligence Magazine, 15 (2): pp. 36–48, 2020. 2.

[54]

Y. Tian, R. Cheng, X. Zhang, and Y. Jin. PlatEMO: A MATLAB platform for evolutionary multi-objective optimization. IEEE Computational Intelligence Magazine, 12 (4): pp. 73–87, 2017. 3,7.

Digital Library

[55]

Y. Tian, L. Si, X. Zhang, R. Cheng, C. He, K. C. Tan, and Y. Jin. Evolutionary large-scale multi-objective optimization: A survey. ACM Computing Surveys, 54 (8), 2021. 2, 3, 4.

[56]

T. Tusar and B. Filipic. Visualization of Pareto front approximations in evolutionary multiobjective optimization: A critical review and the prosection method. IEEE Transactions on Evolutionary Computation, 19 (2): pp. 225–245, 2015. 2.

Digital Library

[57]

F.-Y. Tzeng and K.-L. Ma. Opening the black box - data driven visualization of neural networks. In Proceedings of the IEEE Visualization Conference, pp. 383–390, 2005. 2.

[58]

L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9 (11), 2008. 5.

[59]

J. Vincalek, S. Walton, and B. Evans. It's the journey not the destination: Building genetic algorithms practitioners can trust. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 231–232, 2021. 2.

[60]

D. J. Walker, R. M. Everson, and J. E. Fieldsend. Visualizing mutually nondominating solution sets in many-objective optimization. IEEE Transactions on Evolutionary Computation, 17 (2): pp. 165–184, 2013. 2.

Digital Library

[61]

J. Wang, L. Wang, Y. Zheng, C. C. M. Yeh, S. Jain, and W. Zhang. Learning-from-disagreement: A model comparison and visual analytics framework. IEEE Transactions on Visualization and Computer Graphics, pp. 1–16, 2022. 2.

[62]

Q. Wang, Z. Chen, Y. Wang, and H. Qu. A survey on ML4VIS: Applying machine learning advances to data visualization. IEEE Transactions on Visualization and Computer Graphics, 28 (12): pp. 5134–5153, 2022. 2.

Digital Library

[63]

Q. Wang, Y. Ming, Z. Jin, Q. Shen, D. Liu, M. J. Smith, K. Veeramachaneni, and H. Qu. ATMSeer: Increasing transparency and controllability in automated machine learning. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems, pp. 1–12, 2019. 3.

[64]

X. Wang, W. Chen, J. Xia, Z. Chen, D. Xu, X. Wu, M. Xu, and T. Schreck. ConceptExplorer: Visual analysis of concept drifts in multi-source time-series data. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology, pp. 1–11, 2020. 3.

[65]

Y. Wang, T. Q. Peng, H. Lu, H. Wang, X. Xie, H. Qu, and Y. Wu. Seek for success: A visualization approach for understanding the dynamics of academic careers. IEEE Transactions on Visualization and Computer Graphics, 28 (1): pp. 475–485, 2022. 6.

Digital Library

[66]

Z. J. Wang, R. Turko, O. Shaikh, H. Park, N. Das, F. Hohman, M. Kahng, and D. H. Polo Chau. CNN Explainer: Learning convolutional neural networks with interactive visualization. IEEE Transactions on Visualization and Computer Graphics, 27 (2): pp. 1396–1406, 2021. 2.

[67]

D. Weng, C. Zheng, Z. Deng, M. Ma, J. Bao, Y. Zheng, M. Xu, and Y. Wu. Towards better bus networks: A visual analytics approach. IEEE Transactions on Visualization and Computer Graphics, 27 (2): pp. 817–827, 2021. 2.

[68]

T. Xie, Y. Ma, J. Kang, H. Tong, and R. Maciejewski. FairRankVis: A visual analytics framework for exploring algorithmic fairness in graph mining models. IEEE Transactions on Visualization and Computer Graphics, 28 (1): pp. 368–377, 2022. 2.

Digital Library

[69]

X. Xuan, X. Zhang, O. H. Kwon, and K. L. Ma. VAC-CNN: A visual analytics system for comparative studies of deep convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics, 28 (6): pp. 2326–2337, 2022. 3.

Digital Library

[70]

D. Yadav, D. Nagar, P. Ramu, and K. Deb. Visualization-aided multi-criteria decision-making using interpretable self-organizing maps. European Journal of Operational Research, 309 (3): pp. 1183–1200, 2023. 2.

[71]

W. Yang, Z. Li, M. Liu, Y. Lu, K. Cao, R. Maciejewski, and S. Liu. Diagnosing concept drift with visual analytics. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology, pp. 12–23, 2020. 3.

[72]

Y. Ye, R. Huang, and W. Zeng. VISAtlas: An image-based exploration and query system for large visualization collections via neural image embedding. IEEE Transactions on Visualization and Computer Graphics, 2022. early access. 2.

[73]

J. Yuan, C. Chen, W. Yang, M. Liu, J. Xia, and S. Liu. A survey of visual analytics techniques for machine learning. Computational Visual Media, 7: pp. 3–36, 2021. 2.

[74]

J. Yuan, S. Xiang, J. Xia, L. Yu, and S. Liu. Evaluation of sampling methods for scatterplots. IEEE Transactions on Visualization and Computer Graphics, 27 (2): pp. 1720–1730, 2021. 7, 9.

[75]

H. Zeng, H. Haleem, X. Plantaz, N. Cao, and H. Qu. CNNComparator: Comparative analytics of convolutional neural networks. In Proceedings of the Workshop on Visual Analytics for Deep Learning, 2017. 3.

[76]

J. Zhang, Y. Wang, P. Molino, L. Li, and D. S. Ebert. Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models. IEEE Transactions on Visualization and Computer Graphics, 25 (1): pp. 364–373, 2019. 2.

Digital Library

[77]

E. Zitzler, K. Deb, and L. Thiele. Comparison of multiobjective evolutionary algorithms: empirical results. Evolutionary Computation, 8 (2): 173–195, 2000. 2,3,4.

Digital Library

[78]

E. Zitzler, M. Laumanns, and S. Bleuler. A tutorial on evolutionary multi-objective optimization. In Proceedins of Metaheuristics for Multiobjective Optimisation, pp. 3–37, 2004. 3.

[79]

E. Zitzler and L. Thiele. Multiobjective optimization using evolutionary algorithms - a comparative case study. In Proceedings of the International Conference on Parallel Problem Solving from Nature, pp. 292–301, 1998. 4.

Digital Library

Cited By

Zhang ZYang FCheng RMa Y(2024)ParetoTracker: Understanding Population Dynamics in Multi-Objective Evolutionary Algorithms Through Visual AnalyticsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345614231:1(820-830)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456142

Index Terms

A Comparative Visual Analytics Framework for Evaluating Evolutionary Processes in Multi-Objective Optimization
1. Human-centered computing

Index terms have been assigned to the content through auto-classification.

Recommendations

Hyper multi-objective evolutionary algorithm for multi-objective optimization problems

Multi-objective optimization problems (MOPs) are very common in practice. To solve MOPs, many kinds of multi-objective evolutionary algorithms (MOEAs) are proposed. However, different MOEAs have different performances for different MOPs. Therefore, it ...
A comparative study of constrained multi-objective evolutionary algorithms on constrained multi-objective optimization problems
2017 IEEE Congress on Evolutionary Computation (CEC)
Solving constrained multi-objective optimization problems is a difficult task, it needs to simultaneously optimize multiple conflicting objectives and a number of constraints. This paper first reviews a number of popular constrained multi-objective ...
A robust evolutionary framework for multi-objective optimization
GECCO '08: Proceedings of the 10th annual conference on Genetic and evolutionary computation

Evolutionary multi-objective optimization (EMO) methodologies, suggested in the beginning of Nineties, focussed on the task of finding a set of well-converged and well-distributed set of solutions using evolutionary optimization principles. Of the EMO ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 30, Issue 1

Jan. 2024

1456 pages

Issue’s Table of Contents

1077-2626 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 24 October 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZYang FCheng RMa Y(2024)ParetoTracker: Understanding Population Dynamics in Multi-Objective Evolutionary Algorithms Through Visual AnalyticsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345614231:1(820-830)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456142

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents