Explainable artificial intelligence

Explainable AI (XAI) refers to methods and techniques in the application of artificial intelligence technology (AI) such that the results of the solution can be understood by humans. It contrasts with the concept of the "black box" in machine learning where even its designers cannot explain why an AI arrived at a specific decision.[1] XAI may be an implementation of the social right to explanation.[2] XAI is relevant even if there is no legal right or regulatory requirement—for example, XAI can improve the user experience of a product or service by helping end users trust that the AI is making good decisions. This way the aim of XAI is to explain what has been done, what is done right now, what will be done next and unveil the information the actions are based on.[3] These characteristics make it possible (i) to confirm existing knowledge (ii) to challenge existing knowledge and (iii) to generate new assumptions.[4]

The algorithms used in AI can be differentiated into white-box and black-box machine learning (ML) algorithms. White-box models are ML models that provide results that are understandable for experts in the domain. Black-box models, on the other hand, are extremely hard to explain and can hardly be understood even by domain experts.[5] XAI algorithms are considered to follow the three principles transparency, interpretability and explainability. Transparency is given “if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer”.[6] Interpretability describes the possibility to comprehend the ML model and to present the underlying basis for decision-making in a way that is understandable to humans.[7] Explainability is a concept that is recognized as important, but a joint definition is not yet available.[6] It is suggested that explainability in ML can be considered as “the collection of features of the interpretable domain, that have contributed for a given example to produce a decision (e.g., classification or regression)”.[8] If algorithms meet these requirements, they provide a basis for justifying decisions, tracking and thereby verifying them, improving the algorithms, and exploring new facts.[9]

Sometimes it is also possible to achieve a result with high accuracy with a white-box ML algorithm that is interpretable in itself.[10] This is especially important in domains like medicine, defense, finance and law where it is crucial to understand the decisions and build up trust in the algorithms.[3]

AI systems optimize behavior to satisfy a mathematically-specified goal system chosen by the system designers, such as the command "maximize accuracy of assessing how positive film reviews are in the test dataset". The AI may learn useful general rules from the test set, such as "reviews containing the word 'horrible' are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing 'Daniel Day-Lewis' are usually positive"; such rules may be undesirable if they are deemed likely to fail to generalize outside the test set, or if people consider the rule to be "cheating" or "unfair". A human can audit rules in an XAI to get an idea how likely the system is to generalize to future real-world data outside the test-set.[11]

Goals

Cooperation between agents, in this case algorithms and humans, depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formalization of trust criteria is a barrier to straightforward optimization approaches. For that reason, interpretability and explainability are posited as intermediate goals for checking other criteria.[12]

AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data, but that do not reflect the complicated implicit desires of the human system designers. For example, a 2017 system tasked with image recognition learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures, rather than learning how to tell if a horse was actually pictured.[1] In another 2017 system, a supervised learning AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.[13][14]

One transparency project, the DARPA XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users should be able to understand the AI's cognition (both in real-time and after the fact), and should be able to determine when to trust the AI and when the AI should be distrusted.[15][16] Other applications of XAI are knowledge extraction from black-box models and model comparisons.[17] The term "glass box" has also been used to systems that monitor the inputs and outputs of a system, with the purpose of verifying the system's adherence to ethical and socio-legal values and, therefore, producing value-based explanations.[18] Furthermore, the same term has been used to name a voice assistant that produces counterfactual statements as explanations.[19]

History and methods

During the 1970s to 1990s, symbolic reasoning systems, such as MYCIN,[20] GUIDON,[21] SOPHIE,[22] and PROTOS[23][24] were explored that could represent, reason about, and explain their reasoning for diagnostic, instructional, or machine-learning (explanation-based learning) purposes. MYCIN, developed in the early 1970s as a research prototype for diagnosing bacteremia infections of the bloodstream, could explain [25] which of its hand-coded rules contributed to a diagnosis in a specific case. Research in intelligent tutoring systems developed systems such as SOPHIE that could act as an 'articulate expert', explaining problem-solving strategy at a level the student could understand, so they would know what action to take next. For instance, SOPHIE could explain the qualitative reasoning behind its electronics troubleshooting, even though it ultimately relied on the SPICE circuit simulator. Similarly, GUIDON added tutorial rules to supplement MYCIN's domain-level rules so it could explain strategy for medical diagnosis. Symbolic approaches to machine learning, especially those relying on explanation-based learning, such as PROTOS, explicitly relied on representations of explanations, both to explain their actions and to acquire new knowledge.

In the 1980s through early 1990s, truth maintenance systems (TMS) were developed to extend the capabilities of causal-reasoning, rule-based, and logic-based inference systems.[26]:360–362 A TMS acts to explicitly track alternate lines of reasoning, justifications for conclusions, and lines of reasoning that lead to contradictions, allowing future reasoning to avoid these dead ends. To provide explanation, they trace reasoning from conclusions to assumptions through rule operations or logical inferences, allowing explanations to be generated from the reasoning traces. As an example, consider a rule-based problem solver with just a few rules about Socrates that concludes he has died from poison:

By just tracing through the dependency structure the problem solver can construct the following explanation: "Socrates died because he was mortal and drank poison, and all mortals die when they drink poison. Socrates was mortal because he was a man and all men are mortal. Socrates drank poison because he held dissident beliefs, the government was conservative, and those holding conservative dissident beliefs under conservative governments must drink poison."[27]:164–165

By the 1990s researchers also began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks.[28] Researchers in clinical expert systems creating neural network-powered decision support for clinicians have sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice.[2] In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence.[1] As a result, many academics and organizations are developing tools to help detect bias in their systems.[29]

Marvin Minsky et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.[30]

Modern complex AI techniques, such as deep learning and genetic algorithms are naturally opaque.[31] To address this issue, there has been a development of many new methods to make new models more explainable and interpretable.[32][33][34][35][36] This includes many methods, such as Layerwise relevance propagation (LRP), a technique for determining which features in a particular input vector contribute most strongly to a neural network's output.[37][38][39] Other techniques have been developed to explain one particular prediction made by a (nonlinear) black-box model, a goal referred to as "local interpretability".[40][41][42][43][44][45] It is worth noting that the mere transposition of the concepts of local interpretability into a remote context (where the black-box model is executed at a third party) is currently under scrutiny.[46][47]

In addition, there has been work on decision trees, tree ensembles and Bayesian networks, which are more transparent to inspection.[48][49] In 2018 an interdisciplinary conference called FAT* (Fairness, Accountability, and Transparency) was established to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.[50][51][52]

Some techniques allow visualisations of the inputs which individual neurons respond to most strongly. Several groups have found that neurons can be aggregated into circuits which perform human-comprehensible functions, some of which reliably arise across different networks trained independently.[53][54]

At a higher level, various techniques exist to extract compressed representations of the features of given inputs, which can then be analysed by standard clustering techniques. Alternatively, networks can be trained to output linguistic explanations of their behaviour, which are then directly human-interpretable.[55] Model behaviour can also be explained with reference to training data - for example, by evaluating which training inputs influenced a given behaviour the most.[56]

Regulation

As regulators, official bodies and general users come to depend on AI-based dynamic systems, clearer accountability will be required for decision making processes to ensure trust and transparency. Evidence of this requirement gaining more momentum can be seen with the launch of the first global conference exclusively dedicated to this emerging discipline, the International Joint Conference on Artificial Intelligence: Workshop on Explainable Artificial Intelligence (XAI).[57]

The European Union introduced a right to explanation in General Data Protection Right (GDPR) as an attempt to deal with the potential problems stemming from the rising importance of algorithms. The implementation of the regulation began in 2018. However, the right to explanation in GDPR covers only the local aspect of interpretability. In the United States, insurance companies are required to be able to explain their rate and coverage decisions.[58]

Sectors

XAI has been researched in many sectors, including:

References
  1. Sample, Ian (5 November 2017). "Computer says no: why making AIs fair, accountable and transparent is crucial". the Guardian. Retrieved 30 January 2018.
  2. Edwards, Lilian; Veale, Michael (2017). "Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For". Duke Law and Technology Review. 16: 18. SSRN 2972855.
  3. Gunning, D.; Stefik, M.; Choi, J.; Miller, T.; Stumpf, S.; Yang, G.-Z. (2019-12-18). "XAI-Explainable artificial intelligence". Science Robotics. 4 (37): eaay7120. doi:10.1126/scirobotics.aay7120. ISSN 2470-9476.
  4. Rieg, Thilo; Frick, Janek; Baumgartl, Hermann; Buettner, Ricardo (2020-12-17). "Demonstration of the potential of white-box machine learning approaches to gain insights from cardiovascular disease electrocardiograms". PLOS ONE. 15 (12): e0243615. doi:10.1371/journal.pone.0243615. ISSN 1932-6203. PMC 7746264. PMID 33332440.
  5. Loyola-González, O. (2019). "Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View". IEEE Access. 7: 154096–154113. doi:10.1109/ACCESS.2019.2949286. ISSN 2169-3536.
  6. Roscher, R.; Bohn, B.; Duarte, M. F.; Garcke, J. (2020). "Explainable Machine Learning for Scientific Insights and Discoveries". IEEE Access. 8: 42200–42216. doi:10.1109/ACCESS.2020.2976199. ISSN 2169-3536.
  7. "Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI". DeepAI. 2019-10-22. Retrieved 2021-01-13.
  8. "Methods for interpreting and understanding deep neural networks". Digital Signal Processing. 73: 1–15. 2018-02-01. doi:10.1016/j.dsp.2017.10.011. ISSN 1051-2004.
  9. Adadi, A.; Berrada, M. (2018). "Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)". IEEE Access. 6: 52138–52160. doi:10.1109/ACCESS.2018.2870052. ISSN 2169-3536.
  10. Rudin, Cynthia (2019). "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead". Nature Machine Intelligence. 1 (5): 206–215. doi:10.1038/s42256-019-0048-x. ISSN 2522-5839.
  11. "How AI detectives are cracking open the black box of deep learning". Science. 5 July 2017. Retrieved 30 January 2018..
  12. Dosilovic, Filip; Brcic, Mario; Hlupic, Nikica (2018-05-25). "Explainable Artificial Intelligence: A Survey" (PDF). MIPRO 2018 - 41st International Convention Proceedings. MIPRO 2018. Opatija, Croatia. pp. 210–215. doi:10.23919/MIPRO.2018.8400040.
  13. "DeepMind Has Simple Tests That Might Prevent Elon Musk's AI Apocalypse". Bloomberg.com. 11 December 2017. Retrieved 30 January 2018.
  14. "Learning from Human Preferences". OpenAI Blog. 13 June 2017. Retrieved 30 January 2018.
  15. "Explainable Artificial Intelligence (XAI)". DARPA. DARPA. Retrieved 17 July 2017.
  16. Holzinger, Andreas; Plass, Markus; Holzinger, Katharina; Crisan, Gloria Cerasela; Pintea, Camelia-M.; Palade, Vasile (2017-08-03). "A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop". arXiv:1708.01104 [cs.AI].
  17. Biecek, Przemyslaw (23 June 2018). "DALEX: explainers for complex predictive models". Journal of Machine Learning Research. 19: 1–5. arXiv:1806.08915. Bibcode:2018arXiv180608915B.
  18. Aler Tubella, Andrea; Theodorou, Andreas; Dignum, Frank; Dignum, Virginia (2019). Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour. California: International Joint Conferences on Artificial Intelligence Organization. doi:10.24963/ijcai.2019/802. ISBN 978-0-9992411-4-1.
  19. Sokol, Kacper; Flach, Peter (2018). "Glass-Box: Explaining AI Decisions With Counterfactual Statements Through Conversation With a Voice-enabled Virtual Assistant". Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. pp. 5868–5870. doi:10.24963/ijcai.2018/865. ISBN 9780999241127.
  20. Fagan, L. M.; Shortliffe, E. H.; Buchanan, B. G. (1980). "Computer-based medical decision making: from MYCIN to VM". Automedica. 3 (2): 97–108.
  21. Clancey, William (1987). Knowledge-Based Tutoring: The GUIDON Program. Cambridge, Massachusetts: The MIT Press.
  22. Brown, John S.; Burton, R. R.; De Kleer, Johan (1982). "Pedagogical, natural language, and knowledge engineering techniques in SOPHIE I, II, and III". Intelligent Tutoring Systems. Academic Press. ISBN 0-12-648680-8.
  23. Bareiss, Ray; Porter, Bruce; Weir, Craig; Holte, Robert (1990). "Protos: An Exemplar-Based Learning Apprentice". Machine Learning. 3. Morgan Kaufmann Publishers Inc. pp. 112–139. ISBN 1-55860-119-8.
  24. Bareiss, Ray. Exemplar-Based Knowledge Acquisition: A Unified Approach to Concept Representation, Classification, and Learning. Perspectives in Artificial Intelligence.
  25. Van Lent, M.; Fisher, W.; Mancuso, M. (July 2004). "An explainable artificial intelligence system for small-unit tactical behavior". Proceedings of the National Conference on Artificial Intelligence. San Jose, CA: AAAI Press. pp. 900–907. ISBN 0262511835.
  26. Russell, Stuart; Norvig, Peter (2003). Artificial Intelligence: A Modern Approach. Prentice Hall Series in Artificial Intelligence (Second ed.). Upper Saddle River, New Jersey: Prentice Hall, Pearson Education. ISBN 0-13-790395-2.
  27. Forbus, Kenneth; De Kleer, Johan (1993). Building Problem Solvers. Cambridge, Massachusetts: The MIT Press. ISBN 0-262-06157-0.
  28. Tickle, A. B.; Andrews, R.; Golea, M.; Diederich, J. (November 1998). "The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks". IEEE Transactions on Neural Networks. 9 (6): 1057–1068. doi:10.1109/72.728352. ISSN 1045-9227. PMID 18255792.
  29. "Accenture Unveils Tool to Help Companies Insure Their AI Is Fair". Bloomberg.com. June 2018. Retrieved 5 August 2018.
  30. Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.
  31. Mukherjee, Siddhartha (27 March 2017). "A.I. Versus M.D." The New Yorker. Retrieved 30 January 2018.
  32. "Interpretable neural networks based on continuous-valued logic and multicriteria decision operators". Knowledge-Based Systems. 199: 105972. 2020-07-08. doi:10.1016/j.knosys.2020.105972. ISSN 0950-7051.
  33. Lipton, Zachary C. (2016-06-10). "The Mythos of Model Interpretability". arXiv:1606.03490 [cs.LG].
  34. Murdoch, W. James; Singh, Chandan; Kumbier, Karl; Abbasi-Asl, Reza; Yu, Bin (2019-01-14). "Interpretable machine learning: definitions, methods, and applications". Proceedings of the National Academy of Sciences of the United States of America. 116 (44): 22071–22080. arXiv:1901.04592. Bibcode:2019arXiv190104592M. doi:10.1073/pnas.1900654116. PMC 6825274. PMID 31619572.
  35. Doshi-Velez, Finale; Kim, Been (2017-02-27). "Towards A Rigorous Science of Interpretable Machine Learning". arXiv:1702.08608 [stat.ML].
  36. Abdollahi, Behnoush, and Olfa Nasraoui. (2016). "Explainable Restricted Boltzmann Machines for Collaborative Filtering". arXiv:1606.07129 [stat.ML].CS1 maint: multiple names: authors list (link)
  37. Shiebler, Dan (2017-04-16). "Understanding Neural Networks with Layerwise Relevance Propagation and Deep Taylor Series". Dan Shiebler. Retrieved 2017-11-03.
  38. Bach, Sebastian; Binder, Alexander; Montavon, Grégoire; Klauschen, Frederick; Müller, Klaus-Robert; Samek, Wojciech (2015-07-10). Suarez, Oscar Deniz (ed.). "On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation". PLOS ONE. 10 (7): e0130140. Bibcode:2015PLoSO..1030140B. doi:10.1371/journal.pone.0130140. ISSN 1932-6203. PMC 4498753. PMID 26161953.
  39. Sample, Ian (5 November 2017). "Computer says no: why making AIs fair, accountable and transparent is crucial". the Guardian. Retrieved 5 August 2018.
  40. Martens, David; Provost, Foster (2014). "Explaining data-driven document classifications" (PDF). MIS Quarterly. 38: 73–99. doi:10.25300/MISQ/2014/38.1.04.
  41. ""Why Should I Trust You?" | Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining". doi:10.1145/2939672.2939778. S2CID 13029170. Cite journal requires |journal= (help)
  42. Lundberg, Scott M; Lee, Su-In (2017), Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H. (eds.), "A Unified Approach to Interpreting Model Predictions" (PDF), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., pp. 4765–4774, arXiv:1705.07874, Bibcode:2017arXiv170507874L, retrieved 2020-03-13
  43. Carter, Brandon; Mueller, Jonas; Jain, Siddhartha; Gifford, David (2019-04-11). "What made you do this? Understanding black-box decisions with sufficient input subsets". The 22nd International Conference on Artificial Intelligence and Statistics: 567–576.
  44. Shrikumar, Avanti; Greenside, Peyton; Kundaje, Anshul (2017-07-17). "Learning Important Features Through Propagating Activation Differences". International Conference on Machine Learning: 3145–3153.
  45. "Axiomatic attribution for deep networks | Proceedings of the 34th International Conference on Machine Learning - Volume 70". dl.acm.org. Retrieved 2020-03-13.
  46. Aivodji, Ulrich; Arai, Hiromi; Fortineau, Olivier; Gambs, Sébastien; Hara, Satoshi; Tapp, Alain (2019-05-24). "Fairwashing: the risk of rationalization". International Conference on Machine Learning. PMLR: 161–170.
  47. Le Merrer, Erwan; Trédan, Gilles (September 2020). "Remote explainability faces the bouncer problem". Nature Machine Intelligence. 2 (9): 529–539. arXiv:1910.01432. doi:10.1038/s42256-020-0216-z. ISSN 2522-5839.
  48. Bostrom, N., & Yudkowsky, E. (2014). The ethics of artificial intelligence. The Cambridge Handbook of Artificial Intelligence, 316-334.
  49. Vidal, Thibaut; Schiffer, Maximilian (2020). "Born-Again Tree Ensembles". International Conference on Machine Learning. PMLR. 119: 9743–9753.
  50. says, Charlotte Lancaster (2018-10-04). "What is Artificial Intelligence | Artificial Intelligence Explained". Edureka. Retrieved 2020-09-14.
  51. "FAT* Conference".
  52. "Computer programs recognise white men better than black women". The Economist. 2018. Retrieved 5 August 2018.
  53. Olah, Chris; Cammarata, Nick; Schubert, Ludwig; Goh, Gabriel; Petrov, Michael; Carter, Shan (10 March 2020). "Zoom In: An Introduction to Circuits". Distill. 5 (3): e00024.001. doi:10.23915/distill.00024.001. ISSN 2476-0757.
  54. Li, Yixuan; Yosinski, Jason; Clune, Jeff; Lipson, Hod; Hopcroft, John (8 December 2015). "Convergent Learning: Do different neural networks learn the same representations?". Feature Extraction: Modern Questions and Challenges. PMLR: 196–212.
  55. Hendricks, Lisa Anne; Akata, Zeynep; Rohrbach, Marcus; Donahue, Jeff; Schiele, Bernt; Darrell, Trevor (2016). "Generating Visual Explanations". Computer Vision – ECCV 2016. Springer International Publishing: 3–19. doi:10.1007/978-3-319-46493-0_1.
  56. Koh, Pang Wei; Liang, Percy (17 July 2017). "Understanding Black-box Predictions via Influence Functions". International Conference on Machine Learning. PMLR: 1885–1894.
  57. "IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)" (PDF). Earthlink. IJCAI. Retrieved 17 July 2017.
  58. Kahn, Jeremy (12 December 2018). "Artificial Intelligence Has Some Explaining to Do". Bloomberg Businessweek. Retrieved 17 December 2018.
  59. "NASA 'Evolutionary' software automatically designs antenna". NASA. NASA. Retrieved 17 July 2017.
  60. "The Flash Crash: The Impact of High Frequency Trading on an Electronic Market" (PDF). CFTC. CFTC. Retrieved 17 July 2017.
  61. Weng, Stephen F; Reps, Jenna; Kai, Joe; Garibaldi, Jonathan M; Qureshi, Nadeem (2017). "Can machine-learning improve cardiovascular risk prediction using routine clinical data?". PLOS ONE. 12 (4): e0174944. Bibcode:2017PLoSO..1274944W. doi:10.1371/journal.pone.0174944. PMC 5380334. PMID 28376093.
  62. Holzinger, Andreas; Biemann, Chris; Pattichis, Constantinos S.; Kell, Douglas B. (2017-12-28). "What do we need to build explainable AI systems for the medical domain?". arXiv:1712.09923 [cs.AI].
  63. "Tesla says it has 'no way of knowing' if autopilot was used in fatal Chinese crash". Guardian. 2016-09-14. Retrieved 17 July 2017.
  64. Abrams, Rachel; Kurtz, Annalyn (July 2016). "Joshua Brown, Who Died in Self-Driving Accident, Tested Limits of His Tesla". New York Times. Retrieved 17 July 2017.
  65. Olague, Gustavo (2011). "Evolutionary-computer-assisted design of image operators that detect interest points using genetic programming☆". Image and Vision Computing. Elsevier. 29 (7): 484–498. doi:10.1016/j.imavis.2011.03.004.
  66. Qureshi, M. Atif; Greene, Derek (2018-06-04). "EVE: explainable vector based embedding technique using Wikipedia". Journal of Intelligent Information Systems. 53: 137–165. arXiv:1702.06891. doi:10.1007/s10844-018-0511-x. ISSN 0925-9902. S2CID 10656055.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.