Synthetic Intelligence RSS

Abstract Introduction Background Trust LLM Preliminaries  Assessments Trustworthiness Truthfulness Safety Fairness Robustness Privacy Protection Machine Ethics  Transparency AccountabilityOpen Challenges Future WorkConclusions Types of Ethical Agents    Safety Assessment  Synopsis The content focuses on assessing the safety of Large Language Models (LLMs), particularly in the context of various security threats like jailbreak attacks, exaggerated safety, toxicity, and misuse. It introduces datasets like JAILBREAKTRIGGER and XSTEST for evaluating LLMs against these threats. The text details the methodologies for evaluating LLMs’ responses to different types of prompts, with emphasis on their ability to resist harmful outputs and misuse. The content also discusses the...

Abstract Introduction Background Trust LLM Preliminaries  Assessments Trustworthiness Truthfulness Safety Fairness Robustness Privacy Protection Machine Ethics  Transparency AccountabilityOpen Challenges Future WorkConclusions  Types of Ethical Agents  Truthfulness The provided content is a comprehensive analysis of the truthfulness of Large Language Models (LLMs) with a focus on four aspects: misinformation generation, hallucination, sycophancy, and adversarial factuality. Misinformation generation It is evident that LLMs, like GPT-4, struggle with generating accurate information solely from internal knowledge, leading to misinformation. This is particularly pronounced in zero-shot question-answering tasks. However, LLMs show improvement when external knowledge sources are integrated, suggesting that retrieval-augmented models may reduce misinformation....

Abstract Introduction Background Trust LLM Preliminaries  Assessments Trustworthiness Truthfulness Safety Fairness Robustness Privacy Protection Machine Ethics  Transparency AccountabilityOpen Challenges Future WorkConclusions Types of Ethical Agents     TRUSTLLM Preliminaries  The Preliminaries of TRUSTLLM section lays the groundwork for understanding the benchmark design in evaluating various Language Large Models (LLMs). The inclusion of both proprietary and open-weight LLMs showcases an in-depth and inclusive approach. Moreover, the emphasis on experimental setup – detailing datasets, tasks, prompt templates, and evaluation methods – provides a clear and systematic approach for assessment. The ethical consideration highlighted reflects a responsible and conscientious approach to research, especially considering the...

  Abstract Introduction Background Trust LLM Preliminaries  Assessments Trustworthiness Truthfulness Safety Fairness Robustness Privacy Protection Machine Ethics  Transparency AccountabilityOpen Challenges Future WorkConclusions Types of Ethical Agents    Background Large Language Models (LLMs) A language model (LM) aims to predict the probability distribution over a sequence of tokens. Scaling the model size and data size, large language models (LLMs) have shown “emergent abilities” [87, 88, 89] in solving a series of complex tasks that cannot be dealt with by regular-sized LMs. For instance, GPT-3 can handle few-shot tasks by learning in context, in contrast to GPT-2, which struggles in this regard....

Abstract Introduction Background Trust LLM Preliminaries  Assessments Trustworthiness Truthfulness Safety Fairness Robustness Privacy Protection Machine Ethics  Transparency AccountabilityOpen Challenges Future WorkConclusions Types of Ethical Agents    Trustworthiness 1. Truthfulness Score: 85/100 Stars: ⭐⭐⭐⭐✩ The emphasis on truthfulness in LLMs is well-placed, considering the impact misinformation can have. The use of diverse datasets and benchmarks for evaluating truthfulness is a strong approach, but the reliance on large-scale internet data for training LLMs does pose significant challenges in ensuring consistent accuracy. The dual approach of internal knowledge evaluation and adaptability to evolving information is commendable. However, the persistence of misinformation in training datasets...