About Me | Articles | Recommended AI Books

Category: Transparency

  • Capturing Legal Reasoning Paths from Facts to Law in Court Judgments using Knowledge Graphs

    Ryoma Kondo, Riona Matsuoka, Takahiro Yoshida, Kazuyuki Yamasawa, Ryohei Hisano
    August 24, 2025
    PDF

    The paper tackles legal reasoning by building a knowledge graph from 648 Japanese administrative court decisions that makes the hidden reasoning path machine-readable. The system uses large language models to extract the key components of legal reasoning: factual findings, legal provisions cited, and how the court applied those provisions to the facts, and connects them through a purpose-built legal ontology. The result is a structured graph where you can trace the logical steps from a fact to the legal norm it triggers to the outcome it produces. In retrieval tests, the system outperformed standard LLM baselines at finding the correct legal provisions given a set of facts, meaning the knowledge graph adds genuine precision beyond what a general-purpose AI can achieve alone.

    Knowledge Graph (KG) is a database that stores information as a network of entities and relationships rather than rows and columns. In a legal context, entities might be facts, court decisions, legal provisions, parties, and the relationships between them capture how they connect (e.g., “Fact A triggers Provision B which leads to Outcome C”). Knowledge graphs make implicit relationships explicit and queryable.

    Legal Reasoning Path is the structured logical chain a court follows from factual findings to a legal conclusion: facts → applicable legal norm → application of the norm to the facts → decision. In most court opinions this path is written as prose and must be inferred by a human reader. This paper’s core contribution is extracting and storing these paths as structured data.

    Ontology is a formal specification of concepts and relationships within a domain — essentially a vocabulary with rules. A legal ontology defines what entities exist in legal reasoning (facts, norms, parties, outcomes) and how they can relate to each other. It constrains the knowledge graph so that extracted information follows a consistent structure across all cases.

    Expert Annotation is created by having human domain experts (legal professionals) manually label examples to create a “gold standard” dataset for evaluating the system’s accuracy. The annotated examples serve as the benchmark. If the system’s extracted reasoning paths match what the experts identified, the system is considered accurate.

  • The Limits of AI Data Transparency Policy: Three Disclosure Fallacies

    Judy Hanwen Shen, Ken Liu, Angelina Wang, Sarah H. Cen, Andy K. Zhang, Caroline Meinhardt, Daniel Zhang, Kevin Klyman, Rishi Bommasani, Daniel E. Ho (Stanford University)
    January 26, 2026
    arXiv | PDF

    The paper from Stanford argues that current transparency policies are largely symbolic: they suffer from three fundamental gaps that prevent them from actually achieving their stated goals of protecting privacy, copyright, and data quality. The three fallacies are: (1) a specification gap; (2) an enforcement gap; and (3) an impact gap. The paper offers a taxonomy of disclosure levels, maps each transparency objective to what’s actually needed, and proposes technical research directions and policy fixes.

    California AB 2013 is a California state law (effective 2026) requiring developers of generative AI systems to publicly post “high-level summaries” of training datasets on their websites. Covers data sources, synthetic data usage, presence of personal information, copyrighted content, and dataset statistics. It was the first U.S. law specifically mandating AI training data transparency, but the paper argues it was weakened through the legislative process from detailed requirements to vague summaries.

    EU AI Act (Regulation 2024/1689) is the European Union’s comprehensive AI regulation, which classifies AI systems by risk tier and imposes different transparency requirements for each. General-purpose AI model providers must disclose a data summary including data types and copyright status. High-risk systems (healthcare, criminal justice, employment) face stricter requirements under Article 10 for data governance practices. Unlike AB 2013, the EU AI Act assigns enforcement to the EU AI Office and imposes significant fines.

    GDPR (General Data Protection Regulation) is the EU’s data privacy law (2018) that requires organizations to inform individuals about data collection purposes (Article 13) and gives individuals rights over their personal data. Relevant here because GDPR’s data processing requirements apply to AI training data that contains personal information, but the paper notes that GDPR’s individual-level protections don’t map cleanly to the scale of LLM training.

    Membership Inference is a technical method for determining whether a specific data point was used in a model’s training set by analyzing the model’s behavior (e.g., confidence scores, loss values) on that data point. The paper identifies this as critical for copyright and privacy verification but notes it remains unreliable at scale. A model can memorize content without being able to reproduce it verbatim, and content overlap between sources makes attribution difficult.

    Data Provenance is the documented history of a piece of data, where it came from, how it was collected, what licenses apply, and how it was transformed. The paper argues that tracking provenance through the AI data supply chain (from original creators through data vendors to model developers) is essential but rarely required or practiced.

    Foundation Model Transparency Index (FMTI) is a Stanford HAI project that scores major AI model developers on 100+ transparency indicators, including 10 data-related ones. Useful for comparing company practices but doesn’t specify the intended impact of each disclosure.

    N-gram Overlap is a method for detecting text similarity by comparing sequences of N consecutive words between two texts. The paper highlights a critical limitation: courts have granted data owners permission to “inspect” training data using methods like substring search, but research shows LLMs can synthesize and reproduce content without any original n-grams, meaning n-gram-based membership tests can be trivially evaded.

  • Knowledge Graphs Construction from Criminal Court Appeals: Insights from the French Cassation Court

    Alexander V. Belikov (Growgraph, Paris), Sacha Raoult (Institut Universitaire de France, Aix-Marseille University)
    January 24, 2025
    arXiv | PDF

    The paper presents a complete, end-to-end framework for transforming unstructured French criminal court appeals into structured knowledge graphs. The authors process 2,820 appeals from the criminal chamber of the French Cassation Court (France’s Supreme Court for criminal cases) using GPT-4o mini to automatically extract entities and relationships into RDF triples. The core contribution is a domain-specific criminal law ontology developed semi-automatically through iterative interaction with LLMs (GPT-4o mini and Claude 3.5 Sonnet), which guides the extraction process and ensures consistent, structured output. The key finding is that ontology-guided RDF triple generation significantly outperforms property graph approaches — the RDF method achieved >90% accuracy (93% precision, 89% recall) compared to only 50-60% for property graph extraction. This demonstrates that providing a well-designed domain ontology in the LLM prompt is critical for reliable legal knowledge graph construction.

    A knowledge graph is a structured representation of information where nodes represent entities (people, crimes, courts, punishments) and edges represent relationships between them. Unlike flat databases or plain text, KGs capture the connections between pieces of information, making them ideal for domains like law where relationships between actors, events, and legal provisions are critical. Currently, there are two competing ways to store knowledge graphs: (a) Resource Description Framework (RDF) triples follow the format subject-predicate-object; and (b) Property Graphs (e.g., Neo4j) store nodes and edges with arbitrary key-value properties.