Bridging the Gap - Language Models and Structured Knowledge in AI

Organizers

Dr. Gerrit Großmann
Cennet Oguz, M.Sc.
Dr. Simon Ostermann
Prof. Dr. Verena Wolf

For any issues regarding the seminar, please e-mail Gerrit Großmann and have [LanguageModelsSeminar2024] (including the brackets) in the subject line.

Organization

• This block seminar is open to CoLi and CS students (and related courses).
• Depending on the study regulations, you need to register in HISPOS/LSF.
• CS students must have registered in HISPOS/LSF by June 26! Otherwise we cannot issue a certificate (“Schein”).
• If you are a CS student, please use the seminar assignment system to register.
• If you are a CoLi student, please write us an email to register.
• Please register only if you are available at the time slot of the seminar.
• If you want to take the seminar but were not selected by the assignment system, please apply for the waiting list by emailing us.
• The seminar takes place on September 19 (Thursday) and 20 (Friday), in person (DFKI, D3.2, room Reuse, next to the main entrance).
• The seminar language is English.
• The seminar earns you 7 ECTS (CS students) or 3 ECTS (CoLi students).

Grading

To pass the seminar, you have to attend all sessions and:

• give a presentation (CoLi and CS studenets, worth 3 ECTS);
• write reports in which you critically examine the topics in the seminar (only CS students, worth 4 ECTS);
• participate in discussions during the seminar;

… with a passing grade.

For CS students, the final grade is calculated from the weighted average. Both CoLi and CS students have the opportunity to earn a bonus for submitting an optional practical project. A good project can improve your final grade by 0.3 points (e.g., from 1.7 to 1.3), an excellent project can boost your grade by 0.7 points (e.g., from 2.3 to 1.7).

---

Topic Overview

This is a block seminar in September 2024, which will be held in cooperation between the Department of Multilingual Technologies (MLT) and Neuro-Mechanistic Modeling (NMM).

Large language models (LLMs) have swiftly become a cornerstone in AI research, capturing the attention of the public as the most accessible gateway to artificial intelligence. Despite their groundbreaking impact, LLMs are not without their imperfections. Notably, the occurrence of hallucinations and limited reasoning capabilities, particularly in specialized domains, remain significant challenges.

This seminar begins by investigating the theoretical foundations of language representations, tracing the evolution of transformers and their progression towards the cutting-edge LLMs we see today. Building on this foundation, the seminar will then explore promising future directions. Special emphasis will be placed on the integration of LLMs with neuro-symbolic reasoning and the enrichment of these models through knowledge graphs and other forms of structured data.

Requirements

We expect no prior knowledge in language modeling. The seminar is open to CS (including related majors) and CoLi students.

---

Presentation

Identify the key ideas and concepts and give a self-consistent presentation explaining concepts to your fellow students.

The presentation should be about 20 minutes long.

Please submit your your final slides as .pdf until September 18 (23:59, Berlin time).

Here are some suggestions for a good presentation (we will use this as a basis for grading the presentations):

• Do not try to condense all the content into 20 minutes. Find a suitable level of abstraction and determine what you think is interesting for your fellow students.
• Explore and include supplementary material where it seems useful (literature, youtube, GitHub, medium articles, OpenReview, etc.). The papers provided by us can be seen as a starting point.
• The goal is to tell a (self-consistent and entertaining) story - not to convince us that you understand the paper.
• Put time and effort into creating visualizations and preparing (running) examples (Please do not rely solely on paper screenshots).
• Prioritize concreteness, simplicity, and clarity.
• Focus on intuition, high-level understanding, and contextualization, not on technical details.
• Don’t overcrowd your slides. Try to avoid full sentences and be cautious with bullet points.
• Use equations only when necessary; use color-coded equations to improve their readability (example).
• Be critical of the authors’ claims, don’t fall for overselling.
• Use slide numbers.

Reports

Each CS student is required to write six reports on six different topics of their choosing. Each report should be about two to four pages long. The report should contain a short(!) summary of (what you consider to be) the main contribution or most intriguing idea of the papers. Otherwise, you can freely express your own thoughts on the topic. For instance: What did you like/dislike about the papers (both methodically and didactically)? What are connections to other seminar papers? Can you suggest improvements? What do you think is missing?

You can use the Neurips or Springer LNCS Latex template or any other reasonable format (don’t write an abstract). Please use a spell+grammar checker like languagetool or grammarly before submitting. Do not let ChatGPT do the writing for you (it is horrible to read). But you can use LLMs to catch errors or improve formulations.

The time required to write the reports should approximately add up to 4 CP, which translates to 20 hours of work per report (4 CP x 30 hours per CP).

Please email the report to Gerrit and Cennet. The report deadline is September 21 (23:59, Berlin time).

Practical Project (Bonus)

You can submit a practical project where you apply what you learned in the seminar. The project doesn’t have to be closely related to your presentation topic but should have a loose connection to any seminar topic. If you plan to do this, please discuss it with your supervisor before September 19 and propose your idea.

Practical projects previously submitted in other university courses or already published are not allowed. Plagiarism will result in expulsion. Ensure that the code is reproducible and easy to run, for example, on Colab. Your project also needs to be well documented (e.g., by providing a tutorial notebook), and you need to disclose all the resources you used.

Deadline is September 30 (23:59, Berlin time). We will make all projects publically available.

---

Topics

Please vote for your preferred topic: here. Deadline is May 25.

Topic 1

• Name: Pre-Attention Language Models
• Student: TBA
• Papers:
  1. Sequence to Sequence Learning with Neural Networks
  2. A Neural Probabilistic Language Model
  3. Character-Aware Neural Language Models
  4. Deep contextualized word representations

Topic 2

• Name: Self-Attention and Transformer
• Student: TBA
• Papers:
  1. Attention in Natural Language Processing
  2. Attention Is All You Need
  3. Theoretical Limitations of Self-Attention in Neural Sequence Models

Topic 3

• Name: Bert and its Variants
• Student: TBA
• Paper:
  1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  2. RoBERTa: A Robustly Optimized BERT Pretraining Approach
  3. Unsupervised Cross-lingual Representation Learning at Scale

Topic 4

• Name: Foundations of LLMs
• Student: TBA
• Paper:
  1. Training language models to follow instructions with human feedback
  2. GPT-4 Technical Report
  3. Galactica: A Large Language Model for Science

Topic 5

• Name: Properties of LLMs
• Student: TBA
• Paper:
  1. Scaling Laws for Neural Language Models
  2. Training Compute-Optimal Large Language Models
  3. Language Models are Few-Shot Learners

Topic 6

• Name: Multimodal LLMs
• Student: TBA
• Paper:
  1. Multimodal Neural Language Models
  2. Hierarchical Text-Conditional Image Generation with CLIP Latents
  3. Visual Instruction Tuning

Topic 7

• Name: Evaluating LLMs
• Student: TBA
• Paper:
  1. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  2. KGLens: A Parameterized Knowledge Graph Solution to Assess What an LLM Does and Doesn’t Know
  3. Whose Opinions Do Language Models Reflect?

Topic 8

• Name: Foundations of Knowledge Graph Reasoning
• Student: TBA
• Paper:
  1. Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction
  2. Towards Foundation Models for Knowledge Graph Reasoning

Topic 9

• Name: Graphs as Input
• Student: TBA
• Paper:
  1. SKILL: Structured Knowledge Infusion for Large Language Models
  2. G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
  3. Mitigating large language model hallucinations via autonomous knowledge graph-based retrofitting

Topic 10

• Name: Graphs as Output
• Student: TBA
• Paper:
  1. Exploring Large Language Models for Knowledge Graph Completion
  2. Iterative Zero-Shot LLM Prompting for Knowledge Graph Construction
  3. Causal Graph Discovery with Retrieval-Augmented Generation based Large Language Models

Topic 11

• Name: LLM-Based Reasoning on Graphs
• Student: TBA
• Paper:
  1. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
  2. Graph Language Models
  3. Exploring the Limitations of Graph Reasoning in Large Language Models

Topic 12

• Name: Learning to Prompt
• Student: TBA
• Paper:
  1. TEMPERA: Test-Time Prompting via Reinforcement Learning
  2. ReAct: Synergizing Reasoning and Acting in Language Models
  3. RoT: Enhancing Large Language Models with Reflection on Search Trees

---

Recommended Reading/Watch List

LLM Primer

• Andrej Karpathy - Intro to Large Language Models This is a classic.
• promptingguide: Nice and high-level overview on LLMs.
• But what is a GPT? Visual intro to transformers - Chapter 5, Deep Learning See also the next chapters.
• Let’s build GPT: from scratch, in code, spelled out Karpathy again.
• Graph Machine Learning in the Era of Large Language Models (LLMs) A neat review, discussing many topics relevant to this seminar.

Project Inspirations and Resources

• HuggingFace Transformers Tutorials
• ⚡ LitGPT
• From OpenAI to Open Source in 5 Minutes Tutorial (LM Studio + Python) or Streaming Local LLM Responses with LM Studio Inference Server
• Interesting LLM project ideas
• 14 examples of how LLMs can transform materials science and chemistry
• instagraph
• Scrapegraph-ai