Bridging the Gap – Language Models and Structured Knowledge in AI

Thu, 19 Sep 2024 00:00:00 +0000

Organizers

Dr. Gerrit Großmann
Cennet Oguz, M.Sc.
Dr. Simon Ostermann
Prof. Dr. Verena Wolf

Organization

For any issues regarding the seminar, please e-mail Gerrit Großmann and have [LanguageModelsSeminar2024] (including the brackets) in the subject line.

This block seminar is open to CoLi and CS students (and related courses).
Depending on the study regulations, you need to register in HISPOS/LSF.
CS students must have registered in HISPOS/LSF by June 26! Otherwise we cannot issue a certificate (“Schein”).
If you are a CS student, please use the seminar assignment system to register.
If you are a CoLi student, please write us an email to register.
Please register only if you are available at the time slot of the seminar.
If you want to take the seminar but were not selected by the assignment system, please apply for the waiting list by emailing us.
The seminar takes place on September 19 (Thursday) and 20 (Friday), in person (10:00 sharp, DFKI, D3.2, room Reuse, next to the main entrance).
The seminar language is English.
The seminar earns you 7 ECTS.

Grading

To pass the seminar, you have to attend all sessions and:

give a presentation (CoLi and CS studenets, worth 3 ECTS);
write reports in which you critically examine the topics in the seminar (only CS students, worth 4 ECTS);
participate in discussions during the seminar;

… with a passing grade.

For CS students, the final grade is calculated from the weighted average. Both CoLi and CS students have the opportunity to earn a bonus for submitting an optional practical project. A good project can improve your final grade by 0.3 points (e.g., from 1.7 to 1.3), an excellent project can boost your grade by 0.7 points (e.g., from 2.3 to 1.7).

Topic Overview

This is a block seminar in September 2024, which will be held in cooperation between the Department of Multilingual Technologies (MLT) and Neuro-Mechanistic Modeling (NMM).

Large language models (LLMs) have swiftly become a cornerstone in AI research, capturing the attention of the public as the most accessible gateway to artificial intelligence. Despite their groundbreaking impact, LLMs are not without their imperfections. Notably, the occurrence of hallucinations and limited reasoning capabilities, particularly in specialized domains, remain significant challenges.

This seminar begins by investigating the theoretical foundations of language representations, tracing the evolution of transformers and their progression towards the cutting-edge LLMs we see today. Building on this foundation, the seminar will then explore promising future directions. Special emphasis will be placed on the integration of LLMs with neuro-symbolic reasoning and the enrichment of these models through knowledge graphs and other forms of structured data.

Requirements

We expect no prior knowledge in language modeling. The seminar is open to CS (including related majors) and CoLi students.

Course Structure

Presentation

Identify the key ideas and concepts and give a self-consistent presentation explaining concepts to your fellow students.

The presentation should be about 20 minutes long.

Please submit your your final slides as .pdf until September 18 (23:59, Berlin time) and have [LanguageModelsSeminar2024] Slides Submission in the subject line.

Here are some suggestions for a good presentation (we will use this as a basis for grading the presentations):

Do not try to condense all the content into 20 minutes. Find a suitable level of abstraction and determine what you think is interesting for your fellow students.
Explore and include supplementary material where it seems useful (literature, youtube, GitHub, medium articles, OpenReview, etc.). The papers provided by us can be seen as a starting point.
The goal is to tell a (self-consistent and entertaining) story - not to convince us that you understand the paper.
Put time and effort into creating visualizations and preparing (running) examples (Please do not rely solely on paper screenshots).
Prioritize concreteness, simplicity, and clarity.
Focus on intuition, high-level understanding, and contextualization, not on technical details.
Don’t overcrowd your slides. Try to avoid full sentences and be cautious with bullet points.
Use equations only when necessary; use color-coded equations to improve their readability (example).
Be critical of the authors’ claims, don’t fall for overselling.
Use slide numbers.

Reports

Each CS student is required to write six reports on six different topics of their choosing (this can include your own topic). Each report should be about two to four pages long. The report should contain a short(!) summary of (what you consider to be) the main contribution or most intriguing idea of the papers. Otherwise, you can freely express your own thoughts on the topic. For instance: What did you like/dislike about the papers (both methodically and didactically)? What are connections to other seminar papers? Can you suggest improvements? What do you think is missing?

You can use the Neurips or Springer LNCS Latex template or any other reasonable format (don’t write an abstract). Please use a spell+grammar checker like languagetool or grammarly before submitting. Do not let ChatGPT do the writing for you (it is horrible to read). But you can use LLMs to catch errors or improve formulations.

The time required to write the reports should approximately add up to 4 CP, which translates to 20 hours of work per report (4 CP x 30 hours per CP).

Please email the reports as a single .pdf file (with a reasonable filename) to Gerrit and Cennet and have [LanguageModelsSeminar2024] Report Submission in the subject line. The report deadline is September 28 (23:59, Berlin time).

Practical Project (Bonus)

You can submit a practical project where you apply what you learned in the seminar. The project doesn’t have to be closely related to your presentation topic but should have a loose connection to any seminar topic. If you plan to do this, please discuss it with your supervisor before September 19 and propose your idea.

Practical projects previously submitted in other university courses or already published are not allowed. Plagiarism will result in expulsion. Ensure that the code is reproducible and easy to run, for example, on Colab. Your project also needs to be well documented (e.g., by providing a tutorial notebook), and you need to disclose all the resources you used.

Deadline is October 4 (23:59, Berlin time). We will make all projects publically available. Please email the project as a single .zip file (with a reasonable filename) to Gerrit and Cennet and have [LanguageModelsSeminar2024] Project Submission in the subject line.

Topics

~~Please vote for your preferred topic: here. Deadline is May 25.~~

Topic 1

Topic 2

Name: Self-Attention and Transformer
Student: Mayur Deshmukh
Supervisor: Cennet
Papers:

Topic 3

Topic 4

Topic 5

Name: Properties of LLMs
Student: Vianney Jerry Tchouwa Takou
Supervisor: Cennet
Paper:

Topic 6

Topic 7

Topic 8

Name: Foundations of Knowledge Graph Reasoning
Student: Larisa Ivanova
Supervisor: Gerrit
Paper:
1. Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction
2. Towards Foundation Models for Knowledge Graph Reasoning

Topic 9

Topic 10

Topic 11

Topic 12

Language Models | Neuro-Mechanistic Modeling