Dr. Gerrit Großmann
Cennet Oguz, M.Sc.
Dr. Simon Ostermann
Prof. Dr. Verena Wolf
For any issues regarding the seminar, please e-mail Gerrit Großmann and have [LanguageModelsSeminar2024]
(including the brackets) in the subject line.
June 26
! Otherwise we cannot issue a certificate (“Schein”).waiting list
by emailing us.September 19 (Thursday) and 20 (Friday)
, in person (10:00 sharp, DFKI, D3.2, room Reuse, next to the main entrance).To pass the seminar, you have to attend all sessions and:
… with a passing grade.
For CS students, the final grade is calculated from the weighted average. Both CoLi and CS students have the opportunity to earn a bonus for submitting an optional practical project. A good project can improve your final grade by 0.3 points (e.g., from 1.7 to 1.3), an excellent project can boost your grade by 0.7 points (e.g., from 2.3 to 1.7).
This is a block seminar in September 2024, which will be held in cooperation between the Department of Multilingual Technologies (MLT) and Neuro-Mechanistic Modeling (NMM).
Large language models (LLMs) have swiftly become a cornerstone in AI research, capturing the attention of the public as the most accessible gateway to artificial intelligence. Despite their groundbreaking impact, LLMs are not without their imperfections. Notably, the occurrence of hallucinations and limited reasoning capabilities, particularly in specialized domains, remain significant challenges.
This seminar begins by investigating the theoretical foundations of language representations, tracing the evolution of transformers and their progression towards the cutting-edge LLMs we see today. Building on this foundation, the seminar will then explore promising future directions. Special emphasis will be placed on the integration of LLMs with neuro-symbolic reasoning and the enrichment of these models through knowledge graphs and other forms of structured data.
We expect no prior knowledge in language modeling. The seminar is open to CS (including related majors) and CoLi students.
Identify the key ideas and concepts and give a self-consistent presentation explaining concepts to your fellow students.
The presentation should be about 20 minutes long.
Please submit your your final slides as .pdf until September 18
(23:59, Berlin time) and have [LanguageModelsSeminar2024] Slides Submission
in the subject line.
Here are some suggestions for a good presentation (we will use this as a basis for grading the presentations):
Each CS student is required to write six reports on six different topics of their choosing (this can include your own topic). Each report should be about two to four pages long. The report should contain a short(!) summary of (what you consider to be) the main contribution or most intriguing idea of the papers. Otherwise, you can freely express your own thoughts on the topic. For instance: What did you like/dislike about the papers (both methodically and didactically)? What are connections to other seminar papers? Can you suggest improvements? What do you think is missing?
You can use the Neurips or Springer LNCS Latex template or any other reasonable format (don’t write an abstract). Please use a spell+grammar checker like languagetool or grammarly before submitting. Do not let ChatGPT do the writing for you (it is horrible to read). But you can use LLMs to catch errors or improve formulations.
The time required to write the reports should approximately add up to 4 CP, which translates to 20 hours of work per report (4 CP x 30 hours per CP).
Please email the reports as a single .pdf file (with a reasonable filename) to Gerrit and Cennet and have [LanguageModelsSeminar2024] Report Submission
in the subject line. The report deadline is September 28
(23:59, Berlin time).
You can submit a practical project where you apply what you learned in the seminar. The project doesn’t have to be closely related to your presentation topic but should have a loose connection to any seminar topic. If you plan to do this, please discuss it with your supervisor before September 19 and propose your idea.
Practical projects previously submitted in other university courses or already published are not allowed. Plagiarism will result in expulsion. Ensure that the code is reproducible and easy to run, for example, on Colab. Your project also needs to be well documented (e.g., by providing a tutorial notebook), and you need to disclose all the resources you used.
Deadline is October 4
(23:59, Berlin time). We will make all projects publically available. Please email the project as a single .zip file (with a reasonable filename) to Gerrit and Cennet and have [LanguageModelsSeminar2024] Project Submission
in the subject line.
Please vote for your preferred topic: here. Deadline is May 25
.
Topic 1
Topic 2
Topic 3
Topic 4
Topic 5
Topic 6
Topic 7
Topic 8
Topic 9
Topic 10
Topic 11
Topic 12
Project Inspirations