Generative Models | Neuro-Mechanistic Modeling

Advanced Topics in Diffusion Modeling — From Theory to Implementation

Fri, 27 Oct 2023 00:00:00 +0000

Organizers

Dr. Gerrit Großmann, Prof. Dr. Verena Wolf.

For any issues regarding the seminar, please e-mail Gerrit Großmann and have [DeepDiffusion2023] in the subject line.

In case you are interested in writing a thesis on this topic, please also contact Gerrit Großmann.

Organization

Please use the seminar assignment system to register. Please register only if you are available at the time slot.
The seminar takes place Fridays from 14:15 to 16:00 in room 1.06 (E1.1) (in-person).
The kick-off meeting takes place on Friday, October 27.
The seminar language is English.
The seminar earns you 7 ECTS.
The seminar is eligible for bachelor/master/graduate students of computer science and related courses.

Schedule

Oct 27: Kick-off
Oct 30: Milestone I — Send three topic preferences to Gerrit
Nov 3: No seminar
Nov 10: No seminar
Nov 17: Q&A session topics 1–6
Nov 24: Q&A session topics 7–12
Dec 1: No seminar
Dec 8: Milestone II — Submit slides + Q&A session topics 1–6
Dec 15: Presentation slot 1 (topics 1, 2, 4)
Dec 22: Q&A session topics 7–12 (can be moved)
Jan 5: Presentation slot 2 (topics 5, 6, 7)
Jan 12: Presentation slot 3 (topics 8, 9)
Jan 19: No seminar
Jan 26: Q&A session topics 1–6
Feb 2: Q&A session topics 7–12
Feb 9: Milestone III — Submit & present tutorial notebook
Feb 23: Milestone IV — Submit reviews
Mar 23: Milestone V — Submit final tutorial notebook

Topic Overview

Diffusion models have recently transformed the field of generative deep learning. This seminar will explore this vibrant area of research.

The seminar is targeted at students who already have a background in deep learning (theoretical and practical) and are keen to dive deeper into probabilistic diffusion and related concepts such as stochastic processes and normalizing flows. Experience with diffusion models will be beneficial, but not required. While we do not aim to provide comprehensive coverage of the topic, we have selected certain papers that we find exceptionally engaging, fun, and thought-provoking.

The seminar is equally divided into two segments: a classical part, where students present a concept based on a research paper, and a practical part, where students develop a tutorial notebook inspired by the ideas from that paper.

Requirements

General background knowledge and practical experience in deep learning are strongly recommended. Experience with diffusion models will be helpful but is not necessary.

Grade

To pass the seminar, you have to attend all sessions and:

give a presentation;
write a tutorial notebook;
review three tutorials;
update your tutorial notebook based on the reviews you received;
participate in discussions;

… with a passing grade. The final grade is based on your presentation (40%), tutorial notebook (50%), and reviews (10%). You will fail the whole seminar if you receive a failing grade in any of the three parts. If we are undecided, we will also consider the discussion participation.

Presentation (40%)

Identify the key ideas and concepts and give a self-consistent presentation explaining these concepts to your fellow students.

The presentation should be 15 to 20 minutes long.

Here are some suggestions for a good presentation (we will use this as a basis for grading the presentations):

The goal is to tell a (self-consistent and entertaining) story — not to convince us that you understand the paper.
Put time and effort into creating visualizations and preparing (running) examples.
Prioritize concreteness, simplicity, and clarity.
Focus on intuition, high-level understanding, and contextualization, not on technical details.
Don’t overcrowd your slides. Try to avoid full sentences and be cautious with bullet points.
Explore and include supplementary material where it seems useful (literature, YouTube, GitHub, Medium articles, OpenReview, etc.).
Use equations only when necessary; use color-coded equations to improve their readability (example).
Be critical of the authors’ claims, don’t fall for overselling.
Use slide numbers.
Submit your slides as .pdf.

Tutorial Notebook (50%)

Your task is to create a self-consistent tutorial within a Jupyter Notebook that elucidates the key concepts of the selected paper through code examples. The scope of this project may vary depending on the complexity of the paper. A complete re-implementation may be achievable or it may be more appropriate to concentrate on a small toy problem focusing on a single concept. The Notebook should be well-structured, containing text and figures that clarify the concepts, as well as well-documented code. For guidance and inspiration, you may refer to projects like the Annotated Diffusion Model and the TeachOpenCADD platform, for instance, E(3)-invariant Graph Neural Networks. In addition, you can check out the Stanford CS224W Graph ML Tutorials.

Formalities:

The tutorial notebook should run on Google Colab and utilize PyTorch.
We will invite each student to explain their code in an individual meeting. You should be able to explain the code in detail and justify design choices.
Submit by uploading to MS Teams with naming scheme Topic03_Grossmann_GeometricLatentDiffusionModelsFor3DMoleculeGeneration.ipynb.
You can download external data (weights, training data) from within your notebook.
Plagiarism will get you expelled from the seminar and potentially exmatriculated. However, you can discuss solutions with your fellow students and take inspiration from open-source implementations.
You can also copy&paste small code snippets if they are clearly indicated.
Comment your code extensively and assign proper names to your functions, variables, and classes.
Ensure to set seeds to make your code deterministic, and remember to save the final weights for reproducibility.
Include the URL of the original paper, your name, and any external resources you use.
The notebooks will be made publicly available.
The presentation of your notebook should take about 5 minutes.
The deadline for submissions is 23:59 local time.

Reviews (10%)

We will assign three tutorial notebooks to each student to review. The primary objective of this review is to offer suggestions for enhancements and to identify any potential mistakes or ambiguities, whether they be technical, conceptual, or grammatical in nature. Each review should be approximately one to two pages in length and must be sent to the author and Gerrit via email.

Topics

Topic	Student	Paper
1	Salaheldin Y. A. Mohamed	Structured Denoising Diffusion Models in Discrete State-Spaces
2	–	Efficient and Degree-Guided Graph Generation via Discrete Diffusion Modeling
3	–	Geometric Latent Diffusion Models for 3D Molecule Generation
4	Bartłomiej Pogodziński	Consistency Models
5	Akansh Maurya	TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets
6	Davronbek Islamov	Training Diffusion Models with Reinforcement Learning
7	Yasin Esfandiari	Graphically Structured Diffusion Models
8	Soumava Paul	Generative Modelling with Inverse Heat Dissipation
9	Monseej Purkayastha	Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
10	–	Equivariant flow matching
11	–	Generative Modeling with Optimal Transport Maps
12	–	Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport

Deep Generative Diffusion Models

Mon, 09 Jan 2023 00:00:00 +0000

For any issues regarding the seminar, please e-mail Gerrit Großmann and have [DeepDiffusion] in the subject line.

Organizers

Prof. Dr. Verena Wolf, Gerrit Großmann, Dr. Michael Backenköhler.

Organization

The kick-off meeting takes place on Monday, January 9, 14:15 in room 328 (E 1.3). Send three topic preferences (unordered) by Friday 23:59 to Gerrit.
This seminar takes place on March 6 and 7, 2023 in room 1.06 (E1.1) (we start at 9:30!).
The seminar will take place in-person.
The seminar language is English.
The seminar earns you 7 ECTS.
The seminar is eligible for bachelor/master/graduate students of computer science and related courses.

Topic Overview

Diffusion models are a powerful machine learning tool and a thriving area of research. They enable the generation of new samples with extraordinary and unprecedented quality. This seminar explores the theoretical foundations rooted in the theory of stochastic processes and explores their potential in various domains.

Requirements

General background knowledge in deep learning is recommended.

Grade

To pass the seminar, you have to:

attend both sessions;
give a presentation (with a passing grade);
give an ungraded mock presentation;
participate in discussions;
submit reports (with a passing grade);
read the papers and watch the videos on the Compulsory Reading List;
pass the practical project.

The final grade consists of: your presentation (50%), reports (35%), and participation during the seminar discussions (15%). The practical project can be failed or passed. Extraordinary submissions will earn you a bonus on your final grade.

Presentation (50%)

Identify the key ideas and concepts and give a self-consistent presentation explaining these concepts to your fellow students.

The presentation should be about 20 minutes long (no longer than 25 minutes!).

Here are some suggestions for a good presentation (we will use this as a basis for grading the presentations):

The goal is to tell a (self-consistent and entertaining) story — not to convince us that you understand the paper.
Put time and effort into creating visualizations and preparing (running) examples.
Prioritize concreteness, simplicity, and clarity.
Focus on intuition, high-level understanding, and contextualization, not on technical details.
Don’t overcrowd your slides. Try to avoid full sentences and be cautious with bullet points.
Explore and include supplementary material where it seems useful (literature, YouTube, GitHub, Medium articles, OpenReview, etc.).
Use equations only when necessary; use color-coded equations to improve their readability (example).
Be critical of the authors’ claims, don’t fall for overselling.
Use slide numbers.

Send the first draft of your slides as a .pdf file to your supervisor 7 days before the presentation.

You will also be required to give an ungraded mock presentation in your group (see topic list) by March 5.

Reports (35%)

Every participant has to write three short reports (two to four pages each). You can choose freely from the twelve topics (you may also choose your own topic). The report should contain a short summary of (what you consider to be) the main contribution or most intriguing idea of the paper. Otherwise, you can freely express your own thoughts on the topic. For instance: What did you like/dislike about the paper (both methodically and didactically)? What are connections to other seminar papers? Can you suggest improvements? What do you think is missing?

The report deadline is 14 days after the last presentation (March 21, 23:59). You can use the NeurIPS template (activate the preprint flag) or any other reasonable format (don’t write an abstract). Please use a spell+grammar checker like LanguageTool or Grammarly before submitting.

Discussions (15%)

Read each presented paper in advance (of course, we do not expect you to understand them entirely) and actively participate in the discussion.

Practical Project (fail/pass/bonus)

You can work on the project alone or in groups of two. The deadline is 28 days after the last presentation (April 11, 23:59). Specifications are here.

Topics

Compulsory Reading List

A: Diffusion models explained in 4-difficulty levels
B: How AI Image Generators Work (Stable Diffusion / Dall-E) — Computerphile
C: What are Diffusion Models?
D: (optional) How diffusion models work: the math from scratch
E: (optional) The Annotated Diffusion Model
F: (optional) CVPR 2022 Tutorial: Denoising Diffusion-based Generative Modeling: Foundations and Applications

Monday (March 6, 2023)

Student (Group, Supervisor)	Topic	Additional Material
Maryam Meghdadi Esfahani (1, GG)	Generative Modeling by Estimating Gradients of the Data Distribution (2019)	[1,2,3]
Mohammad Sadegh Akhondzadeh (2, MB)	Permutation Invariant Graph Generation via Score-Based Generative Modeling (2020)	[1,2,7,8]
Jorge Augusto Calvimontes Robles (3, GG)	Denoising Diffusion Probabilistic Models (2020)	[4,5,6]
Devikalyan Das (1, MB)	Score-Based Generative Modeling through Stochastic Differential Equations (2021)	[1,2,7,F]
Lisa Dargasz (3, GG)	High-resolution image synthesis with latent diffusion models (2022)	[10,11,12,F]
Simone Antonelli (4, MB)	Equivariant diffusion for molecule generation in 3D (2022)	[17,26,27]

Tuesday (March 7, 2023)

Student (Group, Supervisor)	Topic	Additional Material
~~Advait Ganesh Athreya (2, GG)~~	~~DiGress: Discrete Denoising diffusion for graph generation (2022)~~	[7,13,14]
Simon Graf (2, MB)	Structure-based Drug Design with Equivariant Diffusion Models (2022)	[15,16,17,18]
Tim Valentin Kruse (4, GG)	3D Shape Generation and Completion through Point-Voxel Diffusion (2022)	[19,20]
~~Vikram Singh (4, MB)~~	~~Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem (2022)~~	[17,21,22,23]
Shreyash Arya (3, GG)	Make-A-Video: Text-to-Video Generation without Text-Video Data (2022)	[10,12,24,25]
~~Hamayoon Behmanush (1, MB)~~	~~Continuous diffusion for categorical data (2022)~~	[2,27,28]

The group is only relevant for the mock presentation. Your supervisors are either Gerrit Großmann (GG) or Michael Backenköhler (MB).

Supplementary Material

A non-exhaustive list of supplemental materials that we think might be helpful to you is provided here. We encourage you to seek and consult additional resources.

The header image was created using DALL-E with the prompt a robot painting a picture, abstract oil painting.

Generative Models | Neuro-Mechanistic Modeling

Advanced Topics in Diffusion Modeling — From Theory to Implementation

Organizers

Organization

Schedule

Topic Overview

Requirements

Grade

Presentation (40%)

Tutorial Notebook (50%)

Reviews (10%)

Topics

Recommended Reading List

Deep Generative Diffusion Models

Organizers

Organization

Topic Overview

Requirements

Grade

Presentation (50%)

Reports (35%)

Discussions (15%)

Practical Project (fail/pass/bonus)

Topics

Compulsory Reading List

Monday (March 6, 2023)

Tuesday (March 7, 2023)

Supplementary Material