Neuro-Mechanistic Modeling

PGTG

Mon, 01 Jan 0001 00:00:00 +0000

Overview

PGTG (Procedurally Generated Grid-Based Traffic Gym) is a Gymnasium-compliant, feature-rich, extensible, and customizable reinforcement learning benchmark. An agent has to navigate procedurally generated tracks while managing momentum and coping with stochastic obstacles and the unpredictable behavior of other traffic participants.

Map generation, traffic rules, obstacles, and the observation and reward functions can all be configured in detail, and each feature can be toggled individually. This lets researchers study exploration, safety, and generalization separately or jointly to trade-offs between them (e.g. how safety constraints affect exploration) in one environment. Optional subgoals with intermediate rewards, for instance, decouple safety from exploration. In the accompanying empirical study, no tested state-of-the-art deep RL algorithm reaches satisfactory performance, which leaves plenty of room for future research.

Usage

PGTG requires at least Python 3.11 and is available on PyPI:

pip install pgtg

Importing the package registers the environment with Gymnasium as a side effect:

import gymnasium as gym
import pgtg  # registers "pgtg-v5"

env = gym.make("pgtg-v5")

The repository also includes runnable example notebooks (interactive playground, Stable-Baselines3 training, and statistical evaluation with PyDSMC). See the documentation for the configuration reference and a guide to adding custom obstacles.

Citation

PGTG was presented at the Reinforcement Learning Conference 2026 (RLC 2026) and is published in the accompanying Reinforcement Learning Journal. If you use it in your own work, please cite:

@article{meyerPGTG2026,
  title   = {{{PGTG}}: Procedurally Generated Grid-Based Traffic Gym},
  author  = {Meyer, Joshua and Kuntz, Felix Maurice and Wolf, Verena and
             Hoffmann, J{\"o}rg and Gros, Timo P.},
  journal = {Reinforcement Learning Journal},
  volume  = {7},
  year    = {2026}
}

NEDD

Mon, 01 Jan 0001 00:00:00 +0000

NEXD

Mon, 01 Jan 0001 00:00:00 +0000

PyDSMC

Mon, 01 Jan 0001 00:00:00 +0000

PyDSMC is an open-source Python library for statistical model checking of neural agents on arbitrary Gymnasium environments. It is designed to be lightweight and easy-to-use while being based on established statistical methods to provide guarantees on the investigated agents’ performances on various properties. Implementing the Gymnasium interface, PyDSMC is widely applicable and fully agnostic to the environments underlying implementation.

PyDSMC is based on Deep Statistical Model Checking and aims to facilitate greater adoption of statistical model checking by simplifying its usage.

Please consult the accompanying paper for more details.

📖 Full documentation (with an in-browser demo) is available at neuro-mechanistic-modeling.github.io/PyDSMC.

Deep Statistical Model Checking

The train and evaluation curves of neural agents differ greatly throughout training. Just inspecting the training curves does not suffice and leads to suboptimal policy extraction. We advocate to perform statistically-backed model evaluations periodically throughout training to ensure a good extraction point.

Setup

PyDSMC can be installed from PyPI using pip install pydsmc.

We recommend using a virtual environment and officially tested python versions 3.10, 3.11, and 3.12.

To work on this project, set up a virtual environment and install all necessary dependencies—for instance, using virtualenvwrapper and executing:

mkvirtualenv --python=python3.11 pydsmc
pip install -r requirements.txt

Usage

PyDSMC mainly exposes two functionalities: Property creation and the Evaluator.

Properties

PyDSMC can analyze arbitrary properties. These can also be environment-specific. For ease-of-use, we provide ready-to-use implementations of commonly used, domain-independent properties that are parameterized and can, thus, be adjusted to each individual usecase. You can get a list of all available predefined properties by calling pydsmc.get_predefined_properties().

Creating a predefined property is straightforward. For instance, a property analyzing the achieved return could be defined as follows:

from pydsmc import create_predefined_property

return_property = create_predefined_property(
    property_id="return",   # Which predefined property to use
    name="returnGamma0.97", # Property's name, used for storing the evaluation results
    epsilon=0.025,          # Half-width of the requested confidence interval (CI)
    kappa=0.05,             # Probability that the true mean lies within the CI
    relative_error=True,    # Whether epsilon represents the relative or absolute error
    bounds=(0, 864),        # Bounds of the property, i.e., min and max possible values
    sound=True,             # Whether a sound statistical method should be used
    gamma=0.97              # Property specific attributes
)

Creating a custom property is equally simple. The only additional information required is a checking function check_fn that analyzes trajectory information and returns a float value as a sample result for the property:

from pydsmc import create_custom_property

crash_property = create_custom_property(
    name="crash_prob",      # see above
    epsilon=0.025,          # see above
    kappa=0.05,             # see above
    relative_error=False,   # see above
    binomial=True,          # This property follows a binomial distribution
    bounds=(0, 1),          # see above
    sound=False,            # see above
    # The property's checking function, crash identified by last reward -100
    check_fn=lambda self, t: float(t[-1].reward == -100)
)

Evaluator

We can then register the properties in an Evaluator which will do the work of evaluating all the properties until convergence is reached, or the given resource budget is exhausted.

In order to construct an Evaluator, it is first necessary to create the evaluation environments from the outside. For this, we provide a helper function create_eval_envs that creates a suitable vectorized environment according to the given configuration.

from pydsmc import create_eval_envs, Evaluator

NUM_THREADS = 1  # Recommended: 1; Often better to use more parallel environments instead
NUM_PAR_ENVS = 8  # Number of parallel environments _per thread_

envs = create_eval_envs(
    num_threads=NUM_THREADS,
    num_envs_per_thread=NUM_PAR_ENVS,
    env_seed=42,  # Seeds are incremented for each environment, making them unique
    gym_id="HalfCheetah-v5",
    wrappers=[gym.wrappers.NormalizeObservation, Monitor],
    vecenv_cls=gym.vector.AsyncVectorEnv,  # gym.vector.SyncVectorEnv, sb3.DummyVecEnv, sb3.SubprocVecEnv
    # The following kwargs are passed to the gym.make function, which passes unknown args to the env
    max_episode_steps=1000,
)

evaluator = Evaluator(env=envs)

Having created the Evaluator, we can register the properties and start the evaluation:

evaluator.register_property(return_property)
evaluator.register_property(crash_property)
### Or register multiple properties at once using:
# evaluator.register_properties([return_property, crash_property])

### Start the evaluation
evaluator.eval(
    agent=agent,  # or: predict_fn=agent.predict
    stop_on_convergence=True,  # Stop when all properties have converged
    episode_limit=10_000,  # Early stop evaluation after 10,000 episodes
    time_limit=60,  # Early stop evaluation after 1 hour
    num_episodes_per_policy_run=100,
    num_threads=NUM_THREADS,
    deterministic=True,
)

Full example

A few full examples on a select set of environments can be found in example_agents to try out.

Statistical Method Selection

For each property, PyDSMC automatically selects an appropriate statistical method based on the provided parameters, following decision tree below:

Citing PyDSMC

If you use PyDSMC in your work, you can use the following BibTeX entry, citing the official Version of Record published at QEST2025.

@inproceedings{grosPyDSMC2025,
  title      = {{{PyDSMC}}: {{Statistical Model Checking}} for~{{Neural Agents Using}} the~{{Gymnasium Interface}}},
  shorttitle = {{{PyDSMC}}},
  booktitle  = {Quantitative {{Evaluation}} of {{Systems}} and {{Formal Modeling}} and {{Analysis}} of {{Timed Systems}}},
  author     = {Gros, Timo P. and Hartmanns, Arnd and Hoese, Ivo and Meyer, Joshua and M{\"u}ller, Nicola J. and Wolf, Verena},
  editor     = {Prabhakar, Pavithra and Vandin, Andrea},
  year       = {2025},
  pages      = {134--156},
  publisher  = {Springer Nature Switzerland},
  address    = {Cham},
  doi        = {10.1007/978-3-032-05792-1_8},
  isbn       = {978-3-032-05792-1},
}

License

The code introduced by this project is licensed under the MIT license. Please consult the bundled LICENSE file for the full license text.

MAC-MERLin

Mon, 01 Jan 0001 00:00:00 +0000

ToCertain

Mon, 01 Jan 0001 00:00:00 +0000

CaMoRe

Mon, 01 Jan 0001 00:00:00 +0000

MULTIMODE

Mon, 01 Jan 0001 00:00:00 +0000

Bistable exclusive-switch model — a canonical multimodal stochastic reaction network.

MoDigPro

Mon, 01 Jan 0001 00:00:00 +0000

RAIME

Mon, 01 Jan 0001 00:00:00 +0000

Complex Networks

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-time SIS contagion on a geometric network — infected nodes in red, healthy nodes in blue.

DNA Methylation

Mon, 01 Jan 0001 00:00:00 +0000

Assay-Based Machine Learning: Rethinking Evaluation in Drug Discovery

Sun, 01 Nov 2026 00:00:00 +0000

How PC-based Methods Err: Towards Better Reporting of Assumption Violations and Small Sample Errors

Sat, 01 Aug 2026 00:00:00 +0000

Chaotic and complex dynamics expose the limits of counterfactual reasoning

Mon, 01 Jun 2026 00:00:00 +0000

Evaluation beyond goodness of fit: Quantifying biophysical alignment of AI models for kinase-centric drug discovery

Mon, 01 Jun 2026 00:00:00 +0000

Per-Domain Generalizing Policies: On Learning Efficient and Robust Q-Value Functions

Mon, 01 Jun 2026 00:00:00 +0000

Causal inference on process graphs: Causal structure and effect identification

Fri, 01 May 2026 00:00:00 +0000

AI CHAOS! 2nd Workshop on the Challenges for Human Oversight of AI Systems

Wed, 01 Apr 2026 00:00:00 +0000

Formal Methods for AI Alignment

Wed, 01 Apr 2026 00:00:00 +0000

Team

Kevin Baum, Lisa Dargasz, Felix Jahn, Yannic Muskalla, Andre Steingrüber

Content

AI alignment is, broadly speaking, the task of ensuring that AI systems act in accordance with human values and goals (though what exactly this means remains contested). Many prevalent alignment approaches, such as RLHF and Constitutional AI, suffer from limited interpretability and lack formal guarantees. Bridging these approaches with formal frameworks, both computational and normative, seems tempting, as it promises more rigorous and interpretable alignment techniques.

In this seminar, we explored a rich repertoire of papers: foundational work on the current state of alignment methods; prima facie suitable formal approaches from computer science (including temporal logics and reward machines) and philosophy (such as deontic logics and theories of reasons); and neuro-symbolic architectures that attempt to combine the strengths of learned systems with formal methods. Participants discussed these different strands of research with one another and with members of our interdisciplinary research group.

Requirements

Genuine interest in questions related to AI alignment and AI safety
Familiarity with some of the following areas: agentic AI, reinforcement learning, formal logic and formal methods, practical reasoning, and practical philosophy
Ability and willingness to read and critically discuss English-language research papers
Openness to interdisciplinary perspectives and active, engaged participation in seminar discussions

Topics

Block A: State-of-the-Art AI Alignment

A1, RLHF basics: Deep Reinforcement Learning from Human Preferences; Training Language Models to Follow Instructions with Human Feedback
A2, RLHF-based alignment and Constitutional AI: Constitutional AI: Harmlessness from AI Feedback; Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
A3, LLM guardrails: Symbolic Guardrails for Domain-Specific Agents: Stronger Safety and Security Guarantees Without Sacrificing Utility; AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents

Block B: Normative Logics

B1, Normative systems: A Temporal Logic of Normative Systems; On social laws for artificial agent societies: off-line design
B2, Logic of reasons: Piecemeal Knowledge Acquisition for Computational Normative Reasoning; Precedent and Open Texture
B3, Deontic temporal logic: Strategic Deontic Temporal Logic as a Reduction to ATL, with an Application to Chisholm’s Scenario; Deontic Temporal Logic for Formal Verification of AI Ethics
B4, Logic and verification: Formal verification of ethical choices in autonomous systems; Specifying Agent Ethics (Blue Sky Ideas)

Block C: Logic-Based RL

C1, RL with reward machines: Synthesising Reward Machines for Cooperative Multi-Agent Reinforcement Learning; Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
C2, RL with temporal logics: Reinforcement Learning Based Temporal Logic Control with Maximum Probabilistic Satisfaction; Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions
C3, Ethical RL agents: Enforcing Ethical Goals Over Reinforcement Learning Policies; Learning Normative Behaviour Through Automated Theorem Proving
C4, Shielding: Safe Reinforcement Learning via Shielding; Safe Reinforcement Learning Using Probabilistic Shields

Block D: Neuro-Symbolic Architectures

D1, Neurosymbolic AI: Neurosymbolic AI: The 3rd Wave; The Third AI Summer: AAAI Robert S. Engelmore Memorial Lecture
D2, World models: Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems; The Scientist AI: Safe by Design, by Not Desiring
D3, Neurosymbolic alignment: Hybrid Approaches for Moral Value Alignment in AI Agents
D4, GRACE: Breaking Up with Normatively Monolithic Agency with GRACE: A Reason-Based Neuro-Symbolic Architecture for Safe and Ethical AI Alignment

Organization

All information and materials are available in the CMS.

Special Topics in Deep Reinforcement Learning

Wed, 01 Apr 2026 00:00:00 +0000

Team

Prof. Dr. Verena Wolf, Timo P. Gros (lecturers)
Magnus Cunow (organization)
Julius Gabelmann, Joshua Meyer, Nicola J. Müller (assistants)

Content

Reinforcement Learning (RL) is a popular sub-discipline of Machine Learning for problems that require strategies to solve complex tasks such as board games, scheduling problems, or other discrete optimization problems. Deep Reinforcement Learning (DRL) is the combination of Reinforcement Learning with Neural Networks.

In this seminar, we first covered the most important algorithms of Deep Reinforcement Learning. Afterwards, we looked at special topics that our group is conducting research on: exploration techniques, agentic RL, and AI planning.

Format

After a kick-off in April, each participant was assigned a paper and presented it in one of two seminar blocks in June, wrote a summary paper about it, and worked on a practical project in a team. The seminar ended with a closing session on the project results, with prizes, and short presentations of the group’s current research for students interested in a Master’s thesis or a HiWi position.

Topics

Deep Reinforcement Learning

Value-based DRL Evolution (core: Rainbow)
Actor-Critic: Structure & Estimation (core: A2C/A3C)
Policy Optimization Stability (core: TRPO & PPO)
Continuous Control in DRL (core: DDPG & SAC)

Planning & RL

Exploration

Agentic RL

Organization

All information and materials are available in the CMS.

Statistics Lab

Wed, 01 Apr 2026 00:00:00 +0000

Team

Prof. Dr. Verena Wolf (lecturer)
Pascal L. Held, Pascal Schumann (assistants)

Course Goals

This course serves as an introduction to the fundamentals of statistics and probability theory and covers most of the core principles of modern data analysis and predictive modeling:

Descriptive Statistics and Visualizations in Python
Probabilities and Discrete Random Variables
Continuous Random Variables and Laws of Large Numbers
Parameter Estimation
Statistical Testing
Computer Simulations and Monte Carlo Methods
Stochastic Processes

Format

The course follows an inverted classroom approach and is divided into three modules:

Descriptive Statistics, Probability & Random Variables
Distributions & Parameter Estimation
Statistical Testing

Each week consists of:

Monday lecture (GHH, Prof. Wolf): a large group session covering the week’s material, including worked examples from the exercise sheet.
Tutorials: small group sessions to work on the previous week’s exercises, discuss solutions, and take mini-tests.
Self-study with ALA: practice exercises with ALA, an AI chatbot tailored to the course content that guides you through problems and answers your questions.
Videos and materials: new learning materials for self-study, released weekly.

To be admitted to the final exam, students complete the exercises and mini-tests in the tutorials and submit several dialogues with ALA.

Organization

All information and materials are available in the CMS.

Customkinfraglib: Filtering the kinase-focused fragmentation library

Sun, 01 Mar 2026 00:00:00 +0000

B-cos GNNs: Faithful Explanations through Dynamic Linearity

Thu, 01 Jan 2026 00:00:00 +0000

PGTG: Procedurally Generated Grid-Based Traffic Gym

Thu, 01 Jan 2026 00:00:00 +0000

Toward a General Framework for Evaluating Per-Domain Generalization Using LLMs, Theorem Provers, and Statistical Model Checking

Thu, 01 Jan 2026 00:00:00 +0000

Utilitarismus und das Problem kollektiven Handelns

Thu, 01 Jan 2026 00:00:00 +0000

What are AI systems? Rethinking the core definition in the EU AI Act

Thu, 01 Jan 2026 00:00:00 +0000

Learning Per-Domain Generalizing Policies Using Offline Reinforcement Learning

Sat, 01 Nov 2025 00:00:00 +0000

The Journey of Data: Lessons Learned in Modeling Kinase Affinity, Selectivity, and Resistance [Article v1. 0]

Sat, 01 Nov 2025 00:00:00 +0000

PyDSMC: Statistical Model Checking for Neural Agents Using the Gymnasium Interface

Wed, 01 Oct 2025 00:00:00 +0000

Reinforcement Learning

Wed, 01 Oct 2025 00:00:00 +0000

Team

Prof. Dr. Verena Wolf (lecturer)
Lisa Dargasz, Yannic Muskalla (assistants)

Reinforcement Learning

Reinforcement learning is an area of machine learning where the goal is to develop (near-)optimal policies for solving sequential decision-making problems. The policy is typically represented by an agent who learns to achieve a goal by interacting with the environment. RL is often seen as the third area of machine learning (in addition to supervised and unsupervised learning), in which training samples are generated as a result of the agent’s actions and interaction with the environment. In recent years, there have been remarkable successes in reinforcement learning research in both theoretical and applied fields. These successes are mostly the result of a new development in the field: representing policies by artificial neural networks allows us to solve much more complex decision problems.

Course Content

This course provides a broad introduction to reinforcement learning and its applications. You learn about Markov Decision Processes as the underlying formal framework for decision-making problems, as well as popular reinforcement learning algorithms such as Monte-Carlo methods, temporal difference methods, and different “deep” reinforcement learning approaches. We use the open-source Python library Gymnasium to train RL agents in different pre-built environments.

Recommended prerequisites are Programming 1, Programming 2 and basic knowledge of probability theory, as taught for example in Mathematics for Computer Scientists 3.

Format

The course is a 6 ECTS advanced lecture with weekly lectures, bi-weekly assignment sheets with theoretical and programming exercises, tutorials, office hours and a final exam. Submitting the assignment sheets is voluntary. Admission to the final exam requires passing a mid-term exam.

Literature

Richard S. Sutton and Andrew G. Barto: Reinforcement Learning: An Introduction
Csaba Szepesvári: Algorithms for Reinforcement Learning

Organization

All information and materials are available in the CMS.

Visual Analysis of Action Policy Behavior: A Case Study in Grid-World Driving

Wed, 01 Oct 2025 00:00:00 +0000

Outcomes Addressed by Whole-Body Electromyostimulation Trials in Sportspeople and Athletes—An Evidence Map Summarizing and Categorizing Current Findings

Mon, 01 Sep 2025 00:00:00 +0000

Using Action-Policy Testing in RL to Reduce the Number of Bugs

Fri, 01 Aug 2025 00:00:00 +0000

Sortability of Time Series Data

Tue, 01 Jul 2025 00:00:00 +0000

Structural Causal Bottleneck Models

Sun, 01 Jun 2025 00:00:00 +0000

Reference values for static posturography of sportive and healthy adults aged 18-30 years

Tue, 01 Apr 2025 00:00:00 +0000

Spatiotemporal Models and Inference

Tue, 01 Apr 2025 00:00:00 +0000

Team

Dr. Jonas Wahl (lecturer)
Felix Jahn, Pascal Schumann (assistants)

Content

In many scientific fields, data is not just tied to a single location or a single moment in time: it evolves over both space and time. Whether tracking the spread of infectious diseases, predicting climate patterns, modeling traffic flows, or analyzing environmental pollution, understanding how variables change across both dimensions is crucial. This is where spatiotemporal modeling comes in.

Spatiotemporal models aim to capture dependencies and relationships in data that vary over time and across geographical locations. Unlike purely spatial or purely temporal models, these approaches integrate both dimensions, allowing us to uncover trends, make predictions, and quantify uncertainties in dynamic systems. Spatiotemporal models can be of different types:

Deterministic models, based on physical laws or equations (e.g., fluid dynamics for weather modeling).
Statistical models, which use probability distributions to describe uncertainties (e.g., Gaussian processes, Bayesian hierarchical models).
Machine learning models, which learn patterns from large datasets using neural networks, random forests, or deep learning approaches.

The latter two were the main focus of this seminar. Applications of spatiotemporal modeling are found across the sciences, from forecasting the spread of diseases and analyzing temperature changes over decades to optimizing transportation networks and tracking animal migration or deforestation. The seminar covered the theoretical underpinnings of spatiotemporal statistics, Bayesian approaches and hierarchical models, and machine learning techniques for complex spatiotemporal data.

Format

A block seminar for up to 10 participants, with a kick-off in June. Each participant gave a 60-minute presentation. Grading: presentation (50%), participation in the talks of fellow participants and the discussion (25%), and a final report on the topic (25%).

Prerequisites: successful participation in a machine or deep learning course, or a background in statistical modeling, for instance from a course on statistics or stochastic processes.

Topics

Statistics for Spatiotemporal Data, Chapter 2: Statistical preliminaries
Statistics for Spatiotemporal Data, Chapter 3: Fundamentals of Temporal Processes
Statistics for Spatiotemporal Data, Chapter 4: Fundamentals of Spatial Random Processes
Introduction to Gaussian processes (lecture notes here and here, and the book)
Coding tutorial on spatial Gaussian processes (based on this blog post and this one)
Introduction to point processes (lecture notes here, slides here and here)
Coding tutorial on point processes (with the tick package, this simulator, these Jupyter notebooks, or spatial point patterns in R)
Scalable spatiotemporal prediction with Bayesian neural fields, with a coding tutorial on Bayesian neural fields
A Bayesian machine-learning approach for spatiotemporal prediction of COVID-19 cases
Spatial-temporal graph neural networks for groundwater data

Organization

All information and materials are available in the CMS.

Influence of WB-EMS on strength endurance parameters and subjectively perceived back pain: A multicentre study

Sat, 01 Mar 2025 00:00:00 +0000

A Roadmap for Responsible Robotics: Promoting Human Agency and Collaborative Efforts