Tools | Neuro-Mechanistic Modeling

PGTG

Mon, 01 Jan 0001 00:00:00 +0000

Overview

PGTG (Procedurally Generated Grid-Based Traffic Gym) is a Gymnasium-compliant, feature-rich, extensible, and customizable reinforcement learning benchmark. An agent has to navigate procedurally generated tracks while managing momentum and coping with stochastic obstacles and the unpredictable behavior of other traffic participants.

Map generation, traffic rules, obstacles, and the observation and reward functions can all be configured in detail, and each feature can be toggled individually. This lets researchers study exploration, safety, and generalization separately or jointly to trade-offs between them (e.g. how safety constraints affect exploration) in one environment. Optional subgoals with intermediate rewards, for instance, decouple safety from exploration. In the accompanying empirical study, no tested state-of-the-art deep RL algorithm reaches satisfactory performance, which leaves plenty of room for future research.

Usage

PGTG requires at least Python 3.11 and is available on PyPI:

pip install pgtg

Importing the package registers the environment with Gymnasium as a side effect:

import gymnasium as gym
import pgtg  # registers "pgtg-v5"

env = gym.make("pgtg-v5")

The repository also includes runnable example notebooks (interactive playground, Stable-Baselines3 training, and statistical evaluation with PyDSMC). See the documentation for the configuration reference and a guide to adding custom obstacles.

Citation

PGTG was presented at the Reinforcement Learning Conference 2026 (RLC 2026) and is published in the accompanying Reinforcement Learning Journal. If you use it in your own work, please cite:

@article{meyerPGTG2026,
  title   = {{{PGTG}}: Procedurally Generated Grid-Based Traffic Gym},
  author  = {Meyer, Joshua and Kuntz, Felix Maurice and Wolf, Verena and
             Hoffmann, J{\"o}rg and Gros, Timo P.},
  journal = {Reinforcement Learning Journal},
  volume  = {7},
  year    = {2026}
}

PyDSMC

Mon, 01 Jan 0001 00:00:00 +0000

PyDSMC is an open-source Python library for statistical model checking of neural agents on arbitrary Gymnasium environments. It is designed to be lightweight and easy-to-use while being based on established statistical methods to provide guarantees on the investigated agents’ performances on various properties. Implementing the Gymnasium interface, PyDSMC is widely applicable and fully agnostic to the environments underlying implementation.

PyDSMC is based on Deep Statistical Model Checking and aims to facilitate greater adoption of statistical model checking by simplifying its usage.

Please consult the accompanying paper for more details.

📖 Full documentation (with an in-browser demo) is available at neuro-mechanistic-modeling.github.io/PyDSMC.

Deep Statistical Model Checking

The train and evaluation curves of neural agents differ greatly throughout training. Just inspecting the training curves does not suffice and leads to suboptimal policy extraction. We advocate to perform statistically-backed model evaluations periodically throughout training to ensure a good extraction point.

Setup

PyDSMC can be installed from PyPI using pip install pydsmc.

We recommend using a virtual environment and officially tested python versions 3.10, 3.11, and 3.12.

To work on this project, set up a virtual environment and install all necessary dependencies—for instance, using virtualenvwrapper and executing:

mkvirtualenv --python=python3.11 pydsmc
pip install -r requirements.txt

Usage

PyDSMC mainly exposes two functionalities: Property creation and the Evaluator.

Properties

PyDSMC can analyze arbitrary properties. These can also be environment-specific. For ease-of-use, we provide ready-to-use implementations of commonly used, domain-independent properties that are parameterized and can, thus, be adjusted to each individual usecase. You can get a list of all available predefined properties by calling pydsmc.get_predefined_properties().

Creating a predefined property is straightforward. For instance, a property analyzing the achieved return could be defined as follows:

from pydsmc import create_predefined_property

return_property = create_predefined_property(
    property_id="return",   # Which predefined property to use
    name="returnGamma0.97", # Property's name, used for storing the evaluation results
    epsilon=0.025,          # Half-width of the requested confidence interval (CI)
    kappa=0.05,             # Probability that the true mean lies within the CI
    relative_error=True,    # Whether epsilon represents the relative or absolute error
    bounds=(0, 864),        # Bounds of the property, i.e., min and max possible values
    sound=True,             # Whether a sound statistical method should be used
    gamma=0.97              # Property specific attributes
)

Creating a custom property is equally simple. The only additional information required is a checking function check_fn that analyzes trajectory information and returns a float value as a sample result for the property:

from pydsmc import create_custom_property

crash_property = create_custom_property(
    name="crash_prob",      # see above
    epsilon=0.025,          # see above
    kappa=0.05,             # see above
    relative_error=False,   # see above
    binomial=True,          # This property follows a binomial distribution
    bounds=(0, 1),          # see above
    sound=False,            # see above
    # The property's checking function, crash identified by last reward -100
    check_fn=lambda self, t: float(t[-1].reward == -100)
)

Evaluator

We can then register the properties in an Evaluator which will do the work of evaluating all the properties until convergence is reached, or the given resource budget is exhausted.

In order to construct an Evaluator, it is first necessary to create the evaluation environments from the outside. For this, we provide a helper function create_eval_envs that creates a suitable vectorized environment according to the given configuration.

from pydsmc import create_eval_envs, Evaluator

NUM_THREADS = 1  # Recommended: 1; Often better to use more parallel environments instead
NUM_PAR_ENVS = 8  # Number of parallel environments _per thread_

envs = create_eval_envs(
    num_threads=NUM_THREADS,
    num_envs_per_thread=NUM_PAR_ENVS,
    env_seed=42,  # Seeds are incremented for each environment, making them unique
    gym_id="HalfCheetah-v5",
    wrappers=[gym.wrappers.NormalizeObservation, Monitor],
    vecenv_cls=gym.vector.AsyncVectorEnv,  # gym.vector.SyncVectorEnv, sb3.DummyVecEnv, sb3.SubprocVecEnv
    # The following kwargs are passed to the gym.make function, which passes unknown args to the env
    max_episode_steps=1000,
)

evaluator = Evaluator(env=envs)

Having created the Evaluator, we can register the properties and start the evaluation:

evaluator.register_property(return_property)
evaluator.register_property(crash_property)
### Or register multiple properties at once using:
# evaluator.register_properties([return_property, crash_property])

### Start the evaluation
evaluator.eval(
    agent=agent,  # or: predict_fn=agent.predict
    stop_on_convergence=True,  # Stop when all properties have converged
    episode_limit=10_000,  # Early stop evaluation after 10,000 episodes
    time_limit=60,  # Early stop evaluation after 1 hour
    num_episodes_per_policy_run=100,
    num_threads=NUM_THREADS,
    deterministic=True,
)

Full example

A few full examples on a select set of environments can be found in example_agents to try out.

Statistical Method Selection

For each property, PyDSMC automatically selects an appropriate statistical method based on the provided parameters, following decision tree below:

Citing PyDSMC

If you use PyDSMC in your work, you can use the following BibTeX entry, citing the official Version of Record published at QEST2025.

@inproceedings{grosPyDSMC2025,
  title      = {{{PyDSMC}}: {{Statistical Model Checking}} for~{{Neural Agents Using}} the~{{Gymnasium Interface}}},
  shorttitle = {{{PyDSMC}}},
  booktitle  = {Quantitative {{Evaluation}} of {{Systems}} and {{Formal Modeling}} and {{Analysis}} of {{Timed Systems}}},
  author     = {Gros, Timo P. and Hartmanns, Arnd and Hoese, Ivo and Meyer, Joshua and M{\"u}ller, Nicola J. and Wolf, Verena},
  editor     = {Prabhakar, Pavithra and Vandin, Andrea},
  year       = {2025},
  pages      = {134--156},
  publisher  = {Springer Nature Switzerland},
  address    = {Cham},
  doi        = {10.1007/978-3-032-05792-1_8},
  isbn       = {978-3-032-05792-1},
}

License

The code introduced by this project is licensed under the MIT license. Please consult the bundled LICENSE file for the full license text.

SepDistances

Mon, 01 Jan 0001 00:00:00 +0000

This repository provides an implementation of separation distances and s/c-metrics for causal graphs as introduced in the paper ‘Separation-based distance metrics for causal graphs’.

You can also install it as a package using pip install sep-distances. The test suite is not part of the package on pypi, but it is still available on the git repo.

Below is a short description of the important source files and how to use them.

Files

codebase/mixed_graph.py
- Contains the LabelledMixedGraph class: a compact representation for directed, undirected, bidirected and semidirected edges. Useful helpers include conversion to/from NetworkX, adding/removing edges, methods to compute skeletons and CPDAGs, finding v-structures, computing Markov blankets, BayesBall, minimal d-separators and several converters (get_canonical_directed_graph, get_acyclification, etc.). This is the main graph data structure used by the metrics.
codebase/metrics.py
- Implements the distance/metric functions comparing two graphs. Major groups of functions:
  - SHD (Structural Hamming Distance) for DAGs/CPDAGs/MAGs: SHD_DAGs, SHD_CPDAGs, SHD_MAGs.
  - AID (Adjustment Identification Distances) wrappers using gadjid (e.g. parent_AID_DAGs).
  - Separation distances (SD) for DAGs, CPDAGs, and mixed graphs: SD_DAGs, SD_CPDAGs, SD_mixed_graphs.
  - s/c-metrics and variants (s-metric, c-metric, sc-metric) for DAGs, CPDAGs, mixed graphs and graphs with cycles (metric_DAGs, metric_CPDAGs, metric_mixed_graphs, metric_directed_cyclic_graphs).
  - Utilities: generate_triples (create separation statements) and several helper wrappers.
tests/
- Unit tests covering behavior of mixed_graph.py and metrics.py. The tests include many focused test cases (CPDAGs, DAGs, mixed graphs, AIDs, SHD, SD, s/c metrics, and related helpers). Run them with pytest (instructions below).

Quick examples

Basic usage pattern in example.py (import, construct graphs, compute a metric):

from codebase import mixed_graph as mg
from codebase import metrics as metrics

# create two simple DAGs
G1 = mg.LabelledMixedGraph(nodes={"A", "B", "C"})
G1.add_directed("A", "B")
G1.add_directed("B", "C")

G2 = mg.LabelledMixedGraph(nodes={"A", "B", "C"})
G2.add_directed("A", "B")
G2.add_directed("A", "C")

# Compute SHD (Structural Hamming Distance) between DAGs
shd = metrics.SHD_DAGs(G1, G2, normalized=True)
print("SHD (normalized):", shd)

# Compute separation distance (SD) using parent separation
sd = metrics.SD_DAGs(G1, G2, type='parent', normalized=True)
print("SD (parent, normalized):", sd)

# Compute an sc-metric between DAGs (default uses all orders)
sc = metrics.metric_DAGs(G1, G2, type='sc', normalized=True)
print("sc-metric:", sc)

Notes:

Many metric functions expect the same node set in both graphs. They typically check graph1.nodes == graph2.nodes.
For CPDAGs some functions compute or require a representative DAG of the MEC (see get_representative_of_MEC).
AID functions rely on the external gadjid package; install it to use those functions.

Running tests

Install dependencies (recommended in a virtual environment):

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Run the test suite with pytest from the project root:

pytest -q

If you only want to run a single test file, use e.g. pytest -q tests/test_SHD.py.

Currently, the test suite does not have any tests configured. Instead you should use run it once to check if all the requirements are installed correctly and as example usage for the respective metrics.

If you want to run a test file directly, you need to change the way the imports are called. Currently the imports are structured in a way that work well with pytest, but won’t work if you run the file directly.

Example change in tests/test_AIDs.py.:

''' Comment out current imports '''
# from codebase import mixed_graph as mixed
# from codebase import metrics

# from ..tests import Graphs_for_testing as G_testing

''' Replace with the following '''
import os
import sys

# add project root to sys.path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))

from codebase import mixed_graph as mixed
from codebase import metrics

import Graphs_for_testing as G_testing

Installation / requirements

Python 3.10+ is required (see setup.py).
The project depends on packages listed in requirements.txt / setup.py. Notable dependencies:
- networkx, numpy, scipy, gadjid (optional but required for AID functions).

License

This project is released under the GNU General Public License v3 (GPLv3). See LICENSE.txt for the full text.

Contact & attribution

Original research and initial code: Jonas Wahl & Jakob Runge.

Package implementation and maintenance: Muhammad Haris Owais Ahmed.

If you have questions, bug reports, or performance suggestions, please open an issue or contact the maintainers listed in setup.py.

Citation

@InProceedings{wahlSeparationBasedDistanceMeasures2025,
  title = 	 {Separation-Based Distance Measures for Causal Graphs},
  author =       {Wahl, Jonas and Runge, Jakob},
  booktitle = 	 {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics},
  pages = 	 {3412--3420},
  year = 	 {2025},
  editor = 	 {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz},
  volume = 	 {258},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {03--05 May},
  publisher =    {PMLR},
  url = 	 {https://proceedings.mlr.press/v258/wahl25b.html},
}