model

This module holds all of the neural network definitions that support self-supervised learning.

blendowski

selfsupervised3d.model.blendowski

neural network definitions for Blendowski-style [1,2] model

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: April 28, 2020

class selfsupervised3d.model.blendowski.D2DConvNet(input_channels: int = 3, descriptor_size: int = 64)

D2DConvNet [1] is used to learn anatomical knowledge via spatial relations; serves as the feature extractor. Modified from original (replaces max pool w/ stride) original expected input size: (c,h,w) = (3,42,42) output size: (c,h,w) = (64,1,1)

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

class selfsupervised3d.model.blendowski.HeatNet(descriptor_size: int = 64, heatmap_dim: int = 19)

Network creates a heatmap as described in [1]. Quoting, and paraphrasing where appropriate, from [2]: “in a siamese-fashion, process two patches with the [D2DConvNet] to

generate feature descriptors for these patches and, subsequently, pass both descriptors to HeatNet. These feature descriptors of size (64,1,1) are concatenated and input to this network which is trained to output a spatial heatmap of size (1,19,19); the ground-truth is generated with the function heatmap”

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

class selfsupervised3d.model.blendowski.OffNet(descriptor_size: int = 64)

Network attempts to find the offset parameters as described in [1]. Quoting, and paraphrasing where appropriate, from [2]: “in a siamese-fashion, process two patches with the [D2DConvNet] to

generate feature descriptors for these patches and, subsequently, pass both descriptors to HeatNet. These feature descriptors of size (64,1,1) are concatenated and input to this network which is trained to output the two offset parameters that define the in-plane displacement.”

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

discriminator

selfsupervised3d.model.discriminator

neural network GAN definition (typically to support context encoder-style self-supervised learning methods [1]). Inspired by [2].

References

[1] D. Pathak et al. “Context encoders: Feature learning by inpainting.”

CVPR. 2016.

[2] https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: May 06, 2020

class selfsupervised3d.model.discriminator.PatchDiscriminator(input_nc: int, ndf: int = 64, n_layers: int = 3, norm_layer=<sphinx.ext.autodoc.importer._MockObject object>)

doersch

selfsupervised3d.model.doersch

neural network definitions for Doersch-style [1,2] self-supervised models as described/defined in [2,3]

References

[1] C. Doersch et al. “Unsupervised visual representation learning

by context prediction.” ICCV. 2015.

[2] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[3] https://github.com/multimodallearning/miccai19_self_supervision

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: April 28, 2020

class selfsupervised3d.model.doersch.DoerschDecodeNet(descriptor_size: int = 192, conv_channels: int = 64)

Takes two feature descriptors (viz., vectors) as input; created from DoerschNet In this case, the task is a classification problem; the output guesses the relative position of descriptor1 w.r.t. descriptor2 (one of six major directions) original expected input size: (c,d,h,w) = (192,1,1,1), i.e., a vector w/ extra dimensions output size: (c,) = (6,)

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

class selfsupervised3d.model.doersch.DoerschNet(input_channels: int = 1, descriptor_size: int = 192, conv_channels: int = 16)

Creates the 3D feature descriptors as described in the extended Doersch approach [1]. original expected input size: (c,d,h,w) = ([user-defined],25,25,25) output size: (c,d,h,w) = (192,1,1,1), i.e., a vector w/ extra dimensions

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:

Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

frankunet

selfsupervised3d.model.frankunet

A Frankenstein’s monster version of a U-Net neural network (typically to support context encoder-style self-supervised learning methods [1])

References

[1] D. Pathak et al. “Context encoders: Feature learning by inpainting.”

CVPR. 2016.

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: May 06, 2020

class selfsupervised3d.model.frankunet.FrankUNet(nc: int = 32, ic: int = 1, oc: int = 1, p: float = 0.15, concat: bool = True, noise_lvl: float = 0.01)