selfsupervised3d

This package provides functions to support self-supervised with 3D images (e.g., CT and MR structural images).

dataset

This module holds all of the dataset classes for creating self-supervised specific images.

blendowski

selfsupervised3d.dataset.blendowski

datasets to support blendowski-style self-supervised learning methods

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: April 28, 2020

class selfsupervised3d.dataset.blendowski.BlendowskiDataset(img_dir: List[str], patch_size: float = 0.4, patch_dim: int = 42, offset: float = 0.3, stack_size: float = 0.05, stack_dim: int = 3, min_off_inplane: float = 0.25, max_off_inplane: float = 0.3, min_off_throughplane: float = 0.125, max_off_throughplane: float = 0.25, heatmap_dim: int = 19, scale: float = 10.0, precision: float = 15.0, throughplane_axis: int = 0)
selfsupervised3d.dataset.blendowski.blendowski_collate(lst)

collate function to integrate BlendowskiDataset with PyTorch DataLoader

selfsupervised3d.dataset.blendowski.blendowski_patches(img: <sphinx.ext.autodoc.importer._MockObject object at 0x7f9fd35f14a8>, patch_size: float = 0.4, patch_dim: int = 42, offset: float = 0.3, stack_size: float = 0.05, stack_dim: int = 3, min_off_inplane: float = 0.25, max_off_inplane: float = 0.3, min_off_throughplane: float = 0.125, max_off_throughplane: float = 0.25, heatmap_dim: int = 19, scale: float = 10.0, precision: float = 15.0, throughplane_axis: int = 0)

Creates patches and targets for self-supervised learning as described in [1]

Parameters:
  • img (torch.Tensor) – image from which to create patches
  • patch_size (float) – size of patch as a proportion of the image
  • patch_dim (int) – side length of the cube in voxels to be extracted
  • offset (float) – proportion of image away from the center patch allowed
  • stack_size (float) – proportion of image that comprises the throughplane stack
  • stack_dim (float) – dimension of image that comprises the throughplane stack
  • min_off_inplane (float) – minimum offset in the inplane direction
  • max_off_inplane (float) – maximum offset in the inplane direction
  • min_off_throughplane (float) – minimum offset in the throughplane direction
  • max_off_throughplane (float) – maximum offset in the throughplane direction
  • heatmap_dim (int) – dimension in pixels of the target heatmap
  • scale (float) – constant scale value multiplying the gaussian term (see the eq. in Details on Heatmap Network Training in [1])
  • precision (float) – value of precision (1/var) in the gaussian term (see the eq. in Details on Heatmap Network Training in [1])
  • throughplane_axis (int) – axis selected as throughplane (0, 1, or 2)

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

context

selfsupervised3d.dataset.context

datasets to support context encoder-style self-supervised learning methods

References

[1] D. Pathak et al. “Context encoders: Feature learning by inpainting.”
CVPR. 2016.

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: May 06, 2020

class selfsupervised3d.dataset.context.ContextDataset(img_dir: List[str], mask_dir: Optional[str] = None, n_blocks: int = 5, size: int = 10, n_erode: Optional[int] = 4, patch_size: Optional[int] = None)
selfsupervised3d.dataset.context.context_collate(lst)

collate function to integrate ContextDataset with PyTorch DataLoader

selfsupervised3d.dataset.context.create_block_mask(idx_mask: <sphinx.ext.autodoc.importer._MockObject object at 0x7f9fd35f85c0>, size: int, n_erode: Optional[int] = None, fill_val: float = 1.0)

creates a mask containing a block inside a given mask

selfsupervised3d.dataset.context.create_multiblock_mask(idx_mask: <sphinx.ext.autodoc.importer._MockObject object at 0x7f9fd35f8860>, n_blocks: int, size: int, n_erode: Optional[int] = None, fill_val: float = 1.0)

creates a mask containing multiple (potentially overlapping) blocks inside a given mask

doersch

selfsupervised3d.dataset.doersch

datasets to support doersch-style self-supervised learning methods

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision [3] C. Doersch et al. “Unsupervised visual representation learning

by context prediction.” ICCV. 2015.

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: April 28, 2020

class selfsupervised3d.dataset.doersch.DoerschDataset(img_dir: List[str], patch_size: float = 0.4, patch_dim: int = 25, offset: float = 0.5)
selfsupervised3d.dataset.doersch.doersch_collate(lst)

collate function to integrate DoerschDataset with PyTorch DataLoader

selfsupervised3d.dataset.doersch.doersch_patches(img: <sphinx.ext.autodoc.importer._MockObject object at 0x7f9fd35f8be0>, patch_size: float = 0.2, patch_dim: int = 25, offset: float = 0.5)

Creates Doersch-style patches [1] as described in [2]

Parameters:
  • img (torch.Tensor) – image from which to create patches
  • patch_size (float) – size of patch as a proportion of the image
  • patch_dim (int) – side length of the cube in voxels to be extracted
  • offset (float) – proportion of image away from the center patch allowed

References

[1] C. Doersch et al. “Unsupervised visual representation learning
by context prediction.” ICCV. 2015.
[2] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

model

This module holds all of the neural network definitions that support self-supervised learning.

blendowski

selfsupervised3d.model.blendowski

neural network definitions for Blendowski-style [1,2] model

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: April 28, 2020

class selfsupervised3d.model.blendowski.D2DConvNet(input_channels: int = 3, descriptor_size: int = 64)

D2DConvNet [1] is used to learn anatomical knowledge via spatial relations; serves as the feature extractor. Modified from original (replaces max pool w/ stride) original expected input size: (c,h,w) = (3,42,42) output size: (c,h,w) = (64,1,1)

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

class selfsupervised3d.model.blendowski.HeatNet(descriptor_size: int = 64, heatmap_dim: int = 19)

Network creates a heatmap as described in [1]. Quoting, and paraphrasing where appropriate, from [2]: “in a siamese-fashion, process two patches with the [D2DConvNet] to

generate feature descriptors for these patches and, subsequently, pass both descriptors to HeatNet. These feature descriptors of size (64,1,1) are concatenated and input to this network which is trained to output a spatial heatmap of size (1,19,19); the ground-truth is generated with the function heatmap”

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

class selfsupervised3d.model.blendowski.OffNet(descriptor_size: int = 64)

Network attempts to find the offset parameters as described in [1]. Quoting, and paraphrasing where appropriate, from [2]: “in a siamese-fashion, process two patches with the [D2DConvNet] to

generate feature descriptors for these patches and, subsequently, pass both descriptors to HeatNet. These feature descriptors of size (64,1,1) are concatenated and input to this network which is trained to output the two offset parameters that define the in-plane displacement.”

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

discriminator

selfsupervised3d.model.discriminator

neural network GAN definition (typically to support context encoder-style self-supervised learning methods [1]). Inspired by [2].

References

[1] D. Pathak et al. “Context encoders: Feature learning by inpainting.”
CVPR. 2016.

[2] https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: May 06, 2020

class selfsupervised3d.model.discriminator.PatchDiscriminator(input_nc: int, ndf: int = 64, n_layers: int = 3, norm_layer=<sphinx.ext.autodoc.importer._MockObject object>)

doersch

selfsupervised3d.model.doersch

neural network definitions for Doersch-style [1,2] self-supervised models as described/defined in [2,3]

References

[1] C. Doersch et al. “Unsupervised visual representation learning
by context prediction.” ICCV. 2015.
[2] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[3] https://github.com/multimodallearning/miccai19_self_supervision

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: April 28, 2020

class selfsupervised3d.model.doersch.DoerschDecodeNet(descriptor_size: int = 192, conv_channels: int = 64)

Takes two feature descriptors (viz., vectors) as input; created from DoerschNet In this case, the task is a classification problem; the output guesses the relative position of descriptor1 w.r.t. descriptor2 (one of six major directions) original expected input size: (c,d,h,w) = (192,1,1,1), i.e., a vector w/ extra dimensions output size: (c,) = (6,)

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

class selfsupervised3d.model.doersch.DoerschNet(input_channels: int = 1, descriptor_size: int = 192, conv_channels: int = 16)

Creates the 3D feature descriptors as described in the extended Doersch approach [1]. original expected input size: (c,d,h,w) = ([user-defined],25,25,25) output size: (c,d,h,w) = (192,1,1,1), i.e., a vector w/ extra dimensions

References

[1] M. Blendowski et al. “How to Learn from Unlabeled Volume Data:
Self-supervised 3D Context Feature Learning.” MICCAI. 2019.

[2] https://github.com/multimodallearning/miccai19_self_supervision

frankunet

selfsupervised3d.model.frankunet

A Frankenstein’s monster version of a U-Net neural network (typically to support context encoder-style self-supervised learning methods [1])

References

[1] D. Pathak et al. “Context encoders: Feature learning by inpainting.”
CVPR. 2016.

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: May 06, 2020

class selfsupervised3d.model.frankunet.FrankUNet(nc: int = 32, ic: int = 1, oc: int = 1, p: float = 0.15, concat: bool = True, noise_lvl: float = 0.01)

loss

This module holds some self-supervised specific loss functions.

selfsupervised3d.loss

loss functions to support some self-supervised methods

Author: Jacob Reinhold (jacob.reinhold@jhu.edu)

Created on: May 06, 2020

class selfsupervised3d.loss.LSGANLoss(real_label: float = 1.0, fake_label: float = 0.0)
class selfsupervised3d.loss.InpaintLoss(alpha: Tuple[float, float] = (0.99, 0.01), beta: float = 100.0)

Indices and tables