Learning to Segment: Training Hierarchical Segmentation under a Topological Loss

MICCAI 2015Jan Funke, Fred A. Hamprecht, Chong Zhang

We propose a generic and efficient learning framework that is applicable to segment images in which individual objects are mainly discernible by boundary cues. Our approach starts by first hierarchically clustering the image and then explaining the image in terms of a cost-minimal subset of non-overlapping segments. The cost of a segmentation is defined as a weighted sum of features of the selected candidates. This formulation allows us to take into account an extensible set of arbitrary features. The maximally discriminative linear combination of features is learned from training data using a margin-rescaled structured SVM. At the core of our formulation is a novel and simple topology-based structured loss which is a combination of counts and geodesic distance of topological errors (splits, merges, false positives and false negatives) relative to the training set. We demonstrate the generality and accuracy of our approach on three challenging 2D cell segmentation problems, where we improve accuracy compared to the current state of the art. [paper]

A Tolerant Edit Distance for Evaluation and Training of Electron Microscopy Reconstruction Algorithms

arXiv 2015Jan Funke, Jonas Klein, Albert Cardona, Matthew Cook

We present a measure to compare the labeling of automatic neuron reconstruction methods against ground truth. This measure, which we call tolerant edit distance (TED), is motivated by two observations: (1) Some errors, like small boundary shifts, are tolerable in practice. Which errors are tolerable is application dependent and should be a parameter of the measure. (2) Non-tolerable errors have to be corrected manually. The time needed to do so should be reflected by the error measure and minimized during training. The TED finds the minimal weighted sum of split and merge errors exceeding a given tolerance criterion, and thus provides a time-to-fix estimate. Our measure works on both isotropic and anisotropic EM datasets, the results are intuitive, and errors can be localized in the volume. We also present a structured learning framework for assignment models for anisotropic neuron reconstruction and show how this framework can be used to minimize the TED on annotated training samples. Evaluated on two publicly available EM-datasets, our method shows consistently higher reconstruction accuracy, even on pre-existing measures, than other current learning methods. Furthermore, we show how an appropriately defined tolerance criterion allows us to train on skeleton (i.e., non-volumetric) annotations, which are much faster to obtain in practice. [paper][supplemental]

Optimal joint segmentation and tracking of escherichia coli in the mother machine

BAMBI 2014Florian Jug,Tobias Pietzsch,Dagmar KainmuellerJan Funke, Matthias Kaiser, Erik van Nimwegen, Carsten Rother, Gene Myers

We introduce a graphical model for the joint segmentation and tracking of E. coli cells from time lapse videos. In our setup cells are grown in narrow columns (growth channels) in a so-called “Mother Machine” [1]. In these growth channels, cells are vertically aligned, grow and divide over time, and eventually leave the channel at the top. The model is built on a large set of cell segmentation hypotheses for each video frame that we extract from data using a novel parametric max-flow variation. Possible tracking assignments between segments across time, including cell identity mapping, cell division, and cell exit events are enumerated. Each such assignment is represented as a binary decision variable with unary costs based on image and object features of the involved segments. We find a cost-minimal and consistent solution by solving an integer linear program. We introduce a new and important type of constraint that ensures that cells exit the Mother Machine in the correct order. Our method finds a globally optimal tracking solution with an accuracy of > 95% (1.22 times the inter-observer error) and is on average 2 − 11 times faster than the microscope produces the raw data. [paper]

Automatic Neuron Reconstruction from Anisotropic Electron Microscopy Volumes

PhD ThesisJan Funke

The work presented in this thesis addresses the problem of the automatic extraction of the wiring diagram of a nervous system from anisotropic electron microscopy volumes with high x- and y-resolution but low z-resolution, as obtained by serial section electron microscopy imaging procedures. A necessary step towards this goal is the segmentation of neural tissue to separate neuron cell interior from membrane and extracellular space, and thus reveal the 3D shape of each neuron, a process called neuron reconstruction.
The core of this thesis is a novel method for the reconstruction of neurons from serial section electron microscopy images. Due to the anisotropy of serial section imaging methods, we treat the data as a stack of 2D images, rather then a continuous 3D volume. However, the detection of neuron slices (i.e., cross-sections of neural processes) in 2D images is difficult due to ambiguities in the data. Therefore, we propose to enumerate several diverse and possibly contradictory candidate neuron slices by identifying separating membranes with varying thresholds for each image individually. Between candidates of adjacent images in the stack, we enumerate assignments that reflect possible ways to follow a neural process from one image to another. We assign costs to each candidate and assignment and formulate constraints that ensure consistency between the assignments. We show how a globally cost-minimal segmentation of neuron slices and assignments between images can be found jointly and efficiently. Furthermore, we derive a structured learning formulation to learn the assignment costs from annotated ground truth and show its effectiveness compared to other methods.
Since the candidate selection is a crucial step in our model, we also introduce an alternative candidate generation method that samples candidates from a conditional random field (CRF) based on convolutional neural network predictions. The CRF is designed and trained to capture the statistics of 2D electron microscopy images of neural tissue. We show that sampling from this model produces plausible neuron slice candidates that are well suited for our reconstruction method, while additionally providing labels for synapse, glia cells, and mitochondria.
For the application to very large datasets, inference has to be distributed. However, since our model performs a global optimization, this is not trivial. We tackle this problem by presenting a distribution scheme for our model that is based on dual decomposition and guarantees global optimality. For that, the original problem is decomposed into several regions that communicate with each other to find an agreement. If such an agreement can be found, the collected answers from all regions is provably optimal. We introduce a messaging strategy that ensures that such an agreement can always be found under suitable assumptions.
Finally, we review error measures used for neuron reconstruction algorithms and discuss their properties. We introduce a new measure that reflects the edit distance between a reconstruction and a ground truth within certain tolerated variations and compare it to existing measures.
Given the extremely high accuracy requirements for biological use cases and the challenging ambiguities encountered in EM images, the complete automatic reconstruction of neurons is still out of reach. Nevertheless, we believe that the methods introduced in this thesis made a significant contribution towards this goal and can already be used to assist the tedious manual reconstruction.


Candidate Sampling for Neuron Reconstruction from Anisotropic Electron Microscopy Volumes

MICCAI 2014Jan Funke, Julien Martel, Stephan Gerhard, Björn Andres Dan C. Ciresan, Alessandro Giusti, Luca Gambardella, Jürgen Schmidhuber Hanspeter Pfister, Albert Cardona, Matthew Cook

The automatic reconstruction of neurons from stacks of electron microscopy sections is an important computer vision problem in neuroscience. Recent advances are based on a two step approach: First, a set of possible 2D neuron candidates is generated for each section independently based on membrane predictions of a local classifier. Second, the candidates of all sections of the stack are fed to a neuron tracker that selects and connects them in 3D to yield a reconstruction. The accuracy of the result is currently limited by the quality of the generated candidates. In this paper, we propose to replace the heuristic set of candidates used in previous methods with samples drawn from a conditional random field (CRF) that is trained to label sections of neural tissue. We show on a stack of Drosophila melanogaster neural tissue that neuron candidates generated with our method produce 30% less reconstruction errors than current candidate generation methods. Two properties of our CRF are crucial for the accuracy and applicability of our method: (1) The CRF models the orientation of membranes to produce more plausible neuron candidates. (2) The interactions in the CRF are restricted to form a bipartite graph, which allows a great sampling speed-up without loss of accuracy. [paper][supplemental]

Efficient Automatic 3D-Reconstruction of Branching Neurons from EM Data

CVPR 2012Jan Funke, Björn Andres, Fred A. Hamprecht, Albert Cardona, Matthew Cook

We present an approach for the automatic reconstruction of neurons from 3D stacks of electron microscopy sections. The core of our system is a set of possible assignments, each of which proposes with some cost a link between neuron regions in consecutive sections. These can model the continuation, branching, and end of neurons. The costs are trainable on positive assignment samples. An optimal and consistent set of assignments is found for the whole volume at once by solving an integer linear program. This set of assignments determines both the segmentation into neuron regions and the correspondence between such regions in neighboring slices. For each picked assignment, a confidence value helps to prioritize decisions to be reviewed by a human expert. [paper][supplemental]

Multi-Hypothesis CRF-Segmentation of Neural Tissue in Anisotropic EM Volumes

arXiv.org 2011Jan Funke, Björn Andres, Fred A. Hamprecht, Albert Cardona, Matthew Cook

We present an approach for the joint segmentation and grouping of similar components in anisotropic 3D image data and use it to segment neural tissue in serial sections electron microscopy (EM) images. We first construct a nested set of neuron segmentation hypotheses for each slice. A conditional random field (CRF) then allows us to evaluate both the compatibility of a specific segmentation and a specific inter-slice assignment of neuron candidates with the underlying observations. The model is solved optimally for an entire image stack simultaneously using integer linear programming (ILP), which yields the maximum a posteriori solution in amortized linear time in the number of slices. We evaluate the performance of our approach on an annotated sample of the Drosophila larva neuropil and show that the consideration of different segmentation hypotheses in each slice leads to a significant improvement in the segmentation and assignment accuracy. [paper]

A Framework For Evaluating Visual SLAM

BMVC 2009Jan Funke, Tobias Pietzsch

Performance analysis in the field of camera-based simultaneous localisation and mapping (Visual SLAM, VSLAM) is still an unsolved problem. For VSLAM systems, there is a lack of generally accepted performance measures, test frameworks, and benchmark problems. Most researchers test by visually inspecting their systems on recorded image sequences, or measuring accuracy on simulated data of simplified point-cloud-like environments. Both approaches have their disadvantages. Recorded sequences lack ground truth. Simulations tend to oversimplify low-level aspects of the problem. In this paper, we propose to evaluate VSLAM systems on rendered image sequences. The intention is to move simulations towards more realistic conditions while still having ground truth. For this purpose, we provide a complete and extensible framework which addresses all aspects, from rendering to ground truth generation and automated evaluation. To illustrate the usefulness of this framework, we provide experimental results assessing the benefit of feature normal estimation and subpixel accurate matching on sequences with and without motion blur. [paper][website]

An Integrative Approach to Object Recognition in Visual SLAM

Diploma ThesisJan Funke

With this work we show how object recognition can be integrated in Visual SLAM systems to achieve the following goals:

Add semantic information to the map. We extend the point feature map of the used Visual SLAM system to contain objects. This way, the system does not just track random point features but knows about the relative poses of objects to the camera.

Reduce the map size and therefore the filter update speed. We argue that the tracking of a single object is already sufficient to get a good camera pose estimate. In contrast to using several point features for the same purpose, an object consumes less state space. This is especially beneficial during the update step of the EKF filter.

Reduce the error in the camera pose estimate. Whenever an object was recognised and added to the map it provides a lot of valuable information about the current camera pose.

Relocate the camera whenever the system lost track. The object recognition does not rely on a good state estimate. Whenever the system is lost, a single object pose measurement can be used to relocate the camera.