Confocal bi-protocol : a new strategy for 3D living cell imaging Yong Yu, Alain Trouv´e, Bernard Chalmond 1 CMLA, Ecole Normale Sup´erieure de Cachan, CNRS, UniverSud 61 Avenue Pr´esident Wilson, 94230 Cachan Cedex France

Olivier Renaud, Spencer L. Shorte Institut Pasteur, , Imagopole, Plateforme d’Imagerie Dynamique 25-28 rue du Dr. Roux, Paris 75015, France

Abstract The conventional approach for microscopic 3D cellular imaging is based on axial throughstack image series which has some significant limitations such as anisotropic resolution and axial aberration. To overcome these drawbacks, we have recently introduced an alternative approach based on micro-rotation image series. Unfortunately, this new technique suffers from a huge burden of computation that makes its use quite difficult for current applications. Therefore, to alleviate this difficulty, we propose in this paper a new imaging strategy, called bi-protocol, that consists of coupling micro-rotation acquisition and conventional z-stack acquisition. We experimentally prove that this new approach achieves 3D reconstruction with a similar quality to that of the pure micro-rotation 3D imaging. Besides, it has the advantage to achieve simultaneously z-stack reconstruction and micro-rotation reconstruction, and in this way, it enables to interpret the improvement made by the novel method in comparison to the conventional one. Key words: confocal z-stack, micro-rotation confocal microscopy, non-adherent living cells, 3D volume reconstruction, axial abberation, isotropic resolution, slices-to-volume registration, slices positioning, energy-based registration, kernel-based interpolation, deconvolution

1 Introduction Three-dimensional fluorescence imaging microscopy of individual living cells is an essential tool for cellular biology, pathology and the study of infection and viru1

Correspondance to : Pr. B. Chalmond, [email protected]

Preprint submitted to Journal of Microscopy

6 October 2009

lence therein. However, one critical constraint of established techniques is that samples must be stabilized by attachment to an optically transparent surface, thereby completely precluding their use for non-adherent cell types. The severity of this limitation becomes clear when one considers that for basic- and biomedical-research, cell-based assay and cell-diagnostic applications some of the most important targets are non-adherent cells, for example, stem cells, systemic cancer cells and lymphocytes. Many cell types are functionally non-adherent, and others are certainly in a functionally altered state upon adherent attachment. It is noteworthy that among the 3600 cell lines harbored in the depositories of the American Type Culture Collection, there are at least 1400 cells described as displaying distinct growth properties ”in suspension”, that is to say that they present phenotypes related to nonadherence. We therefore need a new 3D imaging strategy targeted specifically at live, non-adherent cells.

1.1 Materials and characteristics Firstly, our spinning disk confocal microscope is equipped with a di-electrophoretic field cage (from Evotec Technologies, now part of Perkin Elmer company) wherein suspended cells can be trapped and manipulated inside the cage [Schnelle et al., 1993, Shorte et al., 2003]. Once an individual cell has been trapped, we modulate the field of the cage so that the cell undergoes continuous full rotations. During the rotation, a sequence of confocal micro-rotation images (say CMR sequence) are sampled at a given rate. We get around 100 slices per full rotation for a low rate and around 300 slices for a high rate. Each slice is an image taken under the same microscopic conditions. Among the advantages of such an apparatus, is the ability to see non adherent living cells under different views without having to manipulate them. As it has been shown in [Le Saux et al., 2009], it is also the ability to alleviate the problem of anisotropy of the microscope resolution due to the fact that axial resolution is to two, to three times lower than lateral resolution, and that spherical aberrations are present. Secondly, our confocal microscope is equipped with an objective piezo-drive. So, in addition to the CMR sequence, an axial ”through-stack” image series (or ”zstack”) can be recorded from the target sample immobilized in suspension in the cage. The piezo step in through-stack axial imaging was 100nm and the x-y sampling resolution was 127 nm. Note that in the case of an adherent cell immobilized upon a glass slide, z-stack data is the standard approach for microscopic 3D cellular imaging. Nevertheless, it is important to appreciate that conventional through-stack imaging has some significant limitations such as anisotropic resolution and axial aberration that are complex phenomena, and we refer to [Booth et al., 2002, Goldman and Spector, 2005, Rietdorf, 2005, Pawley, 2006, Keller, 2006, Egner and Hell, 2006, Sibarita, 2005, Murray, 2005]. In contrast to conventional axial through-stack imaging, the CMR mode yields three dimensional sampling without recourse to any 2

movement of imaging system optical components, ensuring isotropic resolution by obviating dependence upon the objective’s relatively lower axial resolving power. In effect, the higher lateral resolving power is used to sample in three dimension. In another way, z-resolution can be partly improved by fusioning several z-stacks of the same target sample taken under different point-of-views, in general two different angles. To acquire these z-stacks, the trapped cell is rotated around the xor y-axis and then stopped in a stable position by modulating the field of the cage. Although this approach improves significatively the resolution of the 3D representation [Renaud et al., 2008], it remains lower than that of the CMR reconstruction.

1.2 A new protocol Therefore, 3D CMR imaging remains of interest. However, the relative positions of the slices are required to perform 3D cell volume reconstruction because any interpolation algorithm a priori demands both the signal values and their observation positions. In our case, these positions are unknown and are quite difficult to estimate because of the instability of the rotation movement which evolves around an unknown ”mean” rotation movement (see Fig. 1). In our recent work [Yu et al., 2007, 2008a], we have proposed a novel 3D cell volume reconstruction algorithm in which volume reconstruction and slice position estimation are simultaneously performed. The reconstruction results presented in these previous papers show more details than that from the conventional z-stack volume although our experiments did not include any deconvolution process. More recently, in [Le Saux et al., 2008, 2009], we have also performed deconvolution and we have shown that microrotation imaging yields isotropic high resolution, free from axial elongation. However, this approach suffers from a high burden of computation that makes its use quite difficult for current applications. The algorithmic procedure sequentially alternates slice positioning and volume reconstruction in order to minimize a regularity cost function which defines the quality of the reconstructed volume. This alternate estimation explains why the computation time is so long. So, in the present paper, we propose a faster algorithmic procedure that estimates more directly the positions without updating them as before. The number of iterations N of the previous estimation process is therefore reduced to one iteration in the new procedure, dividing the computation time by a factor of order N (typically N = 10). Toward this goal, we propose to couple z-stack and CMR sequence acquisitions in a new process called bi-protocol. The bi-protocol rests on the following simple idea : since z-stack yields a preliminary estimate of the 3D volume, we can use it to obtain an estimate of the positions, reducing in this way the computational burden of the balance procedure cited above. This estimation problem refers to the so-called slice-to-volume registration problem for which an substantial literature exists [Maintz and Viergever, 1998, Thevenaz et al., 1998, Bogush et al., 2004, 3

Fig. 1. Pictorial representation of the rotation instability.

Joni´c et al., 2005, Georg et al., 2008, Dalvi and Abugharbieh, 2008]. In fact, in our paper we deal with a multiple-slices-to-volume registration. It means that the registration treats the CMR slice alignments as a single set and not as a sequence of independent alignments. As we shall see below, this is a key point to solving efficiently the registration problem, and to overcome its particular difficulties. Our contribution to slice-to-volume registration methodology comes from the specified problem which presents two particular difficulties. First, as stated above, the rotation movement (axis and angular velocity) is unstable and corrupted by erratic small translations. Second, because of the axial aberration of the z-stack imaging, the z-stack volume is quite different from the volume associated with the CMR sequence as it is shown in Fig.5 and Fig.7. However, employing a multiple slices procedure -as opposed to single slice procedure- greatly increases the amount of potential matching and therefore allows the construction of a robust registration procedure. Nevertheless, as in our previous paper, the final reconstruction of our new method is mainly based on CMR images, and consequently, the accuracy of the reconstruction should be the same one as that of the preceding method, from the moment that the positioning is of the same accuracy. Let us note that our CMR registration task cannot be treated by the available methods. Most established image registration tools in biology were designed for the alignment of fixed serial sections and, as such, are only capable of performing rigid transformations between reference and target images, [BitPlane-Autoaligner, StackReg, Thevenaz et al., 1998], although more recent sophisticated developments allow non rigid registration [Trouv´e, 1998, Durrleman et al., 2008, Du and Wasser, 2009]. Our paper is organized as follows. In section 2, we give a general formulation of the concerned problem. In sections 3 and 4, the overall procedure is described in detail. Experiments on real data and artificial data are given in section 5 and the conclusions in section 6. 4

2 Problem formulation Let VZ = (VZ,i)1≤i≤nZ be a conventional z-stack composed of nZ parallel images and I = (Ii )1≤i≤n be a CMR sequence of a given cell (see Fig. 2). In order to collect a complete information of the cell during the micro-rotations, the focal plane is positioned near the center of the cell, a location denoted by z = 0. We also denote by P0 the Cartesian plane positioned at this location. The CMR sequence is thus acquired in this plane. We assume that the central slice VZ,i0 , with i0 =round(NZ /2), lies in P0 . The other positions in VZ are driven by the piezo engine below and above P0 . n denotes the rotation period of the cell. We assume that the captured cell undergoes no self-driven motion during the acquisition that is usually short, around 20 images per second. The final goal is to reconstruct the 3D cell volume, based mainly on the CMR sequence, in order to benefit from the isotropic resolution of the CMR sequence. To deal with this issue, we have first to estimate the position of the slices Ii which have been continuously acquired during the unstable rotation of the cell. This position estimation is done with respect to VZ (Section 3). Given the slice position alignment, 3D volume reconstruction of the cell can be performed with respect to I (Section 4). This reconstruction can be achieved at two levels. At the fine level, an estimate of the true volume f is computed, whereas at the coarse level, we compute an estimate of f˜ = f  Ψ which is a convoluted version of f according to the PSF Ψ of the microscope. Let us emphasize that slice positioning is achieved with respect to f˜ by identifying f˜ with VZ . Slice alignment requires interpolation of the z-stack using a continuous 3D representation, say V˜Z , in order to be able to compare each slice Ii with any cross-section of V˜Z . For the same reason, the unknown volume f˜ is also continuously represented. For both V˜Z and f˜, we have used the Gaussian kernel-based interpolation method which is well suited for irregularly sampled data, such as CMR data. This model was already used in our previous approach [Yu et al., 2008a]. It will be summarized in Section 4. Furthermore, this method benefits from the automatic parameter calibration routine and FFT based convolution routine. The input of the numerical procedure is composed of the z-stack VZ located at a given position and the CMR sequence I, stored as a flat stack, in a plane H0 ˜ the unknown positions of the CMR whose position is also given. In V˜Z (or f), slices are represented by Cartesian planes denoted by (Hi )1≤i≤n . They are defined . by n×6 rigid transformation parameters (rotation and translation pairs) Φ = {Φi = (Ri , bi )}1≤i≤n and are related to a reference plane H0 which is arbitrarily chosen : Hi = Ri H0 + bi .

(1)

Hi is the result of an affine rotation denoted ϕi and associated with the parameters 5

Fig. 2. Pictorial representation of the two different acquisitions composing the bi-protocol setting, and their matching. (a) conventional z-stack, (b) micro-rotation sequence, (c) matching of the two acquisitions : two sets of parameters must be estimated, first the affine rotation which defines the relative position between ω (or Hω ) and P0 , and then the affine rotation of the registration between the z-stack volume and the CMR volume.

Φi :

ϕi (u) = Ri u + bi , ∀u ∈ H0 .

(2)

H0 is the Cartesian plane where the CMR slices are stored. With this notation, any cross-section in f˜ is written : ˜ i u + bi ) , ∀u ∈ H0 . f˜ ◦ ϕi (u) = f(R

(3)

The CMR slices Ii are defined on H0 which is assimilated to a finite grid. The task of recovering Φ and f˜ from the data I is an inverse problem, [Chalmond, 2003] among many others. Classically, such a problem is treated using an energy function J (Φ, f˜|I) whose general expression is ˜ , ˜ = J1 (Φ, f˜) + J2 (I|Φ, f) J (Φ, f|I) J1 (Φ, f˜) = J1,1 (Φ) + J1,2 (f˜) .

(4)

J1 (Φ, f˜) is a prior energy that quantifies the plausibility of (Φ, f˜) with respect to ˜ is a fidelity term that quantifies how (Φ, f˜) some prior knowledge, and J2 (I|Φ, f) 6

fits the data. The solution of this inverse problem is then given by the solution ˆ fˆ˜) that minimizes J (Φ, f˜|I). In our previous work, in that VZ is not given, the (Φ, minimization procedure sequentially alternates slice positioning and volume reconstruction. When VZ is given, f˜ can be identified with V˜Z , and then, the procedure can be reduced to two main steps : • Slice positioning : ˆ = arg min J (Φ, V˜Z |I) Φ Φ

= arg min[J1,1 (Φ) + J2 (I|Φ, V˜Z )] .

(5)

Φ

• Volume reconstruction : ˆ ˆ f˜|I) f˜ = arg min J (Φ, f˜

˜ . ˆ f)] = arg min[J1,2 (f˜) + J2 (I|Φ,

(6)

f˜

In the case of reconstruction at fine level, this step is replaced by ˆ f  Ψ) , fˆ = arg min J2 (I | Φ, f

(7)

where J2 denotes the opposite of the log-likelihood of I given f . This minimization is performed using the EM algorithm. In the case of PSF with spatially-varying orientation, this procedure is described in [Le Saux et al., 2009]. Let us again emphasize that z-stack volume and CMR volume are quite different. The explanation is that, in contrast to CMR acquisition, z-stack data suffers from several serious drawbacks : axial aberration including spherical aberration, poor axial resolution, spatially z-varying point spread function, and photobleaching. Section 3 deals with the slice positioning task (5), and section 4 with the volume reconstruction task (6).

3 Matching procedure 3.1 Slice position alignment Let us assume for a while that we have a rough initial estimate of Φ, say Φ0 . This solution, which will be detailed in the next section, is seen as a mean movement, that is the movement in the absence of perturbations involving the cage controller. The true positions Φ are allowed to move in a limited neighborhood of Φ0 in order to conserve the regularity of the rotation movement. For the prior energy in Eq. 5, 7

as in our previous paper, we adopt a simple expression by modeling the movement offset as white noise : J1,1 (Φ) =

n  −2 i=1

σω dis2 (Ri , Ri0 ) + σb−2 bi − b0i 2 ,

(8)

where the parameters σω and σb control the rotation and translation perturbations. Here the distance between two rotation matrices R1 and R2 is defined as the common geodesic distance on the group of rotations SO(3) : 

dis(R1 , R2 ) = cos

−1

trace(R1 R2 −1 ) − 1 2



.

(9)

Since every image Ii is seen as a noisy version of f˜ ◦ ϕi with Gaussian additive noise, the fidelity term is written : J2 (I|Φ, V˜Z ) = V˜Z ◦ ϕi − Ii 2 =

n  −2 i=1

σf V˜Z ◦ ϕi − Ii 2 ,



|V˜Z (ϕi (u)) − Ii (u)|2 ,

u∈H0

(10)

where σf is a signal-to-noise ratio controlling the contrast in the slices. The minimization in Eq. 5 is computed using a gradient-based algorithm. The Φi ’s being independent, the optimization with respect to Φ is reduced to a component-wise procedure in which the analytical gradient expressions are computed as : Ri J (Φ, V˜Z |I) = σf−2 Ri V˜Z ◦ ϕi − Ii 2 + σω−2 Ri dis2 (Ri , Ri0 )

(11)

where Ri V˜Z ◦ ϕi − Ii 2 =

 u∈H0

(V˜Z (Ri (u) + bi ) − Ii (u)) (Ru ∧ Ri u V˜Z (Ri u + bi )) cos

−1



Ri dis2 (Ri , Ri0 ) = − 

1−

trace(Ri R0i 2



−1

trace(Ri R0i 2

)−1

−1



)−1

Ri0

2

. (12)

The expression of bi J is evident and omitted here. 3.2 Mean rotation movement estimation In this section, we search for a rough initial estimate of Φ, denoted Φ0 . This estimation is crucial. Here, we assume that the cell movement is stable : the cell rotation is then characterized by a unique rotation axis ω, in contrast to the general case (see 8

previous Section) where each micro-rotation has is own axis. However, we have to keep in mind that ω does not necessarily belong to the focal plane P0 where the CMR images are acquired : the relative position between ω and P0 is unknown. Two sets of parameters must be estimated : first, the affine rotation which defines the relative position between ω and P0 , and then the affine rotation of the registration between the z-stack volume and the CMR volume as it is now detailed (see also Fig.2). In the 3D Cartesian space {0, x, y , z }, the z-stack VZ is arbitrarily positioned such that its central slice VZ,i0 with i0 =round(NZ /2) lies in the focal plane P0 = {0, x, y }. Let Fω = {0ω ,i, j, k} be the Cartesian frame associated to the CMR volume and assume that ω is for instance the i-axis within the plane Hω = {0ω ,i, j}. Therefore, the relative position between ω (or Hω ) and the focal plane P0 is written as P0 = (Rfoc Hω + bfoc ) where (Rfoc , bfoc ) denotes an affine rotation. Every micro-rotation is defined by a rotation Rω with axis ω and angle θ = 2π n where the period n is assumed to be known (or estimated as briefly introduced in [Yu et al., 2008b]). The set of all the slice positions are {Rωi−1 (P0 ), 1 ≤ i ≤ n}. Let us note that this expression is equivalent to the dual situation where the cell is fixed and the focal plane is uniformly rotating. Now, we have to deal with a registration problem between two volumes : the z-stack volume V˜Z and the CMR volume generated by the slices {Ii } deployed following the positions {Rωi−1 (P0 )}. An affine rotation φω,z = (Rω,z , bω,z ) is then needed to register these two volumes. Gathering the preceding elements, we obtain Hi , the position of the ith slice Ii in V˜Z : Hi = Rω,z Rωi−1 (P0 ) + bω,z P0 = Rfoc Hω + bfoc .

(13)

The L2 dissimilarity between the positioned CMR slices and the interpolated crosssections in V˜Z is then defined as 0

J (Rω,z , bω,z , Rfoc , bfoc ) =

n  i=1

V˜Z (Hi ) − Ii 2 .

(14)

Minimizing Eq. 14 yields the estimated mean rotation movement : 0 0 , b0ω,z , Rfoc , b0foc ) = (Rω,z

argmin

Rω,z , bω,z , Rfoc , bfoc

J0 .

(15)

The optimization procedure involved in Eq. 15 uses the traditional LevenbergMarquardt approach, an iterative gradient-based algorithm for non-linear leastsquares optimization problem [Gill et al., 1993] . Therefore, we need the gradient 9

expression of V˜Z which is computed straightforwardly : Rfoc V˜Z (ϕi (u)) = Rfoc u ∧ (Rω,z Rωi−1 )t ϕi (u) V˜Z (ϕi (u)) b V˜Z (ϕi (u)) = (Rω,z Ri−1 )t ϕ (u) V˜Z (ϕi (u)) ω

foc

i

Rω,z V˜Z (ϕi (u)) = Rω,z Rωi−1 Rfoc u ∧ ϕi (u) V˜Z (ϕi (u)) b V˜Z (ϕi (u)) = ϕ (u) V˜Z (ϕi (u)) , ω,Z

(16)

i

where a ∧ b denotes the cross product. The gradient computation ϕi (u) V˜Z (ϕi (u)) is directly obtained from an interpolation routine. Slices can be then positioned on a trajectory defined by the estimated mean rotation according to Eq. 13. This trajectory Φ0 is the initial solution of the alignment procedure described in the previous section. We state in the Introduction that z-stack images are quite different from CMR images. So, it is legitime to ask why the z-stack volume can be a reliable reference in alignment of CMR slices? The anisotropy and the axial abberation concern the z-axis. Near P0 , the central images of the z-stack are morphologically like their corresponding CMR slices, whereas far from this reference plane, in the top and the bottom of the z-stack, z-stack volume is quite different from CMR volume. However, positioning CMR slices using the rigid deployed positions {Rωi−1 (P0 )} helps to overcome this difficulty. The iterative optimization process tends to match correctly the central images in order to minimize the L2 cost function J 0 . This matching forces the positioning of the others images because of the rigidity of the deployed positions. In others words, the alignment is mainly based on the central zstack images. This is illustrated on Fig. 12-Line (2) : the xy-cross-section near P0 , which is extracted from the reconstructed volume fˆ0 based on Φ0 , is not severely degraded in contrast to the xz-cross-section.

4 3D volume model The representation of f˜ and f are described respectively in [Yu et al., 2008a] and [Le Saux et al., 2009]. Here, we only recall the model of f˜ since, as we said in Section 2, the same model is used both for f˜ and V˜Z : V˜Z for slice positioning in Eq. 5, and f˜ for volume reconstruction in Eq. 6. ˜ we do not use any more the z-stack but only the CMR For the reconstruction of f, ˆ As aforementioned in the introsequence and the previously estimated positions Φ. duction of Section 2, we have to compute a continuous volume fˆ˜ such that every cross-section fˆ˜ ◦ ϕˆi approximates the slice Ii . The data are highly non organized : many data points are present around the rotation axis whereas they are sparse far from this axis. The reconstruction is based on Eq. 6. 10

In the context of the regularization methods, f˜ is represented by a linear combination of basis functions {K(g, ·), g ∈ G} where G is a regular 3D grid for computational purposes :  ˜ ≡ f˜α (·) = αr K(gr , ·) . (17) f(.) r

Doing so, the volume f˜ is replaced by the unknown set of parameters {αr }. Recalling the expression of J2 (Eq. 10), the energy (Eq. 6) becomes : ˆ f˜α |I) = J1,2 (f˜α ) + σ −2 J (Φ, f

n  i=1

f˜α ◦ ϕˆi − Ii 2 ,

(18)

where σf2 is a regularization parameter. The same expression is used for z-stack representation, that is V˜Z ≡ v˜α . By denoting {VZ,i ; i = 1, ..., nZ } the parallel slices of the z-stack, the continuous z-stack v˜α minimizes the energy (Eq. 18) : vα ) + J (˜ vα |VZ ) = J1,2 (˜

σV−2

nZ  i=1

˜ vα,i − VZ,i 2 .

(19)

Therefore, we have to chose J1,2 and the basis functions {K(g, ·), g ∈ G}. In the case of the classical spline approximation, the K(g, ·)’s are the B-spline functions and J1,2 (f˜α ) quantifies the regularity of f˜α [Wahba, 1990]. More generally, this modelization can be expressed using the reproducing Hilbert space formulation. It is quite technical, an introduction is given in [Wahba, 2000]. In this framework, K(·, ·) is a kernel function modeling the spatial dependence : K(g, g ) = ρ(||(g − g  )||/λf ) , ∀ g, g  ∈ G .

(20)

As in our previous paper [Yu et al., 2007], we have chosen the Gaussian function ρ(u) = exp(−u2 /2). λf is a scale parameter that defines the range of the spatial dependence, as a covariance function does. In this context, f˜α belongs to an Hilbert space H called reproducing kernel Hilbert space and it is known [Wahba, 1990] that the norm of f˜α in this space is : f˜α 2H =

 r,r 

αr αr K(gr , gr ) .

(21)

Here again, this norm quantifies the regularity of f˜α , and as for the spline functions, we set : (22) J1,2 (f˜α ) = f˜α 2H . For irregularly sampled data, the kernel-based model is well adapted but it requires to chose carefully the scale parameter λf of the kernel function. In the experiment presented in the next section, for n = 280 we have λf = 3.2 which corresponds to the greatest distance between two successive planes Hi . In fact, given {Ii }, the parameters λf and σf2 in Eq. 18 are automatically estimated using the maximum likelihood principle described in [Yu et al., 2008a]. 11

In summary, we give a pseudo code of the whole so-called bi-protocol 3D volume reconstruction procedure. Algorithm 1 Bi-protocol 3D volume reconstruction procedure Require: Acquire a z-stack and a CMR sequence, - Reconstruct a continuous volume V˜Z from the z-stack, - Estimate the mean rotation movement (Φ0ω,Z , Φ0foc ), ˆ the result. - Realign the positions Φ for each micro-rotation slices, call Φ ˆ and the CMR sequence. - Reconstruct a continuous volume fˆ or fˆ˜ from Φ return the reconstructed volume.

5 Experiment 5.1 Real data The experiment presented in this section has been performed on a sequence of micro-rotation slices acquired on a micro-rotation confocal imaging system. SW13/20 living cells (human tumor cell lines) tagged with nuclear targeted green fluorescent protein (lamin-A-GFP, a kind gift of Christopher Hutchinson) were manipulated inside a DFC-3 chips (Evotec Technologies/Perkin-Elmer) controlled by a Cytocon400TM’s 4-phase high frequency generator. An individual trapped cell was rotated around the x-y plane and imaged using an Andor Revolution XD spinning disk confocal system equipped with an EM-CCD DV885 camera (Andor Technology) mounted on an inverted microscope (Axiovert 200M, Carl Zeiss). The microscope is equipped with a 63x water immersion Zeiss objective with the numerical aperture (NA) value set as 1.2. Fluorescence acquisition used laser light excitation 488nm and emission band-pass filter 500-550nm. The z-sampling resolution is 100nm and the xy-sampling resolution is 127 nm. This xy-sampling resolution is also that of the CMR slices. The slice size is 120 × 120 and the z-stack height is nZ = 161. More details about image acquisition can be found in [Lizundia et al., 2005, Renaud et al., 2008]. Fig. 3 shows 8 micro-rotation slices extracted from a CMR sequence whose period is n = 280. On the z-stack volume visualized in Fig. 5, the z-axis aberration appears clearly along the z-axis. In Fig. 6, this z-stack has been extended to a continuous volume V˜Z by minimizing the energy given in Eq. 19. The reconstructed CMR volume fˆ˜ is shown in Fig. 7. The parameters used in this experiment were σω = π/100, σb = 4, σf = 166, λf = 3.2. Firstly, comparing to the conventional z-stack volume, we see that the z-axis aberration is greatly 12

reduced. Confocal micro rotation imaging appears to be free from axial aberration. This result is also clear in Fig. 10 which shows the nuclear envelope lamin architecture. Secondly, thanks to the isotropy of the CMR sequence, we see that the resolution in the top and the bottom of the CMR reconstruction is greatly enhanced. However, the xy-resolution (see xy-cross-section at the top-left corner of Fig. 5) is not fully recovered on the CMR reconstruction (see for instance the xy-cross-section at the top-left corner of Fig. 7). This is due to the kernel-based interpolation-smoothing process which has a smoothing effect on the cell structure. Nevertheless, in Fig. 8, the deconvolution process removes this drawback and lets appear many new details (see [Le Saux et al., 2008, 2009] for comments). Finally, Fig. 9 illustrates the severe degradation that an erroneous alignment can involve on the CMR reconstruction. 5.2 Artificial data A second round of experiments is based on pseudo-real data and simulation. The microscopic workbench consists of tools to build synthetic objects which realistically represent biological objects and simulate the microscopic image formation. In fact, the pseudo-real data that represent the true volume, have been created by deconvolution of a set of true biological images acquired from a cell nucleus using a confocal microscope on live cell expressing a fluorescent marker (green fluorescent protein fusion that is spatially expressed only around the nuclear membrane of the cell). The result is shown in Fig 11-a for slices of size 200 × 200. This ”cell” shows several specific elements of different shapes and sizes which are particularly well suited for the validation of methods : fine micro-tubes, invagination, fine membrane, membrane junction, etc, ... Our simulator is a means to test some aspects of the reconstruction method by simulating different kind of movement and degradaˆ on the initial estimate tion. We use it to measure the improvement brought by Φ 0 Φ , and to study the impact of the anisotropy of the PSF on the alignment procedure and thus on the reconstruction results. In this experimental study, the PSF is defined by the Gaussian model P SF (x, y, z) ∝ exp (−

x2 y2 z2 − − ). σx2 σy2 σz2

(23)

with σy = σx and σz =√τ (z)σx , τ (z) being a linear function of |z| such that τ (0) = √ 2 and τ (|zmax |) = 5. Here, we limit the phenomenon of the axial aberration to the PSF anisotropy. Simulating the whole phenomenon would demand a huge research investment [Booth et al., 2002, Goldman and Spector, 2005, Murray, 2005, Sibarita, 2005, Rietdorf, 2005, Pawley, 2006, Keller, 2006, Egner and Hell, 2006], and is out of the scope of our paper. However, the PSF anisotropy implies that the z-stack volume is quite different from the CMR volume although the elongation is less pronounced (see Fig. 11-b). That provides a mean to study the robustness of the alignment technique. Fig. 11-d shows the result fˆ of the whole bi-protocol 13

procedure applied to the pseudo-real cell of Fig. 11-a convolved by the PSF defined in Eq.23. The deconvolution process respects the specific elements of the original cell, as micro-tubes, invagination, and membrane. In this experiment, the size of the slices is 200 × 200 and we have set n = 200, nZ = 200. The cell movement (i.e. the simulated movement) is depicted by the red curve in Fig. 13 and has to be compared to the uniform rotation movement represented by the black circle line. For the slice position alignment, the model parameters are σω = π/200, σb = 10, σf = 217, λf = 2. Two orthogonal cross-sections of the simulated z-stack Fig. 11-b are shown in Fig.12-Line (1). We see again that z-stack has a coarse z-resolution : on the xzcross-section, the top and the bottom of the cell are more degraded than the middle around z = 0. The xy-cross-section that has been acquired in the focal plane P0 benefits from the finest resolution, that is xy-resolution. Since all the slices of the CMR sequence are acquired through the same focal plane P0 , some of them are morphologically like this xy-cross-section. Fig.12- Line (2) displays the corresponding cross-section extracted from the reconstructed volume fˆ0 (see Fig. 11-c) that is based on the mean positions Φ0 . The yz-cross-section is strongly degraded whereas the xy-cross-section is much less affected by this initial coarse estimation. This point illustrates our comment at the end of Section 3 : the optimization process tends to match correctly the central images of the z-stack with their corresponding slices in the CMR sequence. In the two next experiments (Fig. 13-14, and Fig. 15), the slice size has been reduced to 50 × 50 in order to have a more readable drawing of the movement and also to reduce the computation times during the repeated simulations. Fig. 13 shows the trajectory of a point located in the cell frame and submitted to the following movements : true micro-rotations Φ, mean rotation Φ0 and final estimated microˆ The mean deviation between the mean trajectory Φ0 and the final esrotations Φ. ˆ is 1.7 voxel that is high compared to the size n = 50 of the timated trajectory Φ slices. Fig. 14 details these deviations by providing the translation components of the micro-rotations. Let us note that the micro-rotations have been simulated according to an independent white noise. In this context, it is surprising to see how ˆ is able to accurately follow the true trajectory Φ. the estimated trajectory Φ The estimation of Φ0 requires 29s and 5s for each micro-rotation slice positioning. Today, the whole procedure (registration and volume reconstruction) takes 20 minutes, ten times less than our previous approach ([Yu et al., 2007, 2008a]). This procedure is carried out by using a non optimized code which could be efficiently rewritten in order to highly reduce the computing time. The last experiment studies the robustness of our method in terms of the anisotropy of the PSF. In this experiment, we assume that τ does not depend any more on z but has a constant value. Let us note that this case is less favorable for the matching. Indeed, in the non-constant case, the alignment is mainly based on the central z14

stack images (see the end of Section 3). When τ (z) is constant, this property does not hold any more and the complexity of the matching is then increased. √ √ We have experimented with several constant τ values between 2 and 5. The quality of the reconstruction has been measured by the Peak-Signal-to-Noise-Ratio 

P SNR(τ ) = 20 log10

max f f − fˆ(τ ) 



,

(24)

where f denotes the true volume and fˆ(τ ) the reconstructed volume. In fact, for every given τ , we have simulated m = 10 data sets and we have then computed the mean PSNR value. Fig. 15 shows these mean values. The noteworthy fact is these values are stable what provides an element to state the robustness of the method with regard to the anisotropy.

6 Conclusion

In this paper we have proposed an efficient matching between z-stack and microrotation sequence which allows to perform non-adherent living cell volume reconstruction. The technical novelty over our previous work comes from the possibility to estimate a mean rotation movement thanks to the available z-stack. Conventional axial through-stack imaging combined with micro-rotation imaging is a powerful means to interrogate microscopic details in a 3D volume such as a cell. Our method significantly enhances the power of this approach by introducing a means to quantitatively interpolate the information content extrapolated from several viewpoints recorded from the same target. Along these lines an especially attractive aspect of this approach is how quickly and simply acquisition and reconstruction may be performed (under optimal conditions). The image acquisition process and the calculation process can be performed in a few minutes using semi-automated control and desktop computer processing. However, the biologist has to respect with accuracy a pre-defined protocol in order to get reconstructions of hight quality, and in particular during the image acquisition [Chalmond et al., 2008]. In fact, the fluorescence emitted is often observed to decrease substantially with time, a phenomenon referred to as photobleaching. In our protocol, the same cell is exposed repeatedly to the excitation light, first during the micro-rotation acquisition and then to acquire a z-stack. This phenomenon is partly responsible of the absence of membrane at the bottom of the cell in Fig 5. This shows that the photostability problem can have a severe impact on the image quality by decreasing the signal-to-noise ratio. At this level, it is crucial to respect the protocol. 15

7 Acknowledgements This research has been supported by the French Ministry of Research (grant ACINIM FLUTOMY 2003 and Postdoctoral Fellowship 2004) and by the European Commission (FP6 NEST 2005 programme) in consortium AUTOMATION (http: //www.pfid.org/AUTOMATION/home/), and by the ” Conseil de la R´egion Ile-deFrance ” (programme SESAME 2007). SW13/20 cells were kindly provided by Christopher Hutchinson (School of Biological and Biomedical Sciences, Durham University, UK). The pseudo-real data were provided by SVI (Scientific Volume Imaging) within the consortium Automation. We also thanks Christophe Machu and Anne Danckaert from the PFID for their technical support. We would like to thank the referees for their helpful comments, which helped greatly improve this article.

References BitPlane-Autoaligner. Available at http://www.bitplane.com/go/products. A. Bogush, A. Tuzikov, and S. Sheynin. 3D object reconstruction from non-parallel cross-sections. In ICPR04, pages III: 542–545, 2004. M. J. Booth, M. A. A. Neil, R. Juskaitis, and T. Wilson. Adaptive aberration correction in a confocal microscope. Proc. Nat. Acad. Sci., 99:5788–5792, 2002. B. Chalmond. Modeling and inverse problems in image analysis. Springer-Verlag, New-York, 2003. B. Chalmond, A. Trouv´e, Y. Yu, J. Wang, O. Renaud, and S. Shorte. Confocal micro-rotation imaging: a complete system for quantitative 3D confocal microscope fluorescence imaging using novel tomography method, Patent No. PCT/FR2008/001483, 2008. R. Dalvi and R. Abugharbieh. Fast feature based multi slice-to-volume registration using phase congruency. In EMBS, pages 5390–5393, 2008. T. Du and M. Wasser. 3d image stack reconstruction in live cell microscopy of drosophila muscles and its validation. Cytometry A, 75:329–243, 2009. S. Durrleman, X. Pennec, A. Trouv´e, P. Thompson, and N. Ayache. Inferring brain variability from diffeomorphic deformations of currents: an integrative approach. Medical Image Analysis, 12:626–637, 2008. A. Egner and S. W. Hell. Aberrations in confocal and multi-photon fluorescence microscopy induced by refractive index mismatch. In J. B. Pawley, editor, Handbook of biological confocal microscopy, chapter 20, pages 404–413. Springer, 2006. M. Georg, R. Souvenir, A. Hope, and R. Pless. Simultaneous data volume reconstruction and pose estimation from slice samples. In CVPR08, pages 1–6, 2008. P. Gill, W. Murray, and M. Wright. Practical optimization. Academic Press, 1993. R. D. Goldman and D. L. Spector, editors. Live cell imaging. CSHL Press, 2005. 16

S. Joni´c, C. Sorzano, P. Th´evenaz, C. El-Bz, S. D. Carlo, and M. Unser. Splinebased image-to-volume registration for three-dimensional electron microscopy. Ultramicroscopy, (103):303–317, 2005. H. E. Keller. Objective lenses for confocal microscopy. In J. Pawley, editor, Handbook of biological confocal microscopy, chapter 7, pages 145–161. Springer, 2006. B. Le Saux, B. Chalmond, Y. Yu, A. Trouv´e, O. Renaud, and S. Shorte. Deconvolution-interpolation in confocal micro-rotation imaging. In ISBI, pages 685–692, 2008. B. Le Saux, B. Chalmond, Y. Yu, A. Trouv´e, O. Renaud, and S. Shorte. Isotropic high resolution 3D confocal micro-rotation imaging for non-adherent living cells. Journal of Microscopy, 233:404–416, 2009. R. Lizundia, L. Sengmanivong, J. Guergnon, T. Muller, T. Schnelle, G. Langsley, and S. Shorte. Use of micro-rotation imaging to study jnk-mediated cell survival in theileria parva-infected b-lymphocytes. Parasitology, 130:629–635, 2005. J. Maintz and M. Viergever. A survey of medical image registration. 2(1):1–16, 1998. J. Murray. Confocal microscopy, deconvolution and structured illumination mthods. In R. D. Goldman and D. L. Spector, editors, Live cell imaging, chapter 14, pages 239–280. Springer, 2005. J. B. Pawley, editor. Handbook of Biological Confocal Microscopy. Springer, 2006. O. Renaud, J. Vina, Y. Yu, C. Machu, A. Trouv´e, H. Van der Voort, B. Chalmond, and S. Shorte. High-resolution imaging of living cells in flow suspension using axial-tomography: 3D imaging flow cytometry. Biotechnology J., 3(1):53–62, 2008. J. Rietdorf, editor. Microscopy Techniques. Springer, 2005. T. Schnelle, R. Hagedorm, G. Fuhr, S. Fielder, and T. Muller. Three-dimensional electric field traps for manipulation of cells - calculation and experimental verification. Biochemica et Biophysica Acta, 1157:127–140, 1993. S. Shorte, T. Muller, and T. Schnelle. Method and device for three dimensional imaging of suspended micro-objects providing for high-resolution microscopy, European patent, No. PCT/EP2003/011818, 2003. J. B. Sibarita. Deconvolution microscopy. In J. Rietdorf, editor, Microscopy Techniques, pages 201–244. Springer, 2005. StackReg. Available at http://bigwww.epfl.ch/thevenaz/stackreg/. P. Thevenaz, U. E. Ruttimann, and M. Unser. A pyramid approach of subpixel registration based on intensity. IEEE Trans. on Image Processing, 7(1):27–41, 1998. A. Trouv´e. Diffeomorphisms groups and pattern matching in image analysis. Int. J. of Comput. Vision, pages 123–221, 1998. G. Wahba. An introduction to model building with reproducing Hilbert space. Technical Report 1020, University of Wisconsin, 2000. G. Wahba. Spline models for observational data. SIAM, 1990. Y. Yu, A. Trouv´e, and B. Chalmond. A Bayesian 3D volume reconstruction for confocal micro-rotation cell imaging. In A. M. N. Ayache, S. Ourselin, editor, 17

Fig. 3. 8 CMR fluorescence slices Ii extracted from a CMR sequence.

Fig. 4. Cross sections V˜Z ◦ ϕˆi based on the estimated positions corresponding to the slices of Fig. 3.

MICCAI 2007, volume II, pages 685–692. Springer-Verlag Berlin Heidelberg, 2007. Y. Yu, A. Trouv´e, J. Wang, and B. Chalmond. An integrated statistical approach for volume reconstruction from unregistered sequential slices. Inverse Problems, 24:17 pages, 2008a. Y. Yu, J. Trouv´e, and B. Chalmond. A fast 3d volume reconstruction for confocal micro-rotation cell imaging. In International Conference on BioMedical Engineering and Informatics, pages 775–778, 2008b.

18

Fig. 5. Z-stack volume rendering of VZ (thanks to Volume Viewer of ImageJ software, http://rsb.info.nih.gov/ij/). Left column, from the top : xy-, xz- and yz-cross-sections.

Fig. 6. V˜Z , with same viewing position as the z-stack.

19

ˆ Fig. 7. 3D CMR reconstruction f˜ (with same viewing position as the z-stack).

Fig. 8. 3D CMR reconstruction fˆ (with same viewing position as the z-stack).

20

Fig. 9. Effect of a erroneous alignment on the 3D CMR reconstruction fˆ (with same viewing position as the z-stack).

Fig. 10. Comparison of z-stack volume reconstruction and CMR reconstruction under four orthogonal views (thanks to OsiriX software, http://www.osirix-viewer.com/). Line (1) : ˆ z-stack VZ , Line (2) : V˜s , Line (3) : Reconstructed volume f˜.

21

Fig. 11. Artificial data : (a) True volume f , (b) Z-stack volume rendering of VZ , (c) Reconstructed volume fˆ0 for the initial positions Φ0 , (d) CMR reconstruction fˆ.

22

Fig. 12. For the artificial data of Fig.11, the two columns show successively -in inverse grey scale- the vertical xz cross-section and the horizontal xy cross-section for z near 0. Line (1) : from the z-stack data of Fig.11-b, Line (2) : from the reconstructed volume fˆ0 of Fig.11-c.

23

20

z−axis

10 0 −10 −20 20

10

0 −10 −20

19

y−axis

20

21

22

x−axis

Fig. 13. Trajectories of a point located near the surface of the cell, computed from φ0 (black circle line), from the true parameters Φ = {Φi = (Ri , bi )} (continuous red line), ˆ (blue dashed line). The mean deviation between and from the final estimated parameters Φ 0 ˆ is 1.7. Φ -trajectory and Φ-trajectory

24

1

0.5

0

−0.5

−1

−1.5

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

1.5

1

0.5

0

−0.5

−1

−1.5

1.5

1

0.5

0

−0.5

−1

−1.5

Fig. 14. Coordinates -x, -y, -z of the final estimation of the translations {bi } associated to the trajectories in Fig.13 (with the same colors).

25

36

35.5

PSNR(τ)

35

34.5

34

33.5

33

1.5

1.6

1.7

1.8

τ

1.9

2

2.1

2.2

2.3

Fig. 15. PSNR curve with regard to the PSF anisotropy.

26