CS 664 Slides #7 Visual Motion. Prof. Dan Huttenlocher Fall 2003

Relevanta dokument
12.6 Heat equation, Wave equation

Isometries of the plane

Solutions to exam in SF1811 Optimization, June 3, 2014

1. Compute the following matrix: (2 p) 2. Compute the determinant of the following matrix: (2 p)

8 < x 1 + x 2 x 3 = 1, x 1 +2x 2 + x 4 = 0, x 1 +2x 3 + x 4 = 2. x 1 2x 12 1A är inverterbar, och bestäm i så fall dess invers.

Pre-Test 1: M0030M - Linear Algebra.

Module 1: Functions, Limits, Continuity

Tentamen i Matematik 2: M0030M.

F ξ (x) = f(y, x)dydx = 1. We say that a random variable ξ has a distribution F (x), if. F (x) =

Kurskod: TAMS28 MATEMATISK STATISTIK Provkod: TEN1 05 June 2017, 14:00-18:00. English Version

(D1.1) 1. (3p) Bestäm ekvationer i ett xyz-koordinatsystem för planet som innehåller punkterna

denna del en poäng. 1. (Dugga 1.1) och v = (a) Beräkna u (2u 2u v) om u = . (1p) och som är parallell

Tentamen i Matematik 3: M0031M.

This exam consists of four problems. The maximum sum of points is 20. The marks 3, 4 and 5 require a minimum

Viktig information för transmittrar med option /A1 Gold-Plated Diaphragm

Graphs (chapter 14) 1

and Mathematical Statistics Gerold Jäger 9:00-15:00 T Compute the following matrix

Support Manual HoistLocatel Electronic Locks

Algoritmer och Komplexitet ht 08. Övning 6. NP-problem

LUNDS TEKNISKA HÖGSKOLA Institutionen för Elektro- och Informationsteknik

Rastercell. Digital Rastrering. AM & FM Raster. Rastercell. AM & FM Raster. Sasan Gooran (VT 2007) Rastrering. Rastercell. Konventionellt, AM

Preschool Kindergarten

and u = och x + y z 2w = 3 (a) Finn alla lösningar till ekvationssystemet

Tentamen i Matematik 2: M0030M.

Högskolan i Skövde (SK, JS) Svensk version Tentamen i matematik

Swedish adaptation of ISO TC 211 Quality principles. Erik Stenborg

1. Varje bevissteg ska motiveras formellt (informella bevis ger 0 poang)

NP-fullständighetsbevis

Module 4 Applications of differentiation

Sammanfattning hydraulik

Hur fattar samhället beslut när forskarna är oeniga?

FORTA M315. Installation. 218 mm.

Second handbook of research on mathematics teaching and learning (NCTM)

Kurskod: TAMS11 Provkod: TENB 28 August 2014, 08:00-12:00. English Version

Stad + Data = Makt. Kart/GIS-dag SamGIS Skåne 6 december 2017

Styrteknik: Binära tal, talsystem och koder D3:1

Chapter 2: Random Variables

Rep MEK föreläsning 2

Grafisk teknik IMCDP IMCDP IMCDP. IMCDP(filter) Sasan Gooran (HT 2006) Assumptions:

English Version. Number of sold cakes Number of days

Grafisk teknik IMCDP. Sasan Gooran (HT 2006) Assumptions:

Calculate check digits according to the modulus-11 method

Ett hållbart boende A sustainable living. Mikael Hassel. Handledare/ Supervisor. Examiner. Katarina Lundeberg/Fredric Benesch

Tentamen MMG610 Diskret Matematik, GU

A study of the performance

Problem som kan uppkomma vid registrering av ansökan

Kurskod: TAMS11 Provkod: TENB 07 April 2015, 14:00-18:00. English Version

Michael Q. Jones & Matt B. Pedersen University of Nevada Las Vegas

Grafisk teknik. Sasan Gooran (HT 2006)

SVENSK STANDARD SS-EN ISO 19108:2005/AC:2015

Adding active and blended learning to an introductory mechanics course

2(x + 1) x f(x) = 3. Find the area of the surface generated by rotating the curve. y = x 3, 0 x 1,

Module 6: Integrals and applications

Kurskod: TAMS24 / Provkod: TEN (8:00-12:00) English Version


Writing with context. Att skriva med sammanhang

och v = 1 och vektorn Svar 11x 7y + z 2 = 0 Enligt uppgiftens information kan vi ta vektorerna 3x + 2y + 2z = 1 y z = 1 6x + 6y + 2z = 4

Statistical Quality Control Statistisk kvalitetsstyrning. 7,5 högskolepoäng. Ladok code: 41T05A, Name: Personal number:

Kurskod: TAIU06 MATEMATISK STATISTIK Provkod: TENA 15 August 2016, 8:00-12:00. English Version

Resultat av den utökade första planeringsövningen inför RRC september 2005

Webbregistrering pa kurs och termin

KTH MMK JH TENTAMEN I HYDRAULIK OCH PNEUMATIK allmän kurs kl

Mapping sequence reads & Calling variants

Authentication Context QC Statement. Stefan Santesson, 3xA Security AB

Kvalitetsarbete I Landstinget i Kalmar län. 24 oktober 2007 Eva Arvidsson

6 th Grade English October 6-10, 2014

Övning 5 ETS052 Datorkommuniktion Routing och Networking

Isolda Purchase - EDI

Image quality Technical/physical aspects

< THE SHELF SYSTEM FILLED WITH POSSIBILITIES. Design Anne Krook

Tentamen del 2 SF1511, , kl , Numeriska metoder och grundläggande programmering

STORSEMINARIET 3. Amplitud. frekvens. frekvens uppgift 9.4 (cylindriskt rör)

NO NEWS ON MATRIX MULTIPLICATION. Manuel Kauers Institute for Algebra JKU

Kursplan MD2022. Matematik III 30 högskolepoäng, Grundnivå 2

Vad kännetecknar en god klass. Vad kännetecknar en god klass. F12 Nested & Inner Classes

2.1 Installation of driver using Internet Installation of driver from disk... 3

Bridging the gap - state-of-the-art testing research, Explanea, and why you should care

Exempel på uppgifter från 2010, 2011 och 2012 års ämnesprov i matematik för årskurs 3. Engelsk version

Make a speech. How to make the perfect speech. söndag 6 oktober 13

Accomodations at Anfasteröd Gårdsvik, Ljungskile

Beijer Electronics AB 2000, MA00336A,

the standard scalar product, i.e. L E 4. Find the orthogonal projection of the vector w = (2, 1, 2, 1) on the orthogonal complement L of L (where

Gradientbaserad Optimering,

Mönster. Ulf Cederling Växjö University Slide 1

Measuring child participation in immunization registries: two national surveys, 2001

Bernoullis ekvation Rörelsemängdsekvationen Energiekvation applikationer Rörströmning Friktionskoefficient, Moody s diagram Pumpsystem.

Examensarbete i matematik på grundnivå med inriktning mot optimeringslära och systemteori

English Version. 1 x 4x 3 dx = 0.8. = P (N(0, 1) < 3.47) = =

A QUEST FOR MISSING PULSARS

Schenker Privpak AB Telefon VAT Nr. SE Schenker ABs ansvarsbestämmelser, identiska med Box 905 Faxnr Säte: Borås

How to format the different elements of a page in the CMS :

ALGEBRA I SEMESTER 1 EXAM ITEM SPECIFICATION SHEET & KEY

Discovering!!!!! Swedish ÅÄÖ. EPISODE 6 Norrlänningar and numbers Misi.se

Kursplan. AB1029 Introduktion till Professionell kommunikation - mer än bara samtal. 7,5 högskolepoäng, Grundnivå 1

Flervariabel Analys för Civilingenjörsutbildning i datateknik

7.5 Experiment with a single factor having more than two levels

MMA129 Linear Algebra academic year 2015/16

Rättningstiden är i normalfall 15 arbetsdagar, annars är det detta datum som gäller:

Consumer attitudes regarding durability and labelling

1. Find an equation for the line λ which is orthogonal to the plane

Transkript:

CS 664 Slides #7 Visual Motion Prof. Dan Huttenlocher Fall 2003

Visual Motion Over sequence of images can determine which pixels move where Differs from motion in the world Camera motion Pan, tilt, zoom Motion parallax Information about depth from camera motion Scene motion Reveals independent objects and behaviors Un-detectable motion No/low intensity variation 2

Some Uses of Visual Motion Human-machine interaction Animation, gestures, facial expressions Surveillance and monitoring Tracking and analyzing behaviors Collision detection and avoidance Camera stabilization Remove jitter Autonomous navigation Path finding and depth from parallax Constructing panoramic mosaics 3

Motion Analysis in Video Video insertion Compute motion in one image sequence Use to transform frames of another sequence and superimpose Today used to insert signs and markings into sporting events Panoramic mosaics Synthesized views from video sequence 4

Estimating Visual Motion Historically two different approaches Direct methods, based on local image derivatives at each pixel Feature based methods, sparse correspondence We will focus on direct methods Used most in practice Recover image motion from spatio-temporal variations in brightness Dense estimates but can be sensitive to variations in appearance 5

Direct Motion Estimation Methods Based on the following assumptions Every pixel in image I goes to some location in subsequent image J Overall brightness of images I,J does not change (much) Called brightness constancy equation I(x,y) J(x+u(x,y), y+v(x,y)) 1 2 3 4 5 6 7 8 0 0 0 0 1 1 1 1 5 6 7 8 1 2 3 4 0 0 0 0-1 -1-1 -1 9 10 11 12 12 11 10 9 3 1-1 -3 0 0 0 0 15 16 I 13 14 13 14 J 15 16 2 2 u -2-2 0 0 v 0 0 6

Using Brightness Constancy Minimization formulation Seek (u(x,y),v(x,y)) minimizing error (I(x,y)-J(x+u(x,y),y+v(x,y)) 2 Not practical to search explicitly! Linearization Relate motion to image derivatives Gradient constraint Assuming small u,v (on order of a pixel) First order term of Taylor series expansion of brightness constancy 7

Gradient Constraint One-dimensional example linearization Estimate displacement d using derivative Two functions f(x) and g(x)=f(x-d) Taylor series expansion f(x-d) = f(x) d f (x) + E Where f denotes derivative Now write difference as f(x)-g(x) = d f (x) + E Neglecting higher order terms δ = (f(x)-g(x))/f (x) Note only for small d f g x f g x 8

Gradient Constraint (or Optical Flow Constraint) Same approach extends naturally to 2D I(x,y) J(x+u,y+v), u=u(x,y), v=v(x,y) Assume time-varying image intensity well approximated by first order Taylor series J(x+u,y+v) I(x,y)+I x (x,y) u+i y (x,y) v+i t Substituting I x (x,y) u+i y (x,y) v -I t Using gradient notation I (u,v) -I t Linear constraint on motion (u,v) at each pixel Can only estimate motion in gradient direction 9

Aperture Problem (Normal Flow) Can only measure motion in direction normal to edge (along gradient) 10

Aperture Problem (Normal Flow) Gradient constraint defines line in (u,v) space I (u,v) -I t Methods based solely on per pixel estimates don t work well v u 11

Combining Local Constraints Each pixel defines linear constraint on possible (u,v) displacement For set of pixels with same displacement combine constraints to get estimate For pixels with different displacements, somehow identify that is case v v u u 12

Translational Motion Assume single displacement (u,v) for all pixels within some region of image Over-constrained system of linear equations I x (x,y) u+i y (x,y) v=-i t Find least squares solution In matrix form: min z Dz - t where D = I x (x 1,y 1 ) I y (x 1,y 1 ) I x (x n,y n ) I y (x n,y n ) and t = [I t (x 1,y 1 ) I t (x n,y n )] T 13

Least Squares Solution z * = (D T D) -1 D T t Method of normal equations, can derive from setting partial derivatives to zero Σ I x 2 Σ I x I y Σ I x I y D T D = D T t = Σ I y 2 Inverse of 2x2 closed form a b A = A -1 = 1/(ad-bc) c d Σ I x I t Σ I y I t d-b -c a Where det(a)=ad-bc not (near) zero 14

Translational Motion Can estimate small translation over local patch around each pixel Fast using box sums Note relation to corner detection Poor estimate if A nearly singular Also poor if patch contains more than one underlying motion Better handling of multiple motions Robust statistical techniques Handling larger translations Pyramid method 15

Multiple Motions Robust statistical techniques for finding predominant motion in a region Consider approach of iteratively reweighted least squares (IRLS) As illustration of robust methods Generalize minimization problem to min z W(Dz t) Weight matrix W is diagonal Lessen importance of pixels that don t match Iterate to find good weights Note in unweighted case W is identity matrix 16

Finding Predominant Motion Minimization generalizes in obvious way z * = (D T W 2 D) -1 D T W 2 t Determining good weights to use Start by computing least squares solution, z 0 Iteratively compute better solutions Compute error for each pixel based on previous solution z k-1 and use that to set weight per pixel Depends on initial solution being good enough to allow bad pixels to have largest error Have to measure error based on image intensity matches, it s the only thing we can measure 17

Updating Weights To solve for z k given z k-1 Create weights W k =diag(w 1k w nk ) where w ik = 1 if r i k-1 c c/r i k-1 otherwise Where r i k-1 is measure of error at i-th pixel with motion estimate from iteration k-1 Compare i-th pixel value to matching pixel of other image (using z k-1 for correspondence) And c is set based on robust measure of good versus bad data, such as median Common value is 1/.6745 median(r i k-1 ) 18

Weights Example 8 6 5 7 4 5 5 4 4 z k-1 8 6 11 7 5 10 4 3 10 I r i k-1 : 0,0,1,0,1,1,6,5,6 J median = 1 c 1.48 w i k : 1,1,1,1,1,1,.24,.29,.24 19

Global Motion Estimation Estimate motion vectors that are parameterized over some region Each vector fits some low-order model of how vectors change Affine motion model is commonly used u(x,y) = a 1 +a 2 x+a 3 y v(x,y) = a 4 + a 5 x +a 6 y Substituting into grad. constr. equation I x (a 1 +a 2 x+a 3 y) + I y (a 4 + a 5 x +a 6 y) -I t Each pixel provides a linear constraint in six unknowns 20

Affine Transformations Consider points (x,y) in plane rather than vectors for the moment Linear transformation and translation x = a 1 +a 2 x+a 3 y y = a 4 + a 5 x +a 6 y In matrix form A(z)=Lz+b x y = a 2 a 3 x a 5 a 6 y + a 1 a 4 Maps any triangle to any triangle Defined by three corresponding pairs of points 21

Why Affine Transformations Simple (and often inaccurate) model of projection Point (x,y,z) in space maps to (x,y) in image Orthographic or parallel projection Somewhat reasonable model for telephoto lens Yields affine transformation of plane for viewing flat objects 3D rotation, translation followed by orthographic projection and scaling 22

Affine Motion Estimation Minimization problem become that of estimating the parameters a 1, a 6 Rather than just two parameters u,v Still (over-constrained) linear system but in more unknowns Again use least squares to solve Separable into two independent 3 variable problems a 1, a 2, a 3 reflect only u-component of motion a 4, a 5, a 6 reflect only v-component of motion 23

Affine Motion Equations Again compute (D T D) -1 D T t Or (re)weighted version for IRLS Now two 3x3 problems, one for I x and one for I y, as opposed to single 2x2 problem Problem for I x and u motion (I y analogous) T remains same, D changes I x1 x 1 I x1 y 1 I x1 D = I xn x n I xn y n I xn 24

Multiple (Layered) Motions Combining global parametric motion estimation with robust estimation Calculate predominant parameterized motion over entire image (e.g., affine) Corresponds to largest planar surface in scene under orthographic projection If doesn t occupy majority of pixels robust estimator will probably fail to recover its motion Outlier pixels (low weights in IRLS) are not part of this surface Recursively try estimating their motion If no good estimate, then remain outliers 25

Other Global Motion Models The affine model is simple but not that accurate in some imaging situations For instance pinhole rather than parallel camera model for closer objects Non-planar surfaces Explicit modeling of motion parallax Projective planar case x = (h 1 +h 2 x+h 3 y)/(h 7 +h 8 x+h 9 y) y = (h 4 +h 5 x+h 6 y)/(h 7 +h 8 x+h 9 y) and u=x -x, v=y -y 3D models such as residual planar parallax 26

Handling Larger Motions Methods based on image gradients are restricted to small displacements Two different approaches Abandon gradient method and explicitly search over possible translations Computationally expensive to do for every pixel Consider shifts and products of image patch Block motion provides estimates just for certain pixels, used in compression (e.g., MPEG) Pyramid to guarantee small motions At top level small motion At each level small deviation from one above 27

Coarse to Fine Motion Estimation Estimate residual motion at each level of Gaussian pyramid 1/2 k res I k,j k ½ res I 1,J 1 Original I 0,J 0 Pyramid of image I Pyramid of image J 28

Coarse to Fine Estimation Compute M k, estimate of motion at level k Can be local motion estimate (u k,v k ) Vector field with motion of patch at each pixel Can be global motion estimate Parametric model (e.g., affine) of dominant motion for entire image Choose max k such that motion about one pixel Apply M k at level k-1 and estimate remaining motion at that level, iterate Local estimates: shift I k by 2(u k,v k ) Global estimates: apply inverse transform to J k-1 29

Global Motion Coarse to Fine Compute transformation Tk mapping pixels of I k to J k Warp image J k-1 using T k Apply inverse of T k Double resolution of T k (translations double) Compute transformation T k-1 mapping pixels of I k to warped J k-1 Estimate of residual motion at this level Total estimate of motion at this level is composition of T k-1 and resolution doubled T k In case of translation just add them 30

Affine Mosaic Example Coarse-to-fine affine motion Pan tilt camera sweeping repeatedly over scene Moving objects removed from background Outliers in motion estimate, use other scans 31

SSD An alternative to gradient based methods is template matching Treat a rectangle around each pixel as a template to find best match in other image Search over possible translations minimizing some error criterion (or maximizing quality) Generally use sum squared difference (SSD) ΣΣ(I(x,y)-J(x+u,y+v)) 2 Sometimes compute cross correlation Compute over local neighborhood 32