CS 4330: Computer Vision

Fall 2010 (FY11-1)

Synopsis and Course Goal

This course provides an overview of Computer Vision – enabling computers to understand visual input. The course introduces the major concepts and methods, and gives you a sample of problem settings and techniques for solving them. It is intended for those who want to get an understanding of how computer vision works and how it can perform such diverse tasks as object recognition, robot navigation, and increase situational awareness. The course has strong ties between theory, examples of its application, and practical exercises in order to promote fast and persistent learning.

The course is intended for students with a basic proficiency in linear algebra and a programming language. Example courses at NPS: MA2043 or MA3042

Course Learning Outcomes

What tasks do you need to master in order to achieve the overall course goal? Upon successful completion of this course, you will be able to:

Course Relevance to Computer Science ESRs

The course covers a wide range of ESR subjects as it spans the entire gamut from theory to working application. Students apply mathematical theory and concepts (ESR paragraph 1) as well as important algorithms (ESR paragraph 4) to solve data analysis and data modeling problems. Large data volumes and tight network bandwidth restrictions help develop the students’ systems skills and programming knowledge (ESR paragraphs 2 and 3). Image data handling and storage and video compression teach the students proficiency in core computer science technologies including algorithms, data structures, and software engineering (ESR paragraph 4).

Automated video analysis is of ever-increasing importance to the military, with applications ranging from base protection and surveillance to unmanned system operations and intelligent systems (ESR paragraph 4). In various class projects, students exercise ”innovation, problem solving, systems thinking,” and tangible results ensure the ”real world applicability” (ESR paragraph 5).

Syllabus and Organization

Take a look at the Contents on the following pages to get an idea of the syllabus. Computer vision is a “hot topic” and rapidly evolving, so expect several modifications that reflect recent developments and advances. Every chapter is a learning module with specific learning objectives, reading and study material, exercises, and graded assignments.

Reading assignments are from various textbooks, see :
SZ = Szeliski
BK = Bradski & Kaehler,
SS = Shapiro & Stockman,
BB = Ballard & Brown,
DIP = Gonzalez & Woods (Digital Image Processing),
DIPuM = Gonzalez et al. (Digital Image Processing using Matlab).

The reading report is due before class on the following Monday. For example, the first chapter is to be read in the week of 09/27 and you have to hand in the reading report before class on 10/04. Note that this is a preliminary schedule for the quarter: the most up-to-date and authorative information is on Sakai.







WeekDate ModulesTopic Reading Assignments (due date)












1 09/271, 2 Introduction to Computer Vision, Pixel OperationsSZ1.1, SZ3.1 10/01: Matlab intro






2 10/043, 4 Morphology, Spatial Filtering SZ3.2, SZ3.3 10/05: bg model, histogram, 3x3 average






3 10/114, 5 Spatial Filtering, Color SZ3.4-3.5.3 (Monday is Columbus Day) 10/15: Gaussian, Laplacian, orientation histogram






4 10/18?? Feature detection and matching SZ4 10/19: Object tracking






5 10/256 Segmentation SZ5 10/26: model fitting






6 11/017, 8 Fitting, determistic and probabilistic SZ5 Wed 10/27: midterm






6 11/089 Shapes and Regions, PCA SZ B.1.1 midterm discussion, 11/10: PCA






7 11/15?? Recognition SZ14 11/13: texture-based classification






8 11/22?? Parts-based Recognition SZ14 11/24: recognition






9 11/29?? Alignment SS4






10 12/0614 Capstone unconventional optics12/9-11: Final Exam






Textbook, Web and Data Space

The required textbook for this course is Richard Szeliski: Computer Vision: Algorithms and Applications (see below). We will also work with a reader and several resources provided to you in class. Grades for homework assignments etc. will be on Sakai. Software and other course material can be found here:
http://www.movesinstitute.org/~kolsch/courses/CS4330

Hours

Lectures and lab:
Monday, Tuesday, Wednesday, 0800-0850, GE-B03 IN-250
Friday 1000-1150, GE-117 IN-250
These time slots will be used for lectures and/or lab time. Please bring your laptop for every meeting.

Final:
7-11 Dec, 2010 (exact date TBD)

Holidays:
Mon 11 Oct 2010 (Columbus Day) Thu 11 Nov 2010 (Veterans Day) Thu 25 Nov 2010 (Thanksgiving Day)

Office Hours:

I am available for questions and help whenever I am in my office (WA-279) and have some time. If you would like a firm appointment, please don’t hesitate to set one up by email or phone (656-3402).

Grading

Grades for homework assignments etc. will be on Sakai:
15% programming, lab projects
25% written (theory) homework, quizzes
5% reading reports
15% midterm
15% presentation
25% final

Homework, Late Policy, Teamwork etc.

Homework assignments will be posted by Monday and are due the following Friday at 11:59pm unless otherwise noted. No credit will be given for assignments that are more than 3 days late (Sunday 11:59pm). You can hand in at most one assignment up to 3 days late and still receive full credit. That is: one permitted late assignment for the entire quarter. Additional late assignment will be given a max of 50% credit.

Unless otherwise noted, written assignment can be handed in on a sheet of paper (preferred) or by email. If you email it to me, please make sure your name appears on top of the actual assignment (e.g., in the text file attachment), not just in the email. Programming assignment must be turned in using Sakai file submission or as instructed in class. See the respective assignment for the naming convention.

Collaborative work:

Reading Assignments

There will be weekly reading assignments from the reader and from other reading material. A “reading assignment” requires you to read the material and to write a short, one or two page summary. Demonstrate your understanding of the contents, don’t provide me with a complete recount of the entire text. You are encouraged to add a paragraph in which you critique the reading, note questions that you have or state some other related comment. Print the reports (duplex, please) and bring them to class on the respective due day. If you cannot make it to class, please email them to me prior to class.

Oral Presentations

In case oral presentations will be part of the syllabus (they are currently not planned), they will be evaluated as follows.

Content:
Have you understood the main concepts? Have you learned enough details for a thorough understanding? Can you answer questions well?

Didactics:
Can you explain the high-level view, including possible method applications and limitations? Can you explain critical detail? Is your story and/or presentation well organized or do you jump from here to there and back? Are you making good use of examples? Did you bring your own material and explanations into the presentation, not merely read out my slides?

Delivery:
Do you speak in an appropriate fashior, or too fast, slow, or softly? Are you making eye contact with your audience? Are you relying too much on notes? Are you keeping your audience engaged? If using slides or other screen material, was it legible and not too much text? Are you staying within your allocated time limit?

Framework:
Provide a summary handout that captures the main takehome message in a brief, concise manner. Provide a few references for further information and study of the topic. Devise 5-10 study questions, some general and conceptual, and some detailed and specific, possibly examples. Plan for a 30 minute presentation, plus questions.

Useful links for presentation skills

http://ec.hku.hk/epc/presentation/self_evaluation.htm

Equipment

All special equipment and software will be provided to you, if required:
High-performance laptop, Matlab with Image Processing Toolbox, USB or FireWire (1394) camera, OpenCV

Programming Projects

As part of the course, we will work on small programming projects with the help of a high-level computer vision library (OpenCV or Matlab) to demonstrate some of the methods we have learned in class. Some sample projects are listed below, further ideas and detail can be found in Appendix C:

Resources

Textbooks

Textbooks, more or less in order of relevance

  1. course textbook: Richard Szeliski: Computer Vision: Algorithm and Applications http://szeliski.org/Book/
  2. Gary Bradski and Adrian Kaehler: Learning OpenCV: Computer Vision with the OpenCV Library ISBN: 978-0596516130
  3. res: Linda G. Shapiro, George C. Stockman, Computer Vision (Prentice Hall; 1st edition, 2001), ISBN: 0130307963
  4. res: D. Ballard and C. Brown: Computer Vision (Prentice Hall, 1982)
  5. res: Digital Image Processing (3rd Edition) by Rafael C. Gonzalez, Richard E. Woods Publisher: Prentice Hall ISBN: 978-0131687288
  6. res: Digital Image Processing Using MATLAB by Rafael C. Gonzalez, Richard E. Woods, Steven L. Eddins Hardcover: 782 pages Publisher: Prentice Hall; 1st edition (September 5, 2003) ISBN: 0130085197
  7. D. A. Forsyth, J. Ponce; Computer Vision: A Modern Approach (Prentice Hall), ISBN: 0130851981
    pre-print available here: http://decsai.ugr.es/mia/complementario/t1/book3chaps.html
    Since I was asked: there are no color pictures in this book.
  8. R. Hartley and A. Zisserman: Multiple View Geometry second edition, Cambridge University Proess. ISBN 0521540518
  9. Robot Vision by Berthold K. P. Horn Hardcover: 480 pages Publisher: The MIT Press; Mit Press edition (March 13, 1986) ISBN: 0262081598 co-published with McGrawHill
    out of print
  10. Computer and Robot Vision by R. Haralick, and L. Shapiro Addison-Wesley, 1992
    two-volume book is tons of material, good references
  11. A Guided Tour of Computer Vision by Vishvjit S. Nalwa Addison Wesley, 1993
    great for concepts
  12. Machine Vision by R. Jain, R. Kasturi, and B. G. Schunck McGraw-Hill, 1995
  13. Machine Vision: Theory, Algorithms, Practicalities by E. R. Davies Academic Press, 1997
    chapter 2 is at
    http://www.palantir.swarthmore.edu/maxwell/papers/html/Maxwell-WUEIC-1997.htm
  14. Practical Computer Vision using C by J. R. Parker 1994
  15. J.E. W. Mayhew, J.P. Frisby; 3D Model Recognition From Stereoscopic Cues MIT Press, 1991, ISBN 0-262-13243-5 online at
    http://homepages.inf.ed.ac.uk/rbf/BOOKS/MAYHEW/
  16. M. A. Fischler, O. Firschein Intelligence: The Eye, the Brain and the Computer Addison-Wesley, 1987, ISBN: 0201120011 online at
    http://homepages.inf.ed.ac.uk/rbf/BOOKS/FISCHLER/
  17. a collection of online (and offline) computer vision/image analysis/imaging books at:
    http://homepages.inf.ed.ac.uk/rbf/CVonline/books.htm

Miscellaneous

Contents

Introduction to the Course
 Synopsis and Course Goal
 Course Learning Outcomes
 Syllabus and Organization
 Textbook, Web and Data Space
 Hours
 Grading
 Homework, Late Policy, Teamwork etc.
 Reading Assignments
 Oral Presentations
 Equipment
 Programming Projects
 Resources
Contents
0 Administrativa
 0.1 Reading and Content
 0.2 Assignments
 0.3 Instructor Notes
  0.3.1 TODO before course
  0.3.2 First Lecture
1 Introduction to Computer Vision
 1.1 Reading and Content
 1.2 Assignments
  1.2.1 FY10-3, Spring 2010
  1.2.2 FY10-1, Fall 2009
 1.3 Related and Useful Courses
 1.4 The History of Vision
 1.5 Neat Computer Vision Links
 1.6 New York City’s Ring of Steel
 1.7 Instructor Notes
 1.8 Shapiro & Stockman: Introduction (Chapter 1)
2 Pixel Operations and Histograms
 2.1 Reading and Content
 2.2 Assignments
  2.2.1 FY10-3, Spring 2010
  2.2.2 FY10-1, Fall 2009
  2.2.3 FY09-1, Fall 2008
  2.2.4 FY08-1, Fall 2007
  2.2.5 FY07-1, Fall 2006
 2.3 Instructor Notes
3 Morphology
 3.1 Reading and Content
 3.2 Assignments
  3.2.1 FY09-1, Fall 2008
  3.2.2 FY07-1, Fall 2006
 3.3 Instructor Notes
 3.4 Shapiro & Stockman: Binary Image Analysis (Chapter 3)
4 Spatial Filtering
 4.1 Reading and Content
 4.2 Assignments
  4.2.1 FY10-3, Spring 2010
  4.2.2 FY10-1, Fall 2009
  4.2.3 FY09-1, Fall 2008
  4.2.4 FY08-1, Fall 2007
  4.2.5 FY07-1, Fall 2006
  4.2.6 FY06-1, Fall 2005
 4.3 Instructor Notes
 4.4 Gonzales et al.: Spatial Filtering (Chapter 3.4)
5 Color
 5.1 Reading and Content
 5.2 Assignments
  5.2.1 FY10-3, Spring 2010
  5.2.2 FY10-1, Fall 2009
  5.2.3 FY09-1, Fall 2008, and prior
 5.3 Instructor Notes
 5.4 Gonzalez & Woods: Color Image Processing (Chapter 6)
6 Image Segmentation
 6.1 Reading and Content
 6.2 Group Exercise
 6.3 Assignments
  6.3.1 FY08-1, Fall 2007
  6.3.2 FY07-1, Fall 2006
  6.3.3 FY06-1, Fall 2005
 6.4 Instructor Notes
 6.5 Shapiro & Stockman: Image Segmentation (Chapter 10)
 6.6 Gonzalez et al. DIPuM: Color Segmentation (Chapter 6)
 6.7 Shapiro & Stockman: Region Properties (Chapter 3.6)
 6.8 Gonzalez Matlab: Color Segmentation (regionprops)
7 Fitting
 7.1 Reading and Content
  7.1.1 Range Data Fitting
 7.2 Assignments
  7.2.1 Lab - do not submit
  7.2.2 Theory - submission required
  7.2.3 FY10-1, Fall 2009
  7.2.4 FY09-1, Fall 2008
 7.3 Instructor Notes
8 Probabilistic Segmentation and Model Fitting
 8.1 Reading and Content
 8.2 Assignments
  8.2.1 FY09-1, Fall 2008
  8.2.2 FY08-1, Fall 2007
 8.3 Probabilistic Segmentation
  8.3.1 Probability Basics
  8.3.2 Likelihood, its Estimation and Mixture Models
  8.3.3 Example: Image Segmentation
  8.3.4 Mean-Shift
 8.4 Instructor Notes
  8.4.1 EM Example
9 Shapes and Regions
 9.1 Reading and Content
 9.2 Assignments
 9.3 Principal Component Analysis
 9.4 Instructor Notes
10 Texture
 10.1 Reading and Content
 10.2 Assignments
  10.2.1 FY08-1, Fall 2007
  10.2.2 FY07-1, Fall 2006
 10.3 Instructor Notes
11 Camera Transformations
 11.1 Reading and Content
  11.1.1 Transformation Classes and Properties
 11.2 Assignments
  11.2.1 FY07-1, Fall 2006
 11.3 Advanced Notes
 11.4 Instructor Notes
 11.5 Hartley & Zisserman: Multiple View Geometry (Chapter 1)
 11.6 Hartley & Zisserman: Estimation (Chapter 4, various)
12 Classification and Learning
 12.1 Reading and Content
 12.2 Assignments
  12.2.1 FY09-1, Fall 2009
 12.3 Classification Basics
  12.3.1 ROC curve
 12.4 Support Vector Machines
 12.5 Instructor Notes
13 Image Sequences and Motion
 13.1 Reading and Content
 13.2 Assignments
  13.2.1 FY09 Fall 2008
 13.3 Instructor Notes
14 Capstone
 14.1 Reading and Content
 14.2 Instructor Notes
 14.3 Nayar: Computational Cameras
15 TODO
A Matlab
 A.1 Random Tips
 A.2 Suggested Reading
B OpenCV
C Project 1: Background Modeling
D Project 2: Chroma-Keying in OpenCV
E Project 3: Camera Location Estimation, Homographies
F Project 4: Object Classification in Underwater Video
G Project 5: Kiosk-style human-computer interaction with computer vision
Bibliography