PeAR WPI

Welcome to RBE/CS549 course at WPI. The official name of the course is Computer Vision but a more apt title would be Classical and Deep Learning Approaches for Geometric Computer Vision since it is more descriptive of what the course has to offer. This course is taught by Prof. Nitin J. Sanket. Please sign up for Piazza if you haven't done so already.

All the class announcements will be made through Piazza. Please use Piazza to contact TAs. Please do NOT contact the TA's or the Instructor via email unless it's an emergency, and do NOT contact the TA's on any social media platform such as Facebook or WhatsApp (please respect their privacy) regarding course content. If you want to have a chat about our research, feel free to reach out to Prof. Sanket after class.

All the student reports will be released publicly online to enable better learning experience for others. We will be also announce top three submissions for each announcement on the wesbite. All the projects and homeworks are to be submitted using Canvas. If you find any errors/typo in the course Github website, please edit the markdown '.md' file and send a 'pull request'. If you don't know how to use pull request, please check out this tutorial, alternatively post on Piazza.

The previous course materials of this course can be found here and expect this offering to be of a similar content and difficulty.

Homework 0 (HW0) has been released on Jan 06, 2025 and is due on Jan 16, 2025. The knowledge from HW0 is a pre-requisite for the class.

What is this course about?

This is an advanced course on graduate computer vision and computational photography. This course will explore image formation, image features, image segmentation, image stitching, image recognition, motion estimation, 3D point clouds and will touch upon basics of augmented reality. This course is designed to showcase how to model classical 3D geometry problems using Deep Learning.

Team

Prof. Nitin J. Sanket

Instructor
he/him/his

Manoj Velmurugan

Teaching Assistant
he/him/his

Aabha Tamhankar

Teaching Assistant
she/her/hers

Pre-requisities

Programming proficiency in Python 3, experience with Linux and completion of HW0 is a hard pre-requisite. We assume that the students are proficient with basic Linear Algebra, calculus and machine learning to the level of an undergraduate class.

Office Hours

TA: Manoj Velmurugan: Tuesdays and Thursdays 3:30PM to 5:30PM, UH247.
TA: Aabha Tamhankar: : Mondays and Wednesdays 9:00AM to 11:00AM, UH247.
Prof. Nitin J. Sanket: ONLY for serious issues with Appointment, UH250E.

Expectations

This course is fast-paced and has high-expectations of time and effort commitment from students. The students are expected to complete assignments on time since the concepts build on top of each other and it is very easy to fall behind in this course. Please understand the concepts well, ask questions, use Piazza heavily and come to office hours. Help your peers out, but do not plagiarize. The course content (lectures) might sometime run slower than the assignments, you are expected to do your research from the materials given (or new materials from the books and online) to solve the assignments. This course gives a taste of cutting-edge research by design and there will be multiple ways to solve each assignment. The course will progress to become more and more open-ended as we go along.

Software Environment

We will use Python 3 as the programming platform throughout this course along with packages from OpenCV, PyTorch, Numpy, Scikit and Matplotlib. We officially support the server/cluster only. Feel free to use VirtualMachine if you have a Mac/Windows system. For most assignments, there will be a starter code released in PyTorch.

Assignments

All the projects are to be done in groups of TBD whereas homeworks should be done INDIVIDUALLY. However, we encourage you to discuss with your peers and not cheat. The Midterm exams are to be done INDIVIDUALLY and NO discussion is allowed. For clarifications, post a private post on Piazza. For further details, read the Collaboration Policy and Honor Code below.

Assignment Name With Link	Deadline	Grade Percentage	Student Outputs
HW0: Alohomora	Saturday, Jan 16, 2025 at 11:59:59 PM (Individual Submissions)	5	TBD
P1: My AutoPano	Phase 1: Monday, Jan 27, 2025 at 11:59:59 PM (Group Submissions) Phase 2: Friday, Feb 07, 2025 at 11:59:59 PM (Group Submissions)	12	TBD
HW1: AutoCalib	Saturday, Feb 15 2025 at 11:59:59 PM (Individual Submissions)	5	TBD
P2: Building Built In Minutes: SfM and NeRF	Phase 1: Saturday, Mar 01, 2025 at 11:59:59 PM (Group Submissions) Phase 2: Tuesday, Mar 18, 2025 at 11:59:59 PM (Group Submissions)	20	TBD
Midterm Exam 1 (On Canvas)	Thursday, Mar 20, 2025 at 11:59:59 PM (Individual Submissions)	10	NA
P3: Einstein Vision	Phase 1: Sunday, Mar 30, 2025 at 11:59:59 PM (Group Submissions) Phase 2: Tuesday, Apr 08, 2025 at 11:59:59 PM (Group Submissions) Phase 3 (and Extra Credit): Tuesday, Apr 15, 2025 at 11:59:59 PM (Group Submissions)	16	TBD
P4: Deep and Un-Deep VIO	Phase 1: Sunday, Apr 27, 2025 at 11:59:59 PM (Group Submissions) Phase 2: Sunday, May 04, 2025 at 11:59:59 PM (Group Submissions)	18	TBD
Midterm Exam 2 (On Canvas)	Wednesday, May 07, 2024 at 4:59:59 PM (Individual Submissions)	14	NA

Class Slides

Class Number	Class Date	Slides/Resources	Class Topic(s)
1	Jan 15 2025	Slides, Video:	Introduction and Logistics
2	Jan 16 2025	Slides, Video:	What is a Robot? And Interpreting Color
3	Jan 23 2025	Slides, Video:	More Color And Image Filtering (Space)
4	Jan 27 2025	Slides, Video:	Image Filters in Frequency Domain And Features
5	Jan 30 2025	Slides, Video:	Image Features and Descriptors
6	Feb 03 2025	Slides, Video:	Advanced Features
7	Feb 06 2025	Slides, Video:	Image Stitching And Deep Feature Matching
8	Feb 10 2025	Slides, Video:	Camera Models
9	Feb 13 2025	Slides, Video:	Depth From Monocular Single View
10	Feb 17 2025	Slides, Video:	Epipolar Geometry and Two-View Stereo
11	Feb 20 2025	Slides, Video:	Triangulation and PnP
12	Feb 24 2025		Depth From Moving Monocular Camera (Bundle Adjustment) + Factor Graphs And TagSLAM
13	Mar 03 2025		Neural Rendering
14	Mar 06 2025		More NeRFs And Depth From Stereo
15	Mar 17 2025		Deep Learning and Neural Networks
16	Mar 20 2025		NN Tuning, GANs, Bayesian NNs
17	Mar 24 2025		Image Segmentation And Instance Segmentation
18	Mar 27 2025		Object Detection
19	Apr 01 2025		Learned Monocular Depth And Visual Inertial Odometry Basics
20	Apr 03 2025		Visual Inertial Odometry Details
21	Apr 07 2025		Optical Flow and Velocities
22	Apr 10 2025		LIDAR Depth Fusion
23	Apr 14 2025		Learned Stereo Depth And Flow
24	Apr 17 2025		Event Cameras
25	Apr 24 2025		Motion Segmentation
26	Apr 28 2025		Depth From Focus And Defocus
27	May 01 2025		Fancy Deep Learning
28	May 05 2025		Summary, Recap and Conclusions

Syllabus

Singular Value Decomposition, Linear Least Squares, Random Sample Consensus, Image Acquisition, Color Spaces, Gaussians, Color Segmentation, Expectation Maximization, Gaussian Mixture Model, Convolution, Filtering in images, Corner and Edge Detection, Projective Geometry, Camera Model, Feature Matching, Homography, Classifiers, Optical Flow, Fundamental and Essential Matrix, Epipolar Geometry, Perspective-n-points, Bundle Adjustment, Fiducial markers, Factor graphs, Pose Graph Optimization, Visual Odometry, Structure from Motion, RGB-D reconstruction, perceptron, backpropagation, convolutional neural networks, deep learning.

Piazza and ELMS

If you haven't done so already, register yourself on Piazza with your wpi email. We'll be using Piazza for all announcements and discussions. Please use Piazza to contact the instructor/TAs (Feel free to use private posts on Piazza to contact me/TAs). Please do NOT contact the instructor/TAs via email unless it's an emergency, and do NOT contact the instructor/TAs on any social media platform such as Facebook or WhatsApp (please respect our privacy) regarding course content. If you want to have a chat about research, chat with Prof. Sanket after class.

All the assignments will be released on this website page.

All the assignments are to be submitted using Canvas. If you find any errors/typo in the course Github website, please edit the required markdown '.md' file and send a 'pull request'. If you don't know how to use pull request, please check out this tutorial.

Submission Policy

The submissions are made through Canvas (unless otherwise specified) with the name HWk_DirID.zip (for homeworks, Pk_DirID.zip for projects) (where k is the assignment number, for e.g., for homework 0, this would be HW0_DirID.pdf). Here, DirID is your directory ID, i.e., the first part of your wpi email address. For e.g., if your wpi email address is ABCD@wpi.edu, then your DirID is ABCD. Keep your submissions professional, grammatically correct without spelling mistakes. Do not use slangs and chat shorthands on your submissions. You'll get 25% grade penalty for not following the submission guidelines.

Late Submission Policy

This course moves quickly, and concepts will build on top of each other. Therefore it's very important to keep up with the material. To encourage this, late assignments are penalized for 25% per day after the due date. But life is unpredictable; we all need a break sometimes. So, we allow you seven late days, to spend on projects ONLY excluding the final project. You cannot use late days for the midterm exams or the homeworks or the final project. You may submit an assignment late (after the due date) using a late day without any penalty. Think of a late day as pushing the deadline back by a day. So, to get full credit on a 2-days-late assignment, you'd need to use two late days. Late days can only be spent as full days (i.e., you can't use only half a late day for an assignment you submit 12 hrs late). If you are using a late day, mention it in the title of your submission as "USING X LATE DAY(S)" and post a comment on Canvas about the usage of a late day. We expect you to keep track of number of late days you have remaining and notify us of usage of one or more late days. We will default for penalty if we don't see the mention of late day usage as a comment on Canvas on that particular assignment. Again, If you are using a late day, mention it in the title of your submission as "USING X LATE DAY(S)". Also, there will be NO extensions provided to any of the assignments. You can use late days for any phase of the project (except the last project).

Collaboration Policy and Honor Code

Collaboration is encouraged, but one should know the difference between collaboration and cheating. Cheating is prohibited and will carry serious consequences. Cheating may be defined as using or attempting to use unauthorized assistance, material, or study aids in academic work or examinations. Some examples of cheating are: collaborating on an take-home exam or homework unless explicitly allowed; copying homework; handing in someone else's work as your own; and plagiarism. You are welcome to collaborate with your peers on Piazza and in person. However it's important that the work you submit is an expression of your understanding, and not merely something you copied from a peer. So, we place strict limits on collaboration: Firstly, you must clearly cite your collaborators by name at the top of your report. This includes Piazza posts reference. You may not share or copy each other's code. You can discuss how your code works, and the concepts it implements, but you can't just show someone your code. You may use free and publicly available sources, such as books, journal and conference publications, and web pages, as research material for your answers. (You will not lose points for using external sources.) You may not use any service that involves payment, and you must clearly and explicitly cite all outside sources and materials that you made use of. We consider the use of uncited external sources as portraying someone else's work as your own, and as such it is a violation of the University's policies on academic dishonesty. Instances will be dealt with harshly and typically result in a failing course grade. Unless otherwise specified, you should assume that that the WPI Code of Academic Integrity applies. Unless otherwise specified, you should assume that that the WPI Code of Academic Integrity applies.

Reference Books

All concepts will be covered in class lecture, and in the lecture notes. However, we also recommend the following books as good references:

Multiple view geometry in computer vision, Cambridge university press, Richard Hartley, and Andrew Zisserman, 2003.
Computer Vision: Algorithms and Applications, Springer, Richard Szeliski, 2010.
Computer vision: A Modern Approach, Prentice Hall, David Forsyth, and Jean Ponce, 2002.
Digital Image Processing, Prentice Hall, Rafael Gonzalez, and Richard Woods, 2008.
Deep Learning Book, MIT Press, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, 2016.

Furthermore, refer to the slides from courses in the acknowledgement section.

Acknowledgements

This course is developed by being inspired by adapting some of the best parts of each of the courses at multiple universities and these resources are linked below and you are encouraged to look at their content to learn from them as well. The goal of this course is to be the best computer vision and robotics perception course in the world.