(Advanced Topics in) Machine Learning

PSU CS 410/510
OGI CSEE 559/659
Advanced Topics in Machine Learning (PSU)
Machine Learning (OGI)
Spring Quarter 2005


Time : Tuesdays and Thursdays, 10:00-11:50am
Location: Shattuck Hall (1914 SW Park Ave) (SH), Room 207.

Instructors:
Todd Leen
Bronson Creek Building 150E (at OGI , #16 on the map), 503-748-1160, tleen-AT-cse.ogi.edu.
Office hours: Tuesday and Thursday, 1:00-2:00pm, or by appointment.

Melanie Mitchell
Fourth Avenue Building 120-24 (at PSU), (503) 725-2412, mm-AT-cs.pdx.edu.
Office hours: Tuesdays, 12:00-1:30pm, Thursdays, 2:00-3:30pm, or by appointment.

Teaching Assistants: To be announced.

Course Website: :
http://www.cs.pdx.edu/~mm/machine-learning-spring-2005/index.html

Prerequisites: PSU CS 410/510 "Machine Learning" or equivalent, or permission of the instructors. Undergraduate level courses in calculus, linear algebra, and probability and statistics. Facility in at least one high-level programming language.

Course objectives: : Introduce students to several areas of machine learning from an advanced standpoint, including neural networks, reinforcement learning, Bayesian methods, computational learning theory, support vector machines, and ensemble learning methods. Give students knowledge of and experience with current theoretical tools for designing, analyzing, and evaluating these machine learning methods. Provide students with experience in reading research papers. Give students experience with performing simulations, doing statistical data analysis of the results, and writing up the results of these experiments.

Readings: There will be no textbook. The assigned readings will consist of papers on the web, handouts, or readings on reserve in the library.

Reserve Readings: The following books will be on reserve in the library. Students may be assigned supplementary readings in these books. We will be adding books to this list during the quarter.

  1. T. M. Mitchell, Machine Learning.

Assignments: There will be six homework assignments, consisting of written problems and computer-based projects. All assignments are due at the beginning of class on the due date specified. For assignments turned in late without prior permission, 10% will be taken off for each day late after the due date.

Exams: There will be one midterm exam and one final exam. Both will be in-class closed book exams.

Grading: Homework: 40%, Midterm: 30%, Final: 30%.

Academic integrity: PSU students will be responsible for following the PSU Student Conduct Code, and in particular, the policy concerning academic honesty. OGI students will be responsible for following the OGI guidelines for academic integrity.

Collaboration policy: Students may discuss the general concepts and principles behind an assignment with other students. In fact, they are encouraged to do this whenever possible, because it is often a valuable way to reinforce ideas, and to learn new perspectives. However, in doing assignments, each student is expected to develop, write up, and hand in an individual solution and, in doing so, develop a sufficient understanding of the problem and solution so as to be able to explain it adequately to the instructor. Under no circumstances should a student copy or consult the solution of another student, or copy a solution from any other source, including the Internet.

Cheating will result in a grade of zero on the assignment or exam on which the student cheats and the initiation of disciplinary action at the university level.

Students with disabilities: If you are a student with a disability in need of academic accommodations, you should register with Disability Services for Students at PSU and notify the instructor immediately to arrange for support services.

Syllabus (subject to change):

Date

Topic

Readings and Homework

Tuesday March 29

Class overview;
Neural networks I

Lecturer: Mitchell

Reading: Tom Mitchell, Machine Learning, Chapter 4, pp. 81-117
(Soon to be on electronic reserve at PSU and OGI).

Thursday March 31

Neural networks II

Lecturers: Mitchell and Leen

...

Tuesday April 5

Neural networks III

Lecturer: Leen

...

Thursday April 7

Neural networks IV

Lecturer: Leen

Here are all Todd Leen's lecture notes on neural networks:
Artificial Neural Networks 2
Artificial Neural Networks 3
Artificial Neural Networks 4
Artificial Neural Networks Addendum
Statistics Review
Estimator Bias and Variance

Here is the first homework (in MS word format), due Thursday April 14.

Tuesday April 12

On-line learning

Lecturer: Leen

...

Thursday April 14

Comp. learning theory I

Lecturer: Mitchell

Reading: M. Anthony and N. Biggs, PAC Learning and Artificial Neural Networks

Tuesday April 19

Comp. learning theory II

Lecturer: Mitchell

"Part 2" (computational learning theory) due Tuesday April 26.

"Part 1" (neural network simulations) due Thursday May 5.

Thursday April 21

Support vector machines I

Lecturer: Mitchell

Reading: M. A. Hearst et al. Support Vector Machines. IEEE Intelligent Systems, 18-28, July/August 1998.
Optional reading C. J. C. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2 (2), 121-167, 1998.

Tuesday April 26

Support vector machines II

Lecturer: Mitchell

Review for midterm

...

Thursday April 28

Midterm

...

Tuesday May 3

SVMs continued

Lecturer: Mitchell

Kernel regression and regularization

Lecturer: Leen

...

Thursday May 5

Bayesian inference I

Lecturer: Leen

...

Tuesday May 10

Bayesian inference II

Lecturer: Leen

Homework 3 assigned. Due Tuesday May 17.

Here is the training data .
Here is the test data .

Thursday May 12

Optimal Bayes classifier
Naive Bayes classifier


Lecturer: Mitchell

...

Tuesday May 17

Mixture models and clustering

Lecturer: Leen

Homework 4 assigned. Due Tuesday May 24.

Here are the data files for Homework 4:
PhonemeDat.txt
PhonemeDev.dat
PhonemeTest.dat

Thursday May 19

EM algorithm and k-means

Lecturer: Leen

...

Tuesday May 24

Reinforcement learning 1

Lecturer: Mitchell

Assigned reading: M. E. Harmon and S. S. Harmon, Reinforcement Learning: A Tutorial

Thursday May 26

Reinforcement learning 2

Guest Lecturer: John Moody

Homework 5 assigned. Due Friday June 3.

Tuesday May 31

Ensemble learning methods

Lecturer: Mitchell

Assigned reading:

L. Breiman, Bagging predictors, Sections 1-4.

R. E. Schapire, The boosting approach to machine learning, Sections 1-4 and 9.

Optional supplementary reading:

E. Bauer and R. Kohavi, An empirical comparison of voting classification algorithms: Bagging, boosting, and variants .

P. Domingos, Why does bagging work? A Bayesian account and its implications.

R. E. Schapire and Y. Singer, Improving boosting algorithms using confidence-rated predictions

Thursday June 2

Catch-up / review session

Lecturers: Leen and Mitchell

Review sheet #1 (MM) for final exam.

Tuesday June 7

10:15-12:05 Final exam

...