CS7870: Algorithms for Machine Learning (UNDER CONSTRUCTION)

Course Staff

Huy L. Nguyen (WVH 358)

Office hour: TBD

Class Time: Monday & Thursday 11:45 am – 1:25 pm

Class Room: Ryder Hall 454

Discussion Forum: Piazza

Overview

This seminar course focuses on the design of efficient algorithms for building modern machine learning models at scale. We will aim to cover topics such as adaptive gradient descent algorithms, dimensionality reduction techniques, algorithms for nearest neighbor search and retrieval augmented generation, and algorithms for training and fine-tuning foundational models. The course will emphasize recent algorithmic developments for state of the art deep learning models and highlight directions for future research.

Grading

Students are expected to do some homework (25%), present a research paper (25%), and complete a research project (50%).

Paper presentation

Students are expected to present a foundational paper or a recent research paper related to topics discussed in the course. For longer papers it is possible to have a team of two working on the same paper. Here are some suggestions to start. More papers will be added depending on the students' interests and serendipitous discovery.

• Attention and state space models
  • Attention Is All You Need
  • DeepSeek-V3 Technical Report
  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• Retrieval augmented generation
  • Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • ParlayANN: Scalable and Deterministic Parallel Graph-Based Approximate Nearest Neighbor Search Algorithms
  • RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search
• Adaptive optimization algorithms
  • Adam: A Method for Stochastic Optimization
  • Muon: An optimizer for hidden layers in neural networks
• Locality sensitive hashing and kernel density estimation
  • Space and Time Efficient Kernel Density Estimation in High Dimensions
  • HyperAttention: Long-context Attention in Near-Linear Time
• Quantization
  • QLoRA: Efficient Finetuning of Quantized LLMs
  • PolarQuant: Quantizing KV Caches with Polar Transformation

LaTeX

You should prepare your homework solutions using LaTeX. LaTeX is a scientific document preparation system; most CS technical publications are prepared using this tool. Great editors exist on most platforms. Some recomend TexShop for Mac. TeXstudio is a good cross-platform editor.

The not so short introduction to Latex is a good reference to get you started.