My Data Science Notes

02 Optimization Algorithms

My Data Science Notes

Welcome!
01 The Basics
01 The Basics
02 Math
02 Math
- 01 Probability
  01 Probability
  - 01 Basic Probability Theory
    01 Basic Probability Theory
    
    Basic Probability: Definitions and Terminology
    
    Types of Probability
    
    Combinatorics: Counting Techniques
    
    Conditional Probability
    
    Independence of Events
    
    Bayes' Theorem
  - 02 Random Variables
    02 Random Variables
    
    Random Variables: Definition
    
    Probability Mass Function (PMF)
    
    Probability Density Function (PDF)
    
    Cumulative Distribution Function (CDF)
    
    Joint, Marginal, and Conditional Distributions
    
    Moment Generating Functions (MGF)
  - 03 Common Probability Distributions
    03 Common Probability Distributions
    
    Overview of Common Probability Distributions
    
    Bernoulli Distribution
    
    Binomial Distribution
    
    Poisson Distribution
    
    Uniform Distribution (Continuous)
    
    Normal (Gaussian) Distribution
    
    Exponential Distribution
    
    Geometric Distribution
    
    Negative Binomial Distribution
    
    Gamma Distribution
    
    Beta Distribution
  - 04 Expectation Variance Covariance
    04 Expectation Variance Covariance
    
    Expected Value (Mean of a Random Variable)
    
    Variance and Standard Deviation
    
    Covariance and Correlation
  - 05 Important Theorems
    05 Important Theorems
    
    Law of Large Numbers (LLN)
    
    Central Limit Theorem (CLT)
- 02 Inferential statistics
  02 Inferential statistics
  - 001 Inferential Statistics
  - 002 Degrees of Freedom
  - 003 Central Limit Theorem
  - Hypothesis testing
  - Statistics
  - Topics
  - 01 The Basics
    01 The Basics
    
    Distributions
    Distributions
    
    Guassian Distribution
    
    Measures of central tendency
    Measures of central tendency
    
    Central Tendency
    
    Mean
    
    Median
    
    Mode
    
    Measures of variability
    Measures of variability
    
    Measures of variability
    
    Standard Deviation
    
    Variance
  - Estimation
    Estimation
    
    Standard error
  - Tradeoffs
    Tradeoffs
    
    Bias
    
    Tradeoffs
    
    Variance
  - Maps
    Maps
    
    Drawing 2023 02 09 22.08.03.excalidraw
    
    Anova.excalidraw
    
    Hyp testing map.excalidraw
    
    T test.excalidraw
    
    Two 18.42.09.excalidraw
  - Parametric tests
    Parametric tests
    
    Parametric Tests
    
    01 t test
    01 t test
    
    Implementation
    
    T test
    
    02 ANOVA
    02 ANOVA
    
    ANOVA
    
    Assumptions
    
    Mechanism
- 03 Linear Algebra
  03 Linear Algebra
  - 01 Core Objects
    01 Core Objects
    
    Scalars
    
    Vectors
    
    Matrices
    
    Tensors
  - 02 Basic Operations
    02 Basic Operations
    
    Vector Operations
    
    Matrix Operations
  - 03 Matrix Properties and Concepts
    03 Matrix Properties and Concepts
    
    Identity Matrix
    
    Matrix Inverse
    
    Determinant
    
    Matrix Trace
  - 04 Vector Spaces and Concepts
    04 Vector Spaces and Concepts
    
    Linear Independence
    
    Span and Basis
    
    Matrix Rank
  - 05 Norms
    05 Norms
    
    Vector and Matrix Norms: Definition
    
    L₁ Norm (Manhattan Norm)
    
    L₂ Norm (Euclidean Norm)
    
    Frobenius Norm
  - 06 Decompositions and Factorizations
    06 Decompositions and Factorizations
    
    Eigenvalues and Eigenvectors
    
    Singular Value Decomposition (SVD)
    
    Overview of Matrix Decompositions
- 04 Calculus
  04 Calculus
  - 01 Foundations
    01 Foundations
    
    Functions, Limits, and Continuity
    
    Derivatives
    
    Rules of Differentiation
    
    Chain Rule
  - 02 Multivariable Calculus
    02 Multivariable Calculus
    
    Functions of Multiple Variables
    
    Partial Derivatives
    
    Gradient
    
    Directional Derivatives
  - 03 Optimization
    03 Optimization
    
    Maxima and Minima (Optimization Basics)
    
    Gradient Descent
    
    Gradient Descent Variants (SGD, Mini-batch)
    
    Convexity in Optimization
  - 04 Advanced Topics
    04 Advanced Topics
    
    Hessian Matrix (Placeholder)
    
    Jacobian Matrix (Placeholder)
    
    Lagrange Multipliers (Placeholder)
  - 05 Integrals
    05 Integrals
    
    Definite and Indefinite Integrals (Placeholder)
    
    Applications in Probability (Placeholder)
  - 06 Matrix Calculus
    06 Matrix Calculus
    
    Matrix Calculus Essentials for ML
- 05 Information Theory
  05 Information Theory
03 Data Collection
03 Data Collection
- Data Types
  Data Types
  - Categorical (Qualitative data)
  - Numeric (Quantitative data)
- Populations & samples
  Populations & samples
  - Population & Sample
  - Sampling Methods
04 Feature Engineering
04 Feature Engineering
- Encoding
  Encoding
  - Encoding
05 Data Visualization
05 Data Visualization
06 Machine Learning
06 Machine Learning
- 01 The Basics
  01 The Basics
  - 01 Model Selection
    01 Model Selection
    
    Akaike Information Criterion
    
    Model Selection
    
    Occam’s razor principle
    
    Ensemble
    Ensemble
    
    Ensemble
    
    01 Bagging
    01 Bagging
    
    Bagging
    
    Random Forest Classifier
    
    02 Boosting
    02 Boosting
    
    Boosting
    
    Gradient Boosting
    
    XGBoost
    
    03 Stacking
    03 Stacking
    
    Stacking
  - 02 Model Evaluation
    02 Model Evaluation
    
    ML model
    
    Model Evaluation
    
    Issues
    Issues
    
    Overfitting
    
    Underfitting
    
    Test metrics
    Test metrics
    
    Good fit
    
    Performance Metrics
    Performance Metrics
    
    Performance Metrics
    
    Classification Model Accuracy
    Classification Model Accuracy
    
    Confusion Metrics
    
    F1 Score
    
    Precision
    
    Recall
    
    Regularization
    Regularization
    
    Regularization
    
    Lasso Regression
    Lasso Regression
    
    Implementation
    
    Lasso Regression
    
    Ridge Regression
    Ridge Regression
    
    Implementation
    
    Ridge Regression
  - 03 Parameter Tuning
    03 Parameter Tuning
    
    01 Optimizers Intro
    
    Bayesian Optimization
    
    Ridge Regularization
  - Resampling Method
    Resampling Method
    
    K fold cross validation
- 02 Supervised
  02 Supervised
  - Supervised
  - 01 Regression
    01 Regression
    
    Regression
    
    01 Simple Linear Regression
    01 Simple Linear Regression
    
    Implementation
    
    Linear Regression
    
    Assumptions
    Assumptions
    
    01 Linear relationship
    
    02 Multivariate normality
    
    03 No or little multicollinearity
    
    04 No auto correlation
    
    05 Homoscedasticity
    
    05 Polynomial Regression
    05 Polynomial Regression
    
    Implementation
    
    Polynomial Regression
    
    OTHER
    OTHER
    
    06 Bayesian Linear Regression
    06 Bayesian Linear Regression
    
    Bayesian Linear Regression
    
    Implementation
  - 02 Classification
    02 Classification
    
    Classification
    
    01 Decision Tree Classifier
    01 Decision Tree Classifier
    
    Decision Tree
    
    Implementation
    
    02 Logistic Regression
    02 Logistic Regression
    
    Entropy
    
    Implementation
    
    Logistic Regression vs Linear Regression
    
    Logistic Regression
- 03 Unsupervised
  03 Unsupervised
  - Unsupervised
  - 01 Clustering
    01 Clustering
    
    Clustering
    
    01 Exclusive
    01 Exclusive
    
    Exclusive Clustering
    
    Implementation
    
    02 Overlapping
    02 Overlapping
    
    Implementation
    
    Overlapping Clustering
    
    03 Hierarchical
    03 Hierarchical
    
    Hierarchical Clustering
    
    Implementation
    
    04 Probabilistic
    04 Probabilistic
    
    Implementation
    
    Probabilistic Clustering
  - 02 Dimensionality Reduction
    02 Dimensionality Reduction
    
    Dimensionality Reduction
    
    Implementation
  - 03 Association Rule
    03 Association Rule
    
    Association Rules
    
    Implementation
  - 04 Autoencoder
    04 Autoencoder
    
    Autoencoder
    
    Implementation
- AB Testing
  AB Testing
  - Intro
  - Metric Formation Framework
- Causal Inference Methods
  Causal Inference Methods
  - Intro
07 Deep Learning
07 Deep Learning
- 01 Foundational Models
  01 Foundational Models
  - Neural Networks
    Neural Networks
    
    01 The Basics
    01 The Basics
    
    01 Introduction
    
    02 Math
    
    03 Batches, Layers & Objects
    
    04 Hidden Layer Activation Functions
    
    05 Softmax Activation
    
    02 CNN
    02 CNN
    
    Implimentation incomplete
    
    05 Transformer
    05 Transformer
    
    Transformer Networks
    
    06 GANs
    06 GANs
    
    Generative Adversarial Networks (GAN)
    
    07 VAE
    07 VAE
    
    Autoencoders and Variational Autoencoders (VAE)
- 02 Advanced Techniques
  02 Advanced Techniques
  - 01 Data Preparation
    01 Data Preparation
    
    Data Augmentation
    
    Normalization
    
    Preprocessing Pipelines
  - 02 Training Techniques
    02 Training Techniques
    
    01 Hyperparameter Tuning
    
    02 Optimization Algorithms
    
    03 Regularization Methods
    
    04 Transfer Learning
  - 03 Model Evaluation & Validation
    03 Model Evaluation & Validation
    
    01 Cross Validation
    
    02 Metrics
    
    03 Confusion Matrix
    
    04 ROC Curves
  - 04 Special Topics
    04 Special Topics
    
    01 Attention Mechanisms
    
    02 Self Supervised Learning
    
    03 Meta Learning
    
    04 Few Shot Learning
    
    05 Chaining
    
    06 Prompt Engineering
    
    07 Parameter Efficient Fine Tuning (PEFT)
    
    08 Retrieval Augmented Generation (RAG)
    
    09 Autonomous Agents
- 03 Frameworks and Libraries
  03 Frameworks and Libraries
  - JAX
  - Keras
  - PyTorch
  - Tensor
  - TensorFlow
- 04 Practical Applications
  04 Practical Applications
  - Deployment
    Deployment
    
    01 Model Serving
    
    02 API Development
    
    03 MLOps
    
    Implementation
  - Industry Use Cases
    Industry Use Cases
    
    Autonomous Vehicles
    
    Finance
    
    Healthcare
    
    Retail
  - Fine tune 101
    Fine tune 101
    
    Untitled
08 Big Data Technologies
08 Big Data Technologies
09 Ethics and Bias
09 Ethics and Bias

02 Optimization Algorithms