June 6, 2018
0 comments
Article
Data Science, Deep Learning, Machine Learning, Mathematics and Research

It is always a hassle for students and machine learning engineers to choose the best machine learning algorithm for their data model. As I mentioned in earlier posts, I prefer to follow **Occam’s razor** which states that *“simplest algorithm is the best algorithm”* and **No free lunch theorem**, which states that *“there is not a single machine learning algorithm which works best for all problems”*. Thus our aim is to find out the ** “simplest model which performs the best” **. In order achieve simplest and best models, we need to understand all the machine learning algorithms and their purpose in depth.

In this series of tutorial article, I am going to explain all the ins and outs of the most famous machine learning algorithms. I will explain the mathematics, include python codes and show the results. I will also suggest what algorithms are useful when specific real time problems are given.

Before all, let us develop our basics of linear algebra, probability and statistics needed for machine learning.

Linear algebra is one of the pillars of machine learning. Before dealing further, let us highlight some of the reasons that you need linear algebra for:

- Linear algebra is the absolute key to understanding calculus and statistics used in machine learning.
- Linear algebra provides greater intuition on how and when the machine learning algorithms perform better.
- Linear algebra provides a deeper understanding of the existing machine learning algorithms and their constraints. This in turn allows us to customize their applications, and help us to understand the impact of hyper-parameters tuning on the results.
- Linear algebra helps us to model and design own own algorithms when needed.

Let us take an example: This is an Asian approach for arranged marriage. You want to get married and your parents have looked many boys/girls for you. You have to make a model to choose the one who best fits for you. What approaches will you take?

- First approach will be to list the qualities that you think are most important in a boy or girl you are looking for such as beauty, qualifications, smartness, jobs, family value, loving, caring etc. All these qualities are called ‘features’ in data science term. Let us consider a linear model as follows, which is a very simplistic model obtained by adding all the features:
- We make a system to represent each of the above features using certain numbers. For example we can give scores between 1 to 10 for each of these qualities and represent this score as a vector as:
If we add them up, we get

- But we dont want to give equal priority to all the features. For some, the features such as beauty is of high priority, while for others love or job can be of higher priority. Now, we want to design a model by setting priorities to each of the qualities. In mathematical terms, we call each priorities as ‘weights’ assigned to each qualities. Thus, the weighted model is :
- Further, we can represent each of these weights as a vector. We assign each weight, the values between 0 and 1 such that the lowest priority qualities are given lower values and the highest priorities are given high values. Assume following are the weight vectors:
Then weighted spouse model is

- Likewise, for other persons, the weights can be different for the same qualities. In terms of linear algebra, we can simply represent this model as:
where spouse1Quals and weights are the row vectors.

- When there are many spouses, each with different values of qualities, we can represent all of them at once as a matrix where a column represents all the qualities of a spouse. If we assign same weights to the qualities of different spouses, we can model this problem as a matrix vector multiplication as follows:
Assume that you have to chose between 4 girls who have following values for qualities:

and the weighted spouse model can be expressed as :

Hence we choose the third spouse, who has the best score.

We can understand linear algebra only if we understand the geometric intuition behind it. Algebraically, Linear algebra solves a number of linear equations by expressing them in different matrix and vector forms. But geometrically, it provides a lot of insight on what is exactly happening when we are solving a problem. We can visualize the geometric meaning of vectors and matrices and the operations we perform in them. We can perform multiple **operations in vectors** as well as in matrix. **Vector addition**, **subtraction, dot product, cross product, norm, angles between vectors, multiplying a vector by a scalar, vector calculus** are some common vector operations, while **Matrix addition, subtraction, multiplication, multiplying a matrix by a vector, multiplying a matrix by a scalar, trace of a matrix, determinant of a matrix, eigendecomposition of a matrix, singular value decomposition (SVD) of a matrix, matrix calculus **are some common matrix operations.

Computationally, we represent vectors as one dimensional array, while matrices are the arrays in two dimensions. The arrays in n-dimensions are called **tensors** for n>2. Though it is geometrically hard to visualize multi-dimensional arrays, we can represent them well algebraically and perform number of operations on them such as addition, subtraction, multiplication and others.

There are many resources in the web for learning linear algebra. In the later blogs, I will definitely cover more about matrix operations such as eigen decomposition and singular value decomposition, with regards to dimensionality reduction algorithms such as principal component analysis (PCA) but for now

I strongly recommend you to learn from some of these video lectures to understand more about linear algebra. All of the instructors and videos are great and interesting. You just need some passion to watch them, understand and play around with problems.

1.** Linear Algebra – Gilbert Strang (of the ubiquitous textbook)**: This is an undergraduate course from MIT by the great Gilbert Strang. I like his style of teaching and explaining things. It will provide you deeper insights on linear algebra and explains the mathematical techniques. Watch all if you have enough time and passion.

2.** Linear Algebra – Foundations to Frontiers** : These are short videos, well explained techniques and easy to understand.

3. **Lecture series – Essence of Linear Algebra from 3Blue1Brown youtube channel** : I find this one really interesting and enjoyable, because they have animated many things to make us able to imagine the geometric meaning. 3Blue1Brown has been one of my favorite channels since I discovered them in Youtube.

M | T | W | T | F | S | S |
---|---|---|---|---|---|---|

« Aug | ||||||

1 | 2 | 3 | 4 | 5 | ||

6 | 7 | 8 | 9 | 10 | 11 | 12 |

13 | 14 | 15 | 16 | 17 | 18 | 19 |

20 | 21 | 22 | 23 | 24 | 25 | 26 |

27 | 28 | 29 | 30 | 31 |

## Leave a Reply