Gradient Descent

Earlier, I had a post on predicting Iris Flower types from their flower sizes with the gradient descent algorithm.

The algorithm is actually quite beautiful in its simplicity. I'll try to explain the what the algorithm does and why it works in this post. Hopefully, this will give you a better picture of how machines can "learn" from data.

Note: I'll be splitting my explanation into two parts. This is part 1. Part 2 will be coming out soon.

--------------------------------------

In its most basic implementation, the gradient descent tries to fit a line to a set of points.

So, given a set of an m number points on the xy coordinate system below,

x | y
1 | -1
3 | 1
...
...
x[m] | y[m]

My excellent plotting skills

  The algorithm would want to find a line like the green one I drew.

Because our data set is linear (follows a sort of straight line), we can represent the line as a linear function in the form f(x) = mx + b.

Following the naming convention, the line is called: h(x) = θ[0] + θ[1]x , but essentially their the same.

So, ready? The algorithm is as follows:

Repeat until convergence {
    For θ[j] in every θ {

        update θ[j] to be :

    }
} where a is the learning rate
and m is the number of training points

For our example, the algorithm would essentially be this:

Repeat until the values of θ don't change anymore {
     for θ[j] in the θ we have (θ[0] and θ[1]) {
          update θ[j] to be :
                 itself + the learning rate * the summation with all points i of ( (the actual value - the predicted value) * the x of that point)
     }
}  

 ------------------------------------

You may be wondering where these formulas come from and why they work. To understand that, you'll first have to know what it means to have a "better fitting" line.

First off the cost function is a function that measures a "bad" a line fits a data set. If the cost function is high, the line doesn't fit the data set very well. The goal of gradient descent is to minimize the cost function.

The cost function used here is:

where m is the number of training points. 

If you look closely, you'll find that the cost function is actually a total of the distances between the predicted line and the actual points.

  

------------------------------

This is the part 1. Any questions or comments? Post them below.

An Introduction to command line interfaces

If you've ever seen a hacker on TV or something, chances were that they were working on command line or shell of some sorts. Well, the huge black screen with white bitmap font isn't just for hackers, the interface is commonly used by developers and users just wanting to get a little more power over their computer.

zshell (Source: Wikimedia commons)

Despite its intimidating interface, using command line is actually quite intuitive. It was quite easy for me to learn. In essence, the user first inputs a command (hence the name "command line"), and then the computer executes the command and shows the output.

Not all command lines are the same, however. For my examples, I'll be using the classic "bash shell". So, on my bash shell, the command "echo hello world"

~/ $ echo hello world!

outputs

hello world!

on the computer. Although there are many command lines, their commands share pretty much the same properties. In their basic format, all commands in the command line are made of two parts:

1. The executable/program
2. The arguments/options/inputs 

In my example above, I called the executable "echo" and used the string "hello world!" as the argument. The echo command just types back what you put in as the argument.

In addition to executing commands, the command line navigates around the filesystem of the computer. You can think of the command line as something like a person in a house, where the house is the computer itself. You can do stuff in your house to manipulate or interact with the house. Similarly, you can move to a different room in the house. Anytime you're on the command line, you're are somewhere in the filesystem of the computer.

The command "pwd" shows where I am in the command line.

~/ $ pwd

outputs for me

 /home/Documents/

So, if I wanted to see whats in the current "room", or file "path", I would use the command "ls"

 ~/ $ ls

could show

foo.txt     helloreader.cpp    anyrandomfile    anyrandomfolder

folderofhomework 

You're now probably wondering "How do I change my location?". Well, dear reader, you use the command "cd". The argument for "cd" must be a folder of some sort.

 ~/ $ cd folderofhomework

The command does not output anything, but if you use "pwd" again, you can see that you've moved locations. 

 ~/ $ pwd
/home/Documents/folderofhomework

Those are the basics of any command line interface. Depending on what shell or prompt you have available, the commands may be different.

One of the equivalents of the commands I used above for the "command prompt" on windows machines is "dir" to "ls". The command "cd" is the same. However, on the command prompt, there is no direct equivalent for "pwd". You have to use the command "echo %cd%"


Any questions, comments? Post them below. 

Commands for unix derived command lines (bash, mac terminal, cshell, etc.) 
Commands for the command prompt on windows 
How to get to the Command prompt on Windows 
How to get to the "terminal" app on the Mac