2013-03-01-SVM

Classifiers

describes and distinguishes cases. Yelp may want to find a category for a business based on the reviews and business description

Regressions

Predict a continuous value. Eg. predict a home's selling price given sq footage, # of bedrooms

Clustering

find "natural" groups of data without labels

Outlier

find anomalous transactions, eg. finding fraud for credit cards

• We'd like to know how to price a house based on the square footage
• Let's pretend this is the data we have
• How would we guess that value for 2500 sq ft?

• In English, how would you solve this?
• How to mathematically represent the line?
• What is a good line?

• This is big problem for engineering and math (stats) in general
• We'll cover some concepts, but if you're ever stuck, try looking in related fields
• What are some of the ways we can measure distance between points? Euclidian, Manhattan, Euclidian == L₂ norm
• What is a way to aggrgate numbers? sum, sum of squares, sum of logs
• Differences between the last two?

Log

Useful for de-emphasizing large raw differences

Square

Useful for taking the approximate absolute value

• We want the difference from what we estimate to be the value to what the value actually is

• Now we have info about all the errors from points, how to summarize?
• Some points have negative error, some positive? Do they cancel each other out?
• Imagine data set of two points: one solutions covers lines, other divides them. Which is better?
• Use our squaring trick to make sure we don't have any negative values
• Normalize by the number of points

• Function spits out a metric. Metric can be thought of as fitness or cost
• Find the maximum or minimum of that metric
• Depending on your fitness function, this can be easy or difficult
• img: http://onlinestatbook.com

• What happens to the error as we move line around?
• Decreases until best fit, then increases
• What happens if we plot this error? Say, slope (x) against error (y)?

• How to find the minimum of functions in general?
• Take derivative, find 0
• Taking derivative can be complex or impossible (discontinuities) for some functions, or solving for 0 is difficult
• Instead, well keep getting closer to the minimum using the function we already have

• Take gradient by looking at the local derivative, or perturbating x
• Choose a as step size weight: big a is large step size
• If deriv is large, will also make you step size large.
• If deriv is large, probably means you are far away from minimum
• Keep repeating
• What happens if a is too small?
• What happens if a is too big?

• Desired output of the error as gradient descent runs
• maybe some local problems, as step size is too big, but slowly move down to a small amount of error

• How to handle case where separator line is not along an axis?

• Could say if x>2 and y>2, but not a great intuitive fit
• Draw a line that takes both into account
• y = m*x + b
• img: http://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick.html

• Which is the best?
• Why?

• Best line gives the most distance between the two classes
• Measure distance between closest points
• Closest points == support vectors

• Points can be represented as vectors
• Vector math can be easier to express succinctly
• img: http://www.sciencedirect.com/science/article/pii/S1072751511001918

• Plane
• Hyperplane

• How do you mathematically represent a line?
• Now, we're not going to think of a new letter for every dimension, we're just going to say x₁ , x₂ , x₃ …
• Rewrite mathematically
• How to add more dimensions? x₂₂? Express x as a vector of all attributes
• Again, don't want to come up with a bunch more letters after m, so use w as the matrix representing all the m slopes

• Transform it into linearly separable
• What function can we apply to these data points to make them separable?

Now apply SVM

• img: http://www.sciencedirect.com/science/article/pii/S1072751511001918