MapReduce

• All tests were done with Python 2.7
• This is what is used on AWS EMR, so we're going to try to keep versions compatible.

• I will give some background, answer questions
• I'll introduce a script
• The exercise will have audience extending the script to get it working
• We'll go over an example solution

• This will help focus this talk
• You'll run into bugs, neighbor can be a good person to ask!
• Name, years using Python, favorite Python function
• I'll be helping out as much as I can, but help each other out first!

• On a single machine 40+ hours!
• If we really had only a single machine, we wouldn't be able to keep up!
• Mistake can't be fixed in a day (billing especially important)

• With many machines, how to they get access to the 100 GB of logs?
• How do they coordinate who gets which section of logs?
• How do we calculate the average?
• What happens when one of the boxes dies?
  • Detecting failure (timeout waiting for data? Out of band?)
  • Decide who takes over the data

• Did you have a super-computer?
• What programming language were you using?
• Type of problem being solved (working on graphs, or web logs, …)

• Limiting yourself to what can be expressed may seem like a loss
• But it enables the implementation to handle the problems we talked about
• And then can be used as understandable building blocks

• MapReduce's main benefits are for running over many machines, fault tolerance
• But we'll just practice on one machine
• Then see how to run in the cloud

• URL Shortener
• How many actions have we seen?
• Redirects: 450, Saves: 40, Loads: 60

Input

Key, Value

Output

Keys, Values

Input Key

Log line number

Input Value

Log line text

Output Key

Action

Output Value

times this action has occurred on this line

Input

Key, Values

Output

Keys, Values

Input Key

Action

Input Values

Counts: [1,1,1,1]

Output Key

Action

Output Value

Total Count

Output Key

Action

Output Value

Total Count

"redirect"  4
"save"      2
"load"      3

• By only looking at keys and values, can optimize a lot of backend work
• Where to send the results?
• What to do when a computer fails? (Just restart failed part)

• This was the situation after map
• Keys all jumbled
• What Hadoop does is sort them and distribute them to computers

• Now it is easy to distribute, and can handle all the load at once
• When mrjob is running, you'll notice debugging output, including a "sort" step. This is why.

• MapReduce distributes computing power by distributing input
• Input is distributed by splitting on lines (records)
• You cannot depend on lines being "together" in MapReduce

• URL Shortener logging

app.logger.info("Handling request for " + cookie)
...
# find redirect
...
app.logger.info("Redirecting to " + destination)

• Ability to associate the redirect with the cookie?
  • eg. which cookie had the most redirects?

• Logging once

log_data['cookie'] = cookie
...
# find redirect
log_data['action'] = 'redirect'
app.logger.info(json.dumps(log_data))

• Let's take a look at another example
• instead of counting actions, 1 per line, now we want to count words
• potentially many per line
• Same general idea though. Tunnel vision on one line (even one word!)

• mapper takes keys and values
• reducer takes the keys output by the mapper, and all relevant values
• split takes a string and splits on spaces, giving words
• yield essentially returns <key,value> pairs, but can be called more than once

• Open up file
• File tells us information about a "Vroot"
• Vroot is just a fancy name for a page with a numeric ID
• each line is different information about the vroot:
• A: attributes of a vroot (id, title, url)
• C: customer information, following which are the Visits that customer made to the vroot
• V: a visit to a vroot (id)
• Before we talked about actions of redirecting, saving, etc. Now we have a lot more different actions: each vroot with an ID

• vroot ID => number of visits
• Group by the vroot ID
• Summarize all of the visits

• and what does it look to actually run this?

• Run with python
• Output some debugging information while it is calculating
• Finally, output results

• csv_readline takes in a CSV line, return a list of values

• This is a little out of the scope of this tutorial
• Just know that not all types of data are amenable to MapReduce, and

Hadoop/mrjob specifically

• code/top\_titles.py

• Can tag with a dictionary instead

• You may run with more data by downloading and extracting from http://www.yelp.com/dataset_challenge

• The output of one MapReduce job can be used as the input to another

• PageRank: Multiple steps till solution converges
• Multi-level summaries

• PageRank is an algorithm for calculating the important of a page
• But it depends on the importance of every page pointing to it!
• So iteratively calculate the important of all pages
• Find average presidential donations by candidate, then normalize averages

• For our purposes, what is always the mapper input?
• What feature do we want to calculate first?
• Given this mapper output, what must the reducer input be?
• What property about a review are we interested in?

• Map Input: <review-id, 1>
• Map Output: <review-id, 1>
• Reducer Input: <review-id, [1,1,…]>
• Reducer Output: <review-id, sum>

• Given the reducer output, what must the mapper input be (for chained MapReduce steps)
• What do we want to group by?
• Given this mapper output, what must the reducer input be?
• What are we calculating?

• Open up code/unique_review.py
• We're using a JSON object per line. Other options include JSON key-values,

or writing your own

• Instead of raw text, record will now be a Python dict

• Protocols are straightforward to write
• At their heart contain read and write method
• For example, we've implemented a Protocol for MySQL INSERT statements, so

can easily process data created by mysqldump

• line oriented is limitation of Hadoop Streaming
• It sends data through stdin/stdout. Sometimes people ask if they can process an XML file with arbitrary line breaks
• Not what mrjob is good at. Hadoop may break at any point without extra

configuration

• Some of you may be familiar with Hadoop's InputFormat or OutputFormat.

They serve a slightly different purpose than Protocols

• InputFormats can be used to extract data which is not text on disk, for

example SequenceFiles, or from data which is not simply stored in a directory, like providing a Manifest file.

• One of the core skill of a DS is translating an intuition into a

quantifiable entity

• An example is for recommendation systems. If you want to find people to

follow, or friends to recommend, you often want to find "similar" people to yourself

• But what does similar mean? (Ask)
• In order to recommend one product over another, or sort in any useful way,

you need to associate a number with that similarity

• For example intracity geographic distance
• Or house price and square feet
• Called "vector space" because a point also describes a vector, which you

can get interesting information about (magnitude, angles, etc)

• Simple starting point: Euclidean distance
• Has many downsides, but is an example of translating a concept of

similarity to a number

• Interestingly the notions of distance and similarity are two sides of the

same coin.

• Also called the L₂ norm because we're taking the second root. We could

cube and take the cube root… leads to different semantics

• How to measure if points don't map so easily to this "vector space"?

1. Break up into a set
2. calculate # in intersection
3. calculate # in union
4. divide

• None = 0
• All = 1

• Takes as input the list of reviews that we have
• You'll be more on your own for this one, very limited skeleton

• We do duplicate work, but the trade-off is parallelism in the best case

(not many users are overlapping with each other for dozens of reviews)

• API to bring up a cluster of N machines
• Amazon provides tech for registering them all together, running Bootstrap

commands, terminating them, etc

• This cluster is called a JobFlow
• We'll discuss what it means to have one JobFlow with multiple Jobs in a bit

• A config file is easy to install across developers. Yelp has 3 main ones:

standard, memory-intensive, and cpu-intensive.

• Otherwise all of your developers must learn the different options for the

AWS API

• Often you'll need to make files available to EMR via S3, for example custom

packages you'd like to install. EMR cannot reach back out to your localhost. mrjob will automate the uploading and management of these files.

• mrjob "tags" JobFlows with information about the user running them, and has

tools for aggregating those metrics.

• It uses the tags to find compatible JobFlows, in the case where it will

share an existing one.

Instance types

how beefy a machine do you want to run? Is your load

memory or CPU bound? Yelp uses a few standard options, starting with m1.large

EMR keys

These are used to communicate with AWS to start your instance

Location for logs

Where in S3 do the results of the run go? EMR logs results of

bootstrap actions, Hadoop logs results of running the query.

SSH key

Once the instance is started, mrjob can connect to it via SSH and

stream logs, and setup a tunnel for HTTP traffic to use the Hadoop interface

Bootstrap behavior

Often you'll need to install libraries, or upload your

own repository. Since your code is running on the entire cluster at once, this install needs to happen on all machines

• When repeatedly changing and testing your code, you'll want a new version

deployed to the machine each time. That is what setup commands are for: parts of your application that change

• Spin up a job flow, when finished, kill it
• Remaining time and end of last hour is wasted
• Billed for it, but not used

• Don't end job flow after job, share remaining time

• New jobs can wait N minutes for an idle flow