HW6-Github
Table of Contents
1 RESTful in Action center
Use Github's API
2 curl
- Command-line URL
- Shortcut for manually making Telnet requests
- turned into a library
2.1 We'll just use command line notes
3 Show Headers
curl -i http://people.ischool.berkeley.edu/~jblomo/webarch2014/index.html
HTTP/1.1 200 OK Date: Fri, 28 Sep 2012 06:25:13 GMT Server: Apache/2.2.22 (Fedora) Last-Modified: Fri, 31 Aug 2012 23:27:30 GMT ETag: "a6ec002-48-4c89822b95c80" Accept-Ranges: bytes Content-Length: 72 Content-Type: text/html; charset=UTF-8
<!DOCTYPE html> <html> <head> <title>Jim's Page</title> </head> <body> <p>Hello class! This is a tiny HTML page.</p> </body> </html>
4 Github API
curl -i "https://api.github.com/repos/apache/spark/issues?state=closed"
HTTP/1.1 200 OK Server: GitHub.com Date: Tue, 30 Sep 2014 05:25:16 GMT Content-Type: application/json; charset=utf-8 Status: 200 OK X-RateLimit-Limit: 60 X-RateLimit-Remaining: 55 X-RateLimit-Reset: 1412056221 Cache-Control: public, max-age=60, s-maxage=60 ETag: "6e627a2ea771f6cb488774d4ae38704c" Vary: Accept X-GitHub-Media-Type: github.v3 Link: <https://api.github.com/repositories/17165658/issues?state=closed&page=2>; rel="next", <https://api.github.com/repositories/17165658/issues?state=closed&page=78>; rel="last" X-XSS-Protection: 1; mode=block X-Frame-Options: deny Content-Security-Policy: default-src 'none' Content-Length: 67018 Access-Control-Allow-Credentials: true Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval Access-Control-Allow-Origin: * X-GitHub-Request-Id: 49AA5EE7:13AF:222DF9AF:542A3EBB Strict-Transport-Security: max-age=31536000; includeSubdomains; preload X-Content-Type-Options: nosniff Vary: Accept-Encoding X-Served-By: a8d8e492d6966f0c23dee2eed64c678a
[
{
"url": "https://api.github.com/repos/apache/spark/issues/2587",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2587/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2587/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2587/events",
"html_url": "https://github.com/apache/spark/pull/2587",
"id": 44379254,
"number": 2587,
"title": "SPARK-3737 - [Docs] Broken Link - Minor typo\t",
"user": {
"login": "bigsnarfdude",
"id": 2282364,
"avatar_url": "https://avatars.githubusercontent.com/u/2282364?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/bigsnarfdude",
"html_url": "https://github.com/bigsnarfdude",
"followers_url": "https://api.github.com/users/bigsnarfdude/followers",
"following_url": "https://api.github.com/users/bigsnarfdude/following{/other_user}",
"gists_url": "https://api.github.com/users/bigsnarfdude/gists{/gist_id}",
"starred_url": "https://api.github.com/users/bigsnarfdude/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/bigsnarfdude/subscriptions",
"organizations_url": "https://api.github.com/users/bigsnarfdude/orgs",
"repos_url": "https://api.github.com/users/bigsnarfdude/repos",
"events_url": "https://api.github.com/users/bigsnarfdude/events{/privacy}",
"received_events_url": "https://api.github.com/users/bigsnarfdude/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 2,
"created_at": "2014-09-30T01:43:03Z",
"updated_at": "2014-09-30T01:56:04Z",
"closed_at": "2014-09-30T01:56:04Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2587",
"html_url": "https://github.com/apache/spark/pull/2587",
"diff_url": "https://github.com/apache/spark/pull/2587.diff",
"patch_url": "https://github.com/apache/spark/pull/2587.patch"
},
"body": "fixed broken link to building spark using building-with-maven.html link "
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2585",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2585/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2585/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2585/events",
"html_url": "https://github.com/apache/spark/pull/2585",
"id": 44363165,
"number": 2585,
"title": "[SPARK-3709] Fix the flaky test in BroadcastSuite.",
"user": {
"login": "rxin",
"id": 323388,
"avatar_url": "https://avatars.githubusercontent.com/u/323388?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/rxin",
"html_url": "https://github.com/rxin",
"followers_url": "https://api.github.com/users/rxin/followers",
"following_url": "https://api.github.com/users/rxin/following{/other_user}",
"gists_url": "https://api.github.com/users/rxin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/rxin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/rxin/subscriptions",
"organizations_url": "https://api.github.com/users/rxin/orgs",
"repos_url": "https://api.github.com/users/rxin/repos",
"events_url": "https://api.github.com/users/rxin/events{/privacy}",
"received_events_url": "https://api.github.com/users/rxin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 6,
"created_at": "2014-09-29T22:04:39Z",
"updated_at": "2014-09-30T04:36:35Z",
"closed_at": "2014-09-30T04:36:35Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2585",
"html_url": "https://github.com/apache/spark/pull/2585",
"diff_url": "https://github.com/apache/spark/pull/2585.diff",
"patch_url": "https://github.com/apache/spark/pull/2585.patch"
},
"body": "The problem was introduced when I refactored the network module. Previously ConnectionManager / BlockManagerWorker silently ignores blocks that don't exist, whereas the new code throws an exception.\r\n\r\n"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2582",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2582/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2582/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2582/events",
"html_url": "https://github.com/apache/spark/pull/2582",
"id": 44338342,
"number": 2582,
"title": "Changed the linke to build spark",
"user": {
"login": "anantasty",
"id": 281777,
"avatar_url": "https://avatars.githubusercontent.com/u/281777?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/anantasty",
"html_url": "https://github.com/anantasty",
"followers_url": "https://api.github.com/users/anantasty/followers",
"following_url": "https://api.github.com/users/anantasty/following{/other_user}",
"gists_url": "https://api.github.com/users/anantasty/gists{/gist_id}",
"starred_url": "https://api.github.com/users/anantasty/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/anantasty/subscriptions",
"organizations_url": "https://api.github.com/users/anantasty/orgs",
"repos_url": "https://api.github.com/users/anantasty/repos",
"events_url": "https://api.github.com/users/anantasty/events{/privacy}",
"received_events_url": "https://api.github.com/users/anantasty/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 2,
"created_at": "2014-09-29T18:53:26Z",
"updated_at": "2014-09-29T21:57:22Z",
"closed_at": "2014-09-29T21:57:22Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2582",
"html_url": "https://github.com/apache/spark/pull/2582",
"diff_url": "https://github.com/apache/spark/pull/2582.diff",
"patch_url": "https://github.com/apache/spark/pull/2582.patch"
},
"body": "https://issues.apache.org/jira/browse/SPARK-3725"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2581",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2581/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2581/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2581/events",
"html_url": "https://github.com/apache/spark/pull/2581",
"id": 44333698,
"number": 2581,
"title": "Minor cleanup of code.",
"user": {
"login": "rxin",
"id": 323388,
"avatar_url": "https://avatars.githubusercontent.com/u/323388?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/rxin",
"html_url": "https://github.com/rxin",
"followers_url": "https://api.github.com/users/rxin/followers",
"following_url": "https://api.github.com/users/rxin/following{/other_user}",
"gists_url": "https://api.github.com/users/rxin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/rxin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/rxin/subscriptions",
"organizations_url": "https://api.github.com/users/rxin/orgs",
"repos_url": "https://api.github.com/users/rxin/repos",
"events_url": "https://api.github.com/users/rxin/events{/privacy}",
"received_events_url": "https://api.github.com/users/rxin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 8,
"created_at": "2014-09-29T18:31:06Z",
"updated_at": "2014-09-30T04:57:41Z",
"closed_at": "2014-09-30T04:57:41Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2581",
"html_url": "https://github.com/apache/spark/pull/2581",
"diff_url": "https://github.com/apache/spark/pull/2581.diff",
"patch_url": "https://github.com/apache/spark/pull/2581.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2580",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2580/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2580/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2580/events",
"html_url": "https://github.com/apache/spark/pull/2580",
"id": 44330159,
"number": 2580,
"title": "Add more debug message for ManagedBuffer",
"user": {
"login": "rxin",
"id": 323388,
"avatar_url": "https://avatars.githubusercontent.com/u/323388?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/rxin",
"html_url": "https://github.com/rxin",
"followers_url": "https://api.github.com/users/rxin/followers",
"following_url": "https://api.github.com/users/rxin/following{/other_user}",
"gists_url": "https://api.github.com/users/rxin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/rxin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/rxin/subscriptions",
"organizations_url": "https://api.github.com/users/rxin/orgs",
"repos_url": "https://api.github.com/users/rxin/repos",
"events_url": "https://api.github.com/users/rxin/events{/privacy}",
"received_events_url": "https://api.github.com/users/rxin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 8,
"created_at": "2014-09-29T18:14:09Z",
"updated_at": "2014-09-29T19:46:23Z",
"closed_at": "2014-09-29T19:42:00Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2580",
"html_url": "https://github.com/apache/spark/pull/2580",
"diff_url": "https://github.com/apache/spark/pull/2580.diff",
"patch_url": "https://github.com/apache/spark/pull/2580.patch"
},
"body": "This is to help debug the error reported at http://apache-spark-user-list.1001560.n3.nabble.com/SQL-queries-fail-in-1-2-0-SNAPSHOT-td15327.html"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2578",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2578/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2578/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2578/events",
"html_url": "https://github.com/apache/spark/pull/2578",
"id": 44309818,
"number": 2578,
"title": "[EC2] Sort long, manually-inputted dictionaries",
"user": {
"login": "nchammas",
"id": 1039369,
"avatar_url": "https://avatars.githubusercontent.com/u/1039369?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/nchammas",
"html_url": "https://github.com/nchammas",
"followers_url": "https://api.github.com/users/nchammas/followers",
"following_url": "https://api.github.com/users/nchammas/following{/other_user}",
"gists_url": "https://api.github.com/users/nchammas/gists{/gist_id}",
"starred_url": "https://api.github.com/users/nchammas/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/nchammas/subscriptions",
"organizations_url": "https://api.github.com/users/nchammas/orgs",
"repos_url": "https://api.github.com/users/nchammas/repos",
"events_url": "https://api.github.com/users/nchammas/events{/privacy}",
"received_events_url": "https://api.github.com/users/nchammas/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 5,
"created_at": "2014-09-29T15:55:31Z",
"updated_at": "2014-09-29T17:46:39Z",
"closed_at": "2014-09-29T17:46:39Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2578",
"html_url": "https://github.com/apache/spark/pull/2578",
"diff_url": "https://github.com/apache/spark/pull/2578.diff",
"patch_url": "https://github.com/apache/spark/pull/2578.patch"
},
"body": "Similar to the work done in #2571, this PR just sorts the remaining manually-inputted dicts in the EC2 script so they are easier to maintain."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2572",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2572/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2572/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2572/events",
"html_url": "https://github.com/apache/spark/pull/2572",
"id": 44259800,
"number": 2572,
"title": "[CORE] Bugfix: LogErr format in DAGScheduler.scala",
"user": {
"login": "liyezhang556520",
"id": 4716022,
"avatar_url": "https://avatars.githubusercontent.com/u/4716022?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/liyezhang556520",
"html_url": "https://github.com/liyezhang556520",
"followers_url": "https://api.github.com/users/liyezhang556520/followers",
"following_url": "https://api.github.com/users/liyezhang556520/following{/other_user}",
"gists_url": "https://api.github.com/users/liyezhang556520/gists{/gist_id}",
"starred_url": "https://api.github.com/users/liyezhang556520/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/liyezhang556520/subscriptions",
"organizations_url": "https://api.github.com/users/liyezhang556520/orgs",
"repos_url": "https://api.github.com/users/liyezhang556520/repos",
"events_url": "https://api.github.com/users/liyezhang556520/events{/privacy}",
"received_events_url": "https://api.github.com/users/liyezhang556520/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 5,
"created_at": "2014-09-29T05:56:10Z",
"updated_at": "2014-09-29T08:15:13Z",
"closed_at": "2014-09-29T08:15:02Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2572",
"html_url": "https://github.com/apache/spark/pull/2572",
"diff_url": "https://github.com/apache/spark/pull/2572.diff",
"patch_url": "https://github.com/apache/spark/pull/2572.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2571",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2571/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2571/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2571/events",
"html_url": "https://github.com/apache/spark/pull/2571",
"id": 44254629,
"number": 2571,
"title": "[EC2] Cleanup Python parens and disk dict",
"user": {
"login": "nchammas",
"id": 1039369,
"avatar_url": "https://avatars.githubusercontent.com/u/1039369?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/nchammas",
"html_url": "https://github.com/nchammas",
"followers_url": "https://api.github.com/users/nchammas/followers",
"following_url": "https://api.github.com/users/nchammas/following{/other_user}",
"gists_url": "https://api.github.com/users/nchammas/gists{/gist_id}",
"starred_url": "https://api.github.com/users/nchammas/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/nchammas/subscriptions",
"organizations_url": "https://api.github.com/users/nchammas/orgs",
"repos_url": "https://api.github.com/users/nchammas/repos",
"events_url": "https://api.github.com/users/nchammas/events{/privacy}",
"received_events_url": "https://api.github.com/users/nchammas/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 6,
"created_at": "2014-09-29T03:38:06Z",
"updated_at": "2014-09-29T15:55:31Z",
"closed_at": "2014-09-29T04:56:20Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2571",
"html_url": "https://github.com/apache/spark/pull/2571",
"diff_url": "https://github.com/apache/spark/pull/2571.diff",
"patch_url": "https://github.com/apache/spark/pull/2571.patch"
},
"body": "Minor fixes:\r\n* Remove unnecessary parens (Python style)\r\n* Sort `disks_by_instance` dict and remove duplicate `t1.micro` key"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2568",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2568/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2568/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2568/events",
"html_url": "https://github.com/apache/spark/pull/2568",
"id": 44248715,
"number": 2568,
"title": "SPARK-3716 [GraphX] Update Analytics.scala for partitionStrategy assignment",
"user": {
"login": "NamelessAnalyst",
"id": 8952128,
"avatar_url": "https://avatars.githubusercontent.com/u/8952128?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/NamelessAnalyst",
"html_url": "https://github.com/NamelessAnalyst",
"followers_url": "https://api.github.com/users/NamelessAnalyst/followers",
"following_url": "https://api.github.com/users/NamelessAnalyst/following{/other_user}",
"gists_url": "https://api.github.com/users/NamelessAnalyst/gists{/gist_id}",
"starred_url": "https://api.github.com/users/NamelessAnalyst/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/NamelessAnalyst/subscriptions",
"organizations_url": "https://api.github.com/users/NamelessAnalyst/orgs",
"repos_url": "https://api.github.com/users/NamelessAnalyst/repos",
"events_url": "https://api.github.com/users/NamelessAnalyst/events{/privacy}",
"received_events_url": "https://api.github.com/users/NamelessAnalyst/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 0,
"created_at": "2014-09-29T01:32:10Z",
"updated_at": "2014-09-29T01:33:24Z",
"closed_at": "2014-09-29T01:33:24Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2568",
"html_url": "https://github.com/apache/spark/pull/2568",
"diff_url": "https://github.com/apache/spark/pull/2568.diff",
"patch_url": "https://github.com/apache/spark/pull/2568.patch"
},
"body": "Previously, when the val partitionStrategy was created it called a function in the Analytics object which was a copy of the PartitionStrategy.fromString() method. This function has been removed, and the assignment of partitionStrategy now uses the PartitionStrategy.fromString method instead. In this way, it better matches the declarations of edge/vertex StorageLevel variables. "
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2567",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2567/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2567/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2567/events",
"html_url": "https://github.com/apache/spark/pull/2567",
"id": 44247625,
"number": 2567,
"title": "[SPARK-3715][Docs]minor typo",
"user": {
"login": "WangTaoTheTonic",
"id": 5276001,
"avatar_url": "https://avatars.githubusercontent.com/u/5276001?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/WangTaoTheTonic",
"html_url": "https://github.com/WangTaoTheTonic",
"followers_url": "https://api.github.com/users/WangTaoTheTonic/followers",
"following_url": "https://api.github.com/users/WangTaoTheTonic/following{/other_user}",
"gists_url": "https://api.github.com/users/WangTaoTheTonic/gists{/gist_id}",
"starred_url": "https://api.github.com/users/WangTaoTheTonic/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/WangTaoTheTonic/subscriptions",
"organizations_url": "https://api.github.com/users/WangTaoTheTonic/orgs",
"repos_url": "https://api.github.com/users/WangTaoTheTonic/repos",
"events_url": "https://api.github.com/users/WangTaoTheTonic/events{/privacy}",
"received_events_url": "https://api.github.com/users/WangTaoTheTonic/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 4,
"created_at": "2014-09-29T01:02:00Z",
"updated_at": "2014-09-29T02:13:09Z",
"closed_at": "2014-09-29T01:31:41Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2567",
"html_url": "https://github.com/apache/spark/pull/2567",
"diff_url": "https://github.com/apache/spark/pull/2567.diff",
"patch_url": "https://github.com/apache/spark/pull/2567.patch"
},
"body": "https://issues.apache.org/jira/browse/SPARK-3715\r\n\r\n"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2562",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2562/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2562/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2562/events",
"html_url": "https://github.com/apache/spark/pull/2562",
"id": 44226943,
"number": 2562,
"title": "[SPARK-3712][STREAMING]: add a new UpdateDStream to update a rdd dynamically",
"user": {
"login": "uncleGen",
"id": 7402327,
"avatar_url": "https://avatars.githubusercontent.com/u/7402327?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/uncleGen",
"html_url": "https://github.com/uncleGen",
"followers_url": "https://api.github.com/users/uncleGen/followers",
"following_url": "https://api.github.com/users/uncleGen/following{/other_user}",
"gists_url": "https://api.github.com/users/uncleGen/gists{/gist_id}",
"starred_url": "https://api.github.com/users/uncleGen/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/uncleGen/subscriptions",
"organizations_url": "https://api.github.com/users/uncleGen/orgs",
"repos_url": "https://api.github.com/users/uncleGen/repos",
"events_url": "https://api.github.com/users/uncleGen/events{/privacy}",
"received_events_url": "https://api.github.com/users/uncleGen/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 7,
"created_at": "2014-09-28T10:57:26Z",
"updated_at": "2014-09-29T01:45:50Z",
"closed_at": "2014-09-28T14:43:29Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2562",
"html_url": "https://github.com/apache/spark/pull/2562",
"diff_url": "https://github.com/apache/spark/pull/2562.diff",
"patch_url": "https://github.com/apache/spark/pull/2562.patch"
},
"body": "Maybe, we can achieve the aim by using \"forEachRdd\" function. But it is weird in this way, because I need to pass a closure, like this:\r\n\r\n val baseRdd = ...\r\n var updatedRDD = ...\r\n val inputStream = ...\r\n\r\n val func = (rdd: RDD[T], t: Time) => {\r\n updatedRDD = baseRDD.op(rdd)\r\n }\r\n\r\n inputStream.foreachRDD(func _)\r\n\r\nIn my PR, we can update a rdd like:\r\n\r\n val updateStream = inputStream.updateRDD(baseRDD, func).asInstanceOf[U, V, T]\r\n\r\nand obtain the updatedRDD like this:\r\n\r\n val updatedRDD = updateStream.getUpdatedRDD"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2560",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2560/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2560/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2560/events",
"html_url": "https://github.com/apache/spark/pull/2560",
"id": 44216056,
"number": 2560,
"title": "[SPARK-3543] TaskContext remaining cleanup work.",
"user": {
"login": "rxin",
"id": 323388,
"avatar_url": "https://avatars.githubusercontent.com/u/323388?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/rxin",
"html_url": "https://github.com/rxin",
"followers_url": "https://api.github.com/users/rxin/followers",
"following_url": "https://api.github.com/users/rxin/following{/other_user}",
"gists_url": "https://api.github.com/users/rxin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/rxin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/rxin/subscriptions",
"organizations_url": "https://api.github.com/users/rxin/orgs",
"repos_url": "https://api.github.com/users/rxin/repos",
"events_url": "https://api.github.com/users/rxin/events{/privacy}",
"received_events_url": "https://api.github.com/users/rxin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 10,
"created_at": "2014-09-28T05:44:28Z",
"updated_at": "2014-09-29T03:34:03Z",
"closed_at": "2014-09-29T03:34:03Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2560",
"html_url": "https://github.com/apache/spark/pull/2560",
"diff_url": "https://github.com/apache/spark/pull/2560.diff",
"patch_url": "https://github.com/apache/spark/pull/2560.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2557",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2557/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2557/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2557/events",
"html_url": "https://github.com/apache/spark/pull/2557",
"id": 44138879,
"number": 2557,
"title": "[SPARK-3543] Clean up Java TaskContext implementation.",
"user": {
"login": "rxin",
"id": 323388,
"avatar_url": "https://avatars.githubusercontent.com/u/323388?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/rxin",
"html_url": "https://github.com/rxin",
"followers_url": "https://api.github.com/users/rxin/followers",
"following_url": "https://api.github.com/users/rxin/following{/other_user}",
"gists_url": "https://api.github.com/users/rxin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/rxin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/rxin/subscriptions",
"organizations_url": "https://api.github.com/users/rxin/orgs",
"repos_url": "https://api.github.com/users/rxin/repos",
"events_url": "https://api.github.com/users/rxin/events{/privacy}",
"received_events_url": "https://api.github.com/users/rxin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 11,
"created_at": "2014-09-27T04:42:17Z",
"updated_at": "2014-09-27T22:09:50Z",
"closed_at": "2014-09-27T21:47:58Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2557",
"html_url": "https://github.com/apache/spark/pull/2557",
"diff_url": "https://github.com/apache/spark/pull/2557.diff",
"patch_url": "https://github.com/apache/spark/pull/2557.patch"
},
"body": "This addresses some minor issues in https://github.com/apache/spark/pull/2425"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2555",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2555/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2555/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2555/events",
"html_url": "https://github.com/apache/spark/pull/2555",
"id": 44136807,
"number": 2555,
"title": "Minor cleanup to tighten visibility and remove compilation warning.",
"user": {
"login": "rxin",
"id": 323388,
"avatar_url": "https://avatars.githubusercontent.com/u/323388?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/rxin",
"html_url": "https://github.com/rxin",
"followers_url": "https://api.github.com/users/rxin/followers",
"following_url": "https://api.github.com/users/rxin/following{/other_user}",
"gists_url": "https://api.github.com/users/rxin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/rxin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/rxin/subscriptions",
"organizations_url": "https://api.github.com/users/rxin/orgs",
"repos_url": "https://api.github.com/users/rxin/repos",
"events_url": "https://api.github.com/users/rxin/events{/privacy}",
"received_events_url": "https://api.github.com/users/rxin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 5,
"created_at": "2014-09-27T04:08:30Z",
"updated_at": "2014-09-27T07:58:58Z",
"closed_at": "2014-09-27T07:58:58Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2555",
"html_url": "https://github.com/apache/spark/pull/2555",
"diff_url": "https://github.com/apache/spark/pull/2555.diff",
"patch_url": "https://github.com/apache/spark/pull/2555.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2553",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2553/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2553/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2553/events",
"html_url": "https://github.com/apache/spark/pull/2553",
"id": 44123246,
"number": 2553,
"title": "[graphX] GraphOps: random pick vertex bug",
"user": {
"login": "yingjieMiao",
"id": 3836918,
"avatar_url": "https://avatars.githubusercontent.com/u/3836918?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/yingjieMiao",
"html_url": "https://github.com/yingjieMiao",
"followers_url": "https://api.github.com/users/yingjieMiao/followers",
"following_url": "https://api.github.com/users/yingjieMiao/following{/other_user}",
"gists_url": "https://api.github.com/users/yingjieMiao/gists{/gist_id}",
"starred_url": "https://api.github.com/users/yingjieMiao/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/yingjieMiao/subscriptions",
"organizations_url": "https://api.github.com/users/yingjieMiao/orgs",
"repos_url": "https://api.github.com/users/yingjieMiao/repos",
"events_url": "https://api.github.com/users/yingjieMiao/events{/privacy}",
"received_events_url": "https://api.github.com/users/yingjieMiao/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 6,
"created_at": "2014-09-27T01:33:26Z",
"updated_at": "2014-09-30T01:03:41Z",
"closed_at": "2014-09-30T01:03:41Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2553",
"html_url": "https://github.com/apache/spark/pull/2553",
"diff_url": "https://github.com/apache/spark/pull/2553.diff",
"patch_url": "https://github.com/apache/spark/pull/2553.patch"
},
"body": "When `numVertices > 50`, probability is set to 0. This would cause infinite loop."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2550",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2550/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2550/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2550/events",
"html_url": "https://github.com/apache/spark/pull/2550",
"id": 44118170,
"number": 2550,
"title": "graphX page rank: test convergence bug",
"user": {
"login": "yingjieMiao",
"id": 3836918,
"avatar_url": "https://avatars.githubusercontent.com/u/3836918?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/yingjieMiao",
"html_url": "https://github.com/yingjieMiao",
"followers_url": "https://api.github.com/users/yingjieMiao/followers",
"following_url": "https://api.github.com/users/yingjieMiao/following{/other_user}",
"gists_url": "https://api.github.com/users/yingjieMiao/gists{/gist_id}",
"starred_url": "https://api.github.com/users/yingjieMiao/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/yingjieMiao/subscriptions",
"organizations_url": "https://api.github.com/users/yingjieMiao/orgs",
"repos_url": "https://api.github.com/users/yingjieMiao/repos",
"events_url": "https://api.github.com/users/yingjieMiao/events{/privacy}",
"received_events_url": "https://api.github.com/users/yingjieMiao/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 6,
"created_at": "2014-09-27T00:26:54Z",
"updated_at": "2014-09-27T01:26:02Z",
"closed_at": "2014-09-27T00:48:25Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2550",
"html_url": "https://github.com/apache/spark/pull/2550",
"diff_url": "https://github.com/apache/spark/pull/2550.diff",
"patch_url": "https://github.com/apache/spark/pull/2550.patch"
},
"body": "We stop sending messages when `delta` is no bigger than `tol`. Test convergence would fail if there is a big negative change."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2549",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2549/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2549/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2549/events",
"html_url": "https://github.com/apache/spark/pull/2549",
"id": 44111675,
"number": 2549,
"title": "Slaves file is now a template.",
"user": {
"login": "sarahgerweck",
"id": 3421345,
"avatar_url": "https://avatars.githubusercontent.com/u/3421345?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/sarahgerweck",
"html_url": "https://github.com/sarahgerweck",
"followers_url": "https://api.github.com/users/sarahgerweck/followers",
"following_url": "https://api.github.com/users/sarahgerweck/following{/other_user}",
"gists_url": "https://api.github.com/users/sarahgerweck/gists{/gist_id}",
"starred_url": "https://api.github.com/users/sarahgerweck/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/sarahgerweck/subscriptions",
"organizations_url": "https://api.github.com/users/sarahgerweck/orgs",
"repos_url": "https://api.github.com/users/sarahgerweck/repos",
"events_url": "https://api.github.com/users/sarahgerweck/events{/privacy}",
"received_events_url": "https://api.github.com/users/sarahgerweck/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 7,
"created_at": "2014-09-26T22:17:36Z",
"updated_at": "2014-09-27T05:24:07Z",
"closed_at": "2014-09-27T05:24:07Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2549",
"html_url": "https://github.com/apache/spark/pull/2549",
"diff_url": "https://github.com/apache/spark/pull/2549.diff",
"patch_url": "https://github.com/apache/spark/pull/2549.patch"
},
"body": "Change 0dc868e removed the `conf/slaves` file and made it a template like most of the other configuration files. This means you can no longer run `make-distribution.sh` unless you manually create a slaves file to be statically bundled in your distribution, which seems at odds with making it a template file."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2547",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2547/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2547/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2547/events",
"html_url": "https://github.com/apache/spark/pull/2547",
"id": 44094138,
"number": 2547,
"title": "SPARK-3699: SQL and Hive console tasks now clean up appropriately",
"user": {
"login": "willb",
"id": 1161,
"avatar_url": "https://avatars.githubusercontent.com/u/1161?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/willb",
"html_url": "https://github.com/willb",
"followers_url": "https://api.github.com/users/willb/followers",
"following_url": "https://api.github.com/users/willb/following{/other_user}",
"gists_url": "https://api.github.com/users/willb/gists{/gist_id}",
"starred_url": "https://api.github.com/users/willb/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/willb/subscriptions",
"organizations_url": "https://api.github.com/users/willb/orgs",
"repos_url": "https://api.github.com/users/willb/repos",
"events_url": "https://api.github.com/users/willb/events{/privacy}",
"received_events_url": "https://api.github.com/users/willb/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 10,
"created_at": "2014-09-26T18:48:04Z",
"updated_at": "2014-09-28T08:02:44Z",
"closed_at": "2014-09-28T08:02:44Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2547",
"html_url": "https://github.com/apache/spark/pull/2547",
"diff_url": "https://github.com/apache/spark/pull/2547.diff",
"patch_url": "https://github.com/apache/spark/pull/2547.patch"
},
"body": "The sbt tasks sql/console and hive/console will now `stop()`\r\nthe `SparkContext` upon exit. Previously, they left an ugly stack\r\ntrace when quitting."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2545",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2545/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2545/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2545/events",
"html_url": "https://github.com/apache/spark/pull/2545",
"id": 44048781,
"number": 2545,
"title": "Create historyServer",
"user": {
"login": "397090770",
"id": 5170878,
"avatar_url": "https://avatars.githubusercontent.com/u/5170878?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/397090770",
"html_url": "https://github.com/397090770",
"followers_url": "https://api.github.com/users/397090770/followers",
"following_url": "https://api.github.com/users/397090770/following{/other_user}",
"gists_url": "https://api.github.com/users/397090770/gists{/gist_id}",
"starred_url": "https://api.github.com/users/397090770/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/397090770/subscriptions",
"organizations_url": "https://api.github.com/users/397090770/orgs",
"repos_url": "https://api.github.com/users/397090770/repos",
"events_url": "https://api.github.com/users/397090770/events{/privacy}",
"received_events_url": "https://api.github.com/users/397090770/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 4,
"created_at": "2014-09-26T11:21:05Z",
"updated_at": "2014-09-26T11:39:35Z",
"closed_at": "2014-09-26T11:39:35Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2545",
"html_url": "https://github.com/apache/spark/pull/2545",
"diff_url": "https://github.com/apache/spark/pull/2545.diff",
"patch_url": "https://github.com/apache/spark/pull/2545.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2544",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2544/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2544/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2544/events",
"html_url": "https://github.com/apache/spark/pull/2544",
"id": 44048586,
"number": 2544,
"title": "Create Spark",
"user": {
"login": "397090770",
"id": 5170878,
"avatar_url": "https://avatars.githubusercontent.com/u/5170878?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/397090770",
"html_url": "https://github.com/397090770",
"followers_url": "https://api.github.com/users/397090770/followers",
"following_url": "https://api.github.com/users/397090770/following{/other_user}",
"gists_url": "https://api.github.com/users/397090770/gists{/gist_id}",
"starred_url": "https://api.github.com/users/397090770/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/397090770/subscriptions",
"organizations_url": "https://api.github.com/users/397090770/orgs",
"repos_url": "https://api.github.com/users/397090770/repos",
"events_url": "https://api.github.com/users/397090770/events{/privacy}",
"received_events_url": "https://api.github.com/users/397090770/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 1,
"created_at": "2014-09-26T11:19:02Z",
"updated_at": "2014-09-26T11:39:07Z",
"closed_at": "2014-09-26T11:39:07Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2544",
"html_url": "https://github.com/apache/spark/pull/2544",
"diff_url": "https://github.com/apache/spark/pull/2544.diff",
"patch_url": "https://github.com/apache/spark/pull/2544.patch"
},
"body": "Spark"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2540",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2540/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2540/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2540/events",
"html_url": "https://github.com/apache/spark/pull/2540",
"id": 43993764,
"number": 2540,
"title": "Docs : use \"--total-executor-cores\" rather than \"--cores\" after spark-shell",
"user": {
"login": "CrazyJvm",
"id": 1585761,
"avatar_url": "https://avatars.githubusercontent.com/u/1585761?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/CrazyJvm",
"html_url": "https://github.com/CrazyJvm",
"followers_url": "https://api.github.com/users/CrazyJvm/followers",
"following_url": "https://api.github.com/users/CrazyJvm/following{/other_user}",
"gists_url": "https://api.github.com/users/CrazyJvm/gists{/gist_id}",
"starred_url": "https://api.github.com/users/CrazyJvm/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/CrazyJvm/subscriptions",
"organizations_url": "https://api.github.com/users/CrazyJvm/orgs",
"repos_url": "https://api.github.com/users/CrazyJvm/repos",
"events_url": "https://api.github.com/users/CrazyJvm/events{/privacy}",
"received_events_url": "https://api.github.com/users/CrazyJvm/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 8,
"created_at": "2014-09-26T03:03:30Z",
"updated_at": "2014-09-27T16:45:10Z",
"closed_at": "2014-09-27T16:45:10Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2540",
"html_url": "https://github.com/apache/spark/pull/2540",
"diff_url": "https://github.com/apache/spark/pull/2540.diff",
"patch_url": "https://github.com/apache/spark/pull/2540.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2539",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2539/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2539/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2539/events",
"html_url": "https://github.com/apache/spark/pull/2539",
"id": 43992921,
"number": 2539,
"title": "[SPARK-3695]shuffle fetch fail output",
"user": {
"login": "adrian-wang",
"id": 3078910,
"avatar_url": "https://avatars.githubusercontent.com/u/3078910?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/adrian-wang",
"html_url": "https://github.com/adrian-wang",
"followers_url": "https://api.github.com/users/adrian-wang/followers",
"following_url": "https://api.github.com/users/adrian-wang/following{/other_user}",
"gists_url": "https://api.github.com/users/adrian-wang/gists{/gist_id}",
"starred_url": "https://api.github.com/users/adrian-wang/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/adrian-wang/subscriptions",
"organizations_url": "https://api.github.com/users/adrian-wang/orgs",
"repos_url": "https://api.github.com/users/adrian-wang/repos",
"events_url": "https://api.github.com/users/adrian-wang
72 67018 72 48692 0 0 66453 0 0:00:01 --:--:-- 0:00:01 66428/events{/privacy}",
"received_events_url": "https://api.github.com/users/adrian-wang/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 4,
"created_at": "2014-09-26T02:50:23Z",
"updated_at": "2014-09-26T18:31:41Z",
"closed_at": "2014-09-26T18:31:41Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2539",
"html_url": "https://github.com/apache/spark/pull/2539",
"diff_url": "https://github.com/apache/spark/pull/2539.diff",
"patch_url": "https://github.com/apache/spark/pull/2539.patch"
},
"body": "should output detailed host and port in error message"
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2537",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2537/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2537/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2537/events",
"html_url": "https://github.com/apache/spark/pull/2537",
"id": 43932654,
"number": 2537,
"title": "[SPARK-3690] Closing shuffle writers we swallow more important exception",
"user": {
"login": "epahomov",
"id": 2845390,
"avatar_url": "https://avatars.githubusercontent.com/u/2845390?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/epahomov",
"html_url": "https://github.com/epahomov",
"followers_url": "https://api.github.com/users/epahomov/followers",
"following_url": "https://api.github.com/users/epahomov/following{/other_user}",
"gists_url": "https://api.github.com/users/epahomov/gists{/gist_id}",
"starred_url": "https://api.github.com/users/epahomov/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/epahomov/subscriptions",
"organizations_url": "https://api.github.com/users/epahomov/orgs",
"repos_url": "https://api.github.com/users/epahomov/repos",
"events_url": "https://api.github.com/users/epahomov/events{/privacy}",
"received_events_url": "https://api.github.com/users/epahomov/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 8,
"created_at": "2014-09-25T15:32:22Z",
"updated_at": "2014-09-25T21:52:27Z",
"closed_at": "2014-09-25T21:52:27Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2537",
"html_url": "https://github.com/apache/spark/pull/2537",
"diff_url": "https://github.com/apache/spark/pull/2537.diff",
"patch_url": "https://github.com/apache/spark/pull/2537.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2536",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2536/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2536/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2536/events",
"html_url": "https://github.com/apache/spark/pull/2536",
"id": 43922253,
"number": 2536,
"title": "SPARK-3639 | Removed settings master in examples",
"user": {
"login": "aniketbhatnagar",
"id": 618933,
"avatar_url": "https://avatars.githubusercontent.com/u/618933?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/aniketbhatnagar",
"html_url": "https://github.com/aniketbhatnagar",
"followers_url": "https://api.github.com/users/aniketbhatnagar/followers",
"following_url": "https://api.github.com/users/aniketbhatnagar/following{/other_user}",
"gists_url": "https://api.github.com/users/aniketbhatnagar/gists{/gist_id}",
"starred_url": "https://api.github.com/users/aniketbhatnagar/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/aniketbhatnagar/subscriptions",
"organizations_url": "https://api.github.com/users/aniketbhatnagar/orgs",
"repos_url": "https://api.github.com/users/aniketbhatnagar/repos",
"events_url": "https://api.github.com/users/aniketbhatnagar/events{/privacy}",
"received_events_url": "https://api.github.com/users/aniketbhatnagar/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 10,
"created_at": "2014-09-25T14:00:30Z",
"updated_at": "2014-09-26T17:00:06Z",
"closed_at": "2014-09-26T17:00:06Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2536",
"html_url": "https://github.com/apache/spark/pull/2536",
"diff_url": "https://github.com/apache/spark/pull/2536.diff",
"patch_url": "https://github.com/apache/spark/pull/2536.patch"
},
"body": "This patch removes setting of master as local in Kinesis examples so that users can set it using submit-job."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2533",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2533/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2533/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2533/events",
"html_url": "https://github.com/apache/spark/pull/2533",
"id": 43905498,
"number": 2533,
"title": "SPARK-CORE [SPARK-3651] Group common CoarseGrainedSchedulerBackend variables together",
"user": {
"login": "tigerquoll",
"id": 8340922,
"avatar_url": "https://avatars.githubusercontent.com/u/8340922?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/tigerquoll",
"html_url": "https://github.com/tigerquoll",
"followers_url": "https://api.github.com/users/tigerquoll/followers",
"following_url": "https://api.github.com/users/tigerquoll/following{/other_user}",
"gists_url": "https://api.github.com/users/tigerquoll/gists{/gist_id}",
"starred_url": "https://api.github.com/users/tigerquoll/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/tigerquoll/subscriptions",
"organizations_url": "https://api.github.com/users/tigerquoll/orgs",
"repos_url": "https://api.github.com/users/tigerquoll/repos",
"events_url": "https://api.github.com/users/tigerquoll/events{/privacy}",
"received_events_url": "https://api.github.com/users/tigerquoll/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 8,
"created_at": "2014-09-25T10:59:27Z",
"updated_at": "2014-09-28T05:09:39Z",
"closed_at": "2014-09-28T05:09:39Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2533",
"html_url": "https://github.com/apache/spark/pull/2533",
"diff_url": "https://github.com/apache/spark/pull/2533.diff",
"patch_url": "https://github.com/apache/spark/pull/2533.patch"
},
"body": "from [SPARK-3651]\r\nIn CoarseGrainedSchedulerBackend, we have:\r\n\r\n private val executorActor = new HashMap[String, ActorRef]\r\n private val executorAddress = new HashMap[String, Address]\r\n private val executorHost = new HashMap[String, String]\r\n private val freeCores = new HashMap[String, Int]\r\n private val totalCores = new HashMap[String, Int]\r\n\r\nWe only ever put / remove stuff from these maps together. It would simplify the code if we consolidate these all into one map as we have done in JobProgressListener in https://issues.apache.org/jira/browse/SPARK-2299."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2532",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2532/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2532/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2532/events",
"html_url": "https://github.com/apache/spark/pull/2532",
"id": 43865688,
"number": 2532,
"title": "[Build]remove spark-staging-1030",
"user": {
"login": "scwf",
"id": 7018048,
"avatar_url": "https://avatars.githubusercontent.com/u/7018048?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/scwf",
"html_url": "https://github.com/scwf",
"followers_url": "https://api.github.com/users/scwf/followers",
"following_url": "https://api.github.com/users/scwf/following{/other_user}",
"gists_url": "https://api.github.com/users/scwf/gists{/gist_id}",
"starred_url": "https://api.github.com/users/scwf/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/scwf/subscriptions",
"organizations_url": "https://api.github.com/users/scwf/orgs",
"repos_url": "https://api.github.com/users/scwf/repos",
"events_url": "https://api.github.com/users/scwf/events{/privacy}",
"received_events_url": "https://api.github.com/users/scwf/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 2,
"created_at": "2014-09-25T06:25:37Z",
"updated_at": "2014-09-27T05:24:53Z",
"closed_at": "2014-09-27T05:24:53Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2532",
"html_url": "https://github.com/apache/spark/pull/2532",
"diff_url": "https://github.com/apache/spark/pull/2532.diff",
"patch_url": "https://github.com/apache/spark/pull/2532.patch"
},
"body": "Since 1.1.0 has published, remove spark-staging-1030."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2531",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2531/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2531/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2531/events",
"html_url": "https://github.com/apache/spark/pull/2531",
"id": 43850720,
"number": 2531,
"title": "[SPARK-3686][STREAMING] Wait for sink to commit the channel before check...",
"user": {
"login": "harishreedharan",
"id": 1481194,
"avatar_url": "https://avatars.githubusercontent.com/u/1481194?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/harishreedharan",
"html_url": "https://github.com/harishreedharan",
"followers_url": "https://api.github.com/users/harishreedharan/followers",
"following_url": "https://api.github.com/users/harishreedharan/following{/other_user}",
"gists_url": "https://api.github.com/users/harishreedharan/gists{/gist_id}",
"starred_url": "https://api.github.com/users/harishreedharan/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/harishreedharan/subscriptions",
"organizations_url": "https://api.github.com/users/harishreedharan/orgs",
"repos_url": "https://api.github.com/users/harishreedharan/repos",
"events_url": "https://api.github.com/users/harishreedharan/events{/privacy}",
"received_events_url": "https://api.github.com/users/harishreedharan/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 29,
"created_at": "2014-09-25T03:45:41Z",
"updated_at": "2014-09-26T05:58:41Z",
"closed_at": "2014-09-26T05:58:41Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2531",
"html_url": "https://github.com/apache/spark/pull/2531",
"diff_url": "https://github.com/apache/spark/pull/2531.diff",
"patch_url": "https://github.com/apache/spark/pull/2531.patch"
},
"body": "...ing for the channel size."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2528",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2528/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2528/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2528/events",
"html_url": "https://github.com/apache/spark/pull/2528",
"id": 43831245,
"number": 2528,
"title": "[SPARK-3476] Remove outdated memory checks in Yarn",
"user": {
"login": "andrewor14",
"id": 2133137,
"avatar_url": "https://avatars.githubusercontent.com/u/2133137?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/andrewor14",
"html_url": "https://github.com/andrewor14",
"followers_url": "https://api.github.com/users/andrewor14/followers",
"following_url": "https://api.github.com/users/andrewor14/following{/other_user}",
"gists_url": "https://api.github.com/users/andrewor14/gists{/gist_id}",
"starred_url": "https://api.github.com/users/andrewor14/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/andrewor14/subscriptions",
"organizations_url": "https://api.github.com/users/andrewor14/orgs",
"repos_url": "https://api.github.com/users/andrewor14/repos",
"events_url": "https://api.github.com/users/andrewor14/events{/privacy}",
"received_events_url": "https://api.github.com/users/andrewor14/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 12,
"created_at": "2014-09-25T00:44:21Z",
"updated_at": "2014-09-26T18:52:30Z",
"closed_at": "2014-09-26T18:52:30Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2528",
"html_url": "https://github.com/apache/spark/pull/2528",
"diff_url": "https://github.com/apache/spark/pull/2528.diff",
"patch_url": "https://github.com/apache/spark/pull/2528.patch"
},
"body": "See description in [JIRA](https://issues.apache.org/jira/browse/SPARK-3476)."
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2527",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2527/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2527/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2527/events",
"html_url": "https://github.com/apache/spark/pull/2527",
"id": 43821136,
"number": 2527,
"title": "[SQL][DOCS] Clarify that the server is for JDBC and ODBC",
"user": {
"login": "marmbrus",
"id": 527,
"avatar_url": "https://avatars.githubusercontent.com/u/527?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/marmbrus",
"html_url": "https://github.com/marmbrus",
"followers_url": "https://api.github.com/users/marmbrus/followers",
"following_url": "https://api.github.com/users/marmbrus/following{/other_user}",
"gists_url": "https://api.github.com/users/marmbrus/gists{/gist_id}",
"starred_url": "https://api.github.com/users/marmbrus/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/marmbrus/subscriptions",
"organizations_url": "https://api.github.com/users/marmbrus/orgs",
"repos_url": "https://api.github.com/users/marmbrus/repos",
"events_url": "https://api.github.com/users/marmbrus/events{/privacy}",
"received_events_url": "https://api.github.com/users/marmbrus/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 5,
"created_at": "2014-09-24T22:09:00Z",
"updated_at": "2014-09-27T05:27:18Z",
"closed_at": "2014-09-27T05:27:18Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2527",
"html_url": "https://github.
100 67018 100 67018 0 0 90543 0 --:--:-- --:--:-- --:--:-- 90564
com/apache/spark/pull/2527",
"diff_url": "https://github.com/apache/spark/pull/2527.diff",
"patch_url": "https://github.com/apache/spark/pull/2527.patch"
},
"body": ""
},
{
"url": "https://api.github.com/repos/apache/spark/issues/2526",
"labels_url": "https://api.github.com/repos/apache/spark/issues/2526/labels{/name}",
"comments_url": "https://api.github.com/repos/apache/spark/issues/2526/comments",
"events_url": "https://api.github.com/repos/apache/spark/issues/2526/events",
"html_url": "https://github.com/apache/spark/pull/2526",
"id": 43816326,
"number": 2526,
"title": "[SPARK-3681] [SQL] [PySpark] fix serialization of List and Map in SchemaRDD",
"user": {
"login": "davies",
"id": 40902,
"avatar_url": "https://avatars.githubusercontent.com/u/40902?v=2",
"gravatar_id": "",
"url": "https://api.github.com/users/davies",
"html_url": "https://github.com/davies",
"followers_url": "https://api.github.com/users/davies/followers",
"following_url": "https://api.github.com/users/davies/following{/other_user}",
"gists_url": "https://api.github.com/users/davies/gists{/gist_id}",
"starred_url": "https://api.github.com/users/davies/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/davies/subscriptions",
"organizations_url": "https://api.github.com/users/davies/orgs",
"repos_url": "https://api.github.com/users/davies/repos",
"events_url": "https://api.github.com/users/davies/events{/privacy}",
"received_events_url": "https://api.github.com/users/davies/received_events",
"type": "User",
"site_admin": false
},
"labels": [
],
"state": "closed",
"locked": false,
"assignee": null,
"milestone": null,
"comments": 4,
"created_at": "2014-09-24T21:17:44Z",
"updated_at": "2014-09-28T20:18:18Z",
"closed_at": "2014-09-27T19:23:05Z",
"pull_request": {
"url": "https://api.github.com/repos/apache/spark/pulls/2526",
"html_url": "https://github.com/apache/spark/pull/2526",
"diff_url": "https://github.com/apache/spark/pull/2526.diff",
"patch_url": "https://github.com/apache/spark/pull/2526.patch"
},
"body": "Currently, the schema of object in ArrayType or MapType is attached lazily, it will have better performance but introduce issues while serialization or accessing nested objects.\r\n\r\nThis patch will apply schema to the objects of ArrayType or MapType immediately when accessing them, will be a little bit slower, but much robust."
}
]
5 Homework
- Use the Users resource to
discover how many public repositories (
public_repos) I have (jblomo). - Use the Repositories resource
to discover which repositories (
full_name) I am only a member of. - Use the Repositories resource
to find the
logins of the contributors to themrjobrepository owned byYelp - Use the Commits resource
to find what the
messagewas of the first commit to thewebarch253repository owned byjblomo.
6 Show Work:slide:
- Text file in Pull Request
- Include answer and curl request used
- Long responses can be elided
7 Don't forget your Project
- October 30th (1/5 of your time is gone)
- There are 4 main parts:
- HTML to submit the form
- Handle
POSTtoserver/createto store redirect and return result - Handle
GETto short URL and redirect - Handle
GETto unknown URL and 404