8000 mrjob should use file:/// with local Spark masters · Issue #1985 · Yelp/mrjob · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
mrjob should use file:/// with local Spark masters #1985
New issue
New issue
Closed
Closed
mrjob should use file:/// with local Spark masters#1985
Labels
Cleanup
Milestone
 
v0.6.8
@coyotemarin

Description

@coyotemarin
coyotemarin
opened on Feb 8, 2019

spark-submit from the pyspark package is capable of taking absolute paths for input files, etc. when running on a local master. However, the Spark installation on EMR (at least the Spark 2.3.1 installation on 5.16.0 AMIs) can't; if you don't use file:///, it interprets them as HDFS paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Cleanup

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions
      0