8000 Normalize urlparse fragment behavior across all supported Python versions by irskep · Pull Request #686 · Yelp/mrjob · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Normalize urlparse fragment behavior across all supported Python versions #686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 5, 2013
Merged

Normalize urlparse fragment behavior across all supported Python versions #686

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0a8e554
Normalize urlparse behavior for s3, s3n, hdfs, mapreduce url schemes …
irskep Aug 3, 2013
90dce48
Better solution to py2.7.4 urlparse behavior fix
irskep Aug 5, 2013
9062519
Support allow_fragments in mrjob.parse.urlparse()
irskep Aug 5, 2013
e2318c7
Reinstate @wraps, future proof against extremely unlikely func sig ch…
irskep Aug 5, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 25 additions & 12 deletions mrjob/parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,18 +79,31 @@ def parse_s3_uri(uri):


@wraps(urlparse_buggy)
def urlparse(*args, **kwargs):
"""A wrapper for :py:func:`urlparse.urlparse` that handles buckets in S3
URIs correctly. (:py:func:`~urlparse.urlparse` does this correctly sometime
after 2.6.1; this is just a patch for older Python versions.)"""
components = urlparse_buggy(*args, **kwargs)
if components.netloc == '' and components.path.startswith('//'):
m = NETLOC_RE.match(components.path)
return ParseResult(components.scheme, m.group(1), m.group(2),
components.params, components.query,
components.fragment)
else:
return components
def urlparse(urlstring, scheme='', allow_fragments=True, *args, **kwargs):
"""A wrapper for :py:func:`urlparse.urlparse` with the following
differences:

* Handles buckets in S3 URIs correctly. (:py:func:`~urlparse.urlparse`
does this correctly sometime after 2.6.1; this is just a patch for older
Python versions.)
* Splits the fragment correctly in all URIs, not just Web-related ones.
This behavior was fixed in the Python 2.7.4 standard library but we have
to back-port it for previous versions.
"""
# we're probably going to mess with at least one of these values and
# re-pack the whole thing before we return it.
# NB: urlparse_buggy()'s second argument changes names from
# 'default_scheme' to 'scheme' in Python 2.6, so urlparse_buggy() should
# be called with positional arguments.
(scheme, netloc, path, params, query, fragment) = (
urlparse_buggy(urlstring, scheme, allow_fragments, *args, **kwargs))
if netloc == ' 8000 ;' and path.startswith('//'):
m = NETLOC_RE.match(path)
netloc = m.group(1)
path = m.group(1)
if allow_fragments and '#' in path and not fragment:
path, fragment = path.split('#', 1)
return ParseResult(scheme, netloc, path, params, query, fragment)


### OPTION PARSING ###
Expand Down
2 changes: 1 addition & 1 deletion tests/test_parse.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,7 +540,7 @@ def test_urlparse(self):
self.assertEqual(urlparse('s3://bucket/path'),
('s3', 'bucket', '/path', '', '', ''))
self.assertEqual(urlparse('s3://bucket/path#customname'),
('s3', 'bucket', '/path#customname', '', '', ''))
('s3', 'bucket', '/path', '', '', 'customname'))
self.assertEqual(urlparse('s3://bucket'),
('s3', 'bucket', '', '', '', ''))
self.assertEqual(urlparse('s3://bucket/'),
Expand Down
0