10000 Redirect on upload from file causes IO error from file being closed · Issue #5530 · aio-libs/aiohttp · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Redirect on upload from file causes IO error from file being closed #5530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
MatthewScholefield opened this issue Mar 9, 2021 · 2 comments · May be fixed by #9201
Open

Redirect on upload from file causes IO error from file being closed #5530

MatthewScholefield opened this issue Mar 9, 2021 · 2 comments · May be fixed by #9201
Labels
enhancement

Comments

8000
@MatthewScholefield
Copy link
MatthewScholefield commented Mar 9, 2021 •
edited
Loading

🐞 Describe the bug
As seen from #1907, aiohttp takes ownership of passed files and closes them after writing. However, in the case of a 302 redirect, where I assume that aiohttp is re-performing the request, it attempts to re-read the data from the closed file.

💡 To Reproduce

Client code:

from aiohttp import ClientSession
with open(abs_storage_path(rel_path), "rb") as f:
    async with ClientSession() as session:
        async with session.put("http://localhost:5000/foo", data=f) as r:
            print("Response:", r.status)

Server code:

from flask import Flask, request, redirect

app = Flask(__name__)

@app.route("/foo", methods=['PUT'])
def redirct_route():
    return redirect("http://localhost:5000/bar", code=302)

@app.route("/bar", methods=['PUT'])
def put_route():
    print('Uploading data of size:', len(request.data))
    return 'success', 200

app.run(debug=True)

💡 Expected behavior
Successful PUT upload (or to not follow redirects and return a 302 response).

📋 Logs/tracebacks

    async with session.put("http://localhost:5000/foo", data=f) as r:
               │       │                                     └ <_io.BufferedReader name='../data/foo.dat'>
               │       └ <function ClientSession.put at 0x7faa1294de50>
               └ <aiohttp.client.ClientSession object at 0x7faa101d9a00>

  File ".../aiohttp/client.py", line 1117, in __aenter__
    self._resp = await self._coro
    │    │             │    └ <member '_coro' of '_BaseRequestContextManager' objects>
    │    │             └ <aiohttp.client._RequestContextManager object at 0x7faa101e7bc0>
    │    └ <member '_resp' of '_BaseRequestContextManager' objects>
    └ <aiohttp.client._RequestContextManager object at 0x7faa101e7bc0>
  File ".../aiohttp/client.py", line 492, in _request
    req = self._request_class(
          │    └ <class 'aiohttp.client_reqrep.ClientRequest'>
          └ <aiohttp.client.ClientSession object at 0x7faa101d9a00>
  File ".../aiohttp/client_reqrep.py", line 313, in __init__
    self.update_body_from_data(data)
    │    │                     └ <_io.BufferedReader name='../data/foo.dat'>
    │    └ <function ClientRequest.update_body_from_data at 0x7faa12c5e0d0>
    └ <aiohttp.client_reqrep.ClientRequest object at 0x7faa1015e580>
  File ".../aiohttp/client_reqrep.py", line 519, in update_body_from_data
    size = body.size
           │    └ <property object at 0x7faa12c8ecc0>
           └ <aiohttp.payload.BufferedReaderPayload object at 0x7faa1015e730>
  File ".../aiohttp/payload.py", line 362, in size
    return os.fstat(self._value.fileno()).st_size - self._value.tell()
           │  │     │    │      │                   │    │      └ <method 'tell' of '_io.BufferedReader' objects>
           │  │     │    │      │                   │    └ <_io.BufferedReader name='../data/foo.dat'>
           │  │     │    │      │                   └ <aiohttp.payload.BufferedReaderPayload object at 0x7faa1015e730>
           │  │     │    │      └ <method 'fileno' of '_io.BufferedReader' objects>
           │  │     │    └ <_io.BufferedReader name='../data/foo.dat'>
           │  │     └ <aiohttp.payload.BufferedReaderPayload object at 0x7faa1015e730>
           │  └ <built-in function fstat>
           └ <module 'os' from '/usr/lib/python3.9/os.py'>

ValueError: I/O operation on closed file

📋 Your version of the Python

$ python --version
Python 3.9.1

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.4
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: .../python3.9/site-packages
Requires: yarl, attrs, typing-extensions, chardet, async-timeout, multidict
$ python -m pip show multidict
Name: multidict
Version: 5.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: .../python3.9/site-packages
Requires: 
Required-by: yarl, async-asgi-testclient, aiohttp
$ python -m pip show yarl
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: .../python3.9/site-packages
Requires: idna, multidict
Required-by: aiohttp

📋 Additional context
Relates to the client aspect of aiohttp.
@ghost
Copy link
ghost commented Mar 23, 2021 •
edited by ghost
Loading

I'm facing the same error, though my case is a 307 instead of 302. The workaround I'm using for now is #3654 (comment)

@jesseanttila-cai jesseanttila-cai mentioned this issue Mar 31, 2021
Open
@vadim-su vadim-su mentioned this issue Jun 10, 2024
Closed
1 task
@Dreamsorcerer
Copy link
Member

This appears to be pretty difficult to do correctly.
Currently the file gets closed when the connection is closed.
I was hoping we could use expect100 as some kind of solution, but this doesn't resolve the problem and reveals a limitation on the server-side:

The server-side code doesn't have an easy way to send the 302 without first sending a 100-Continue (maybe you can use a custom expect handler, but it would then need to know about all your endpoints, in order to know whether they can be resolved without further data. Maybe we could improve this somehow to only send 100-Continue when a handler tries to read the body?).

@GLGDLY GLGDLY linked a pull request Sep 19, 2024 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
2 participants
@Dreamsorcerer @MatthewScholefield
0