10000 [Bug]: `TypeError: 'async_generator' object is not iterable ` when processing crawl requests with multiple URLs · Issue #1066 · unclecode/crawl4ai · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

[Bug]: TypeError: 'async_generator' object is not iterable when processing crawl requests with multiple URLs #1066

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
zinodynn opened this issue May 2, 2025 · 0 comments · May be fixed by #1074
Open

[Bug]: TypeError: 'async_generator' object is not iterable when processing crawl requests with multiple URLs #1066

zinodynn opened this issue May 2, 2025 · 0 comments · May be fixed by #1074
Labels
🐞 Bug Something isn't working 🩺 Needs Triage Needs attention of maintainers

Comments

@zinodynn
Copy link
zinodynn commented May 2, 2025

crawl4ai version

0.6.0-r2

Expected Behavior

Many results should be returned in the response when I send a request for a task with many URLs.

Current Behavior

Encounter some errors like this:

Traceback (most recent call last):
  File "/workspace/deploy/docker/api.py", line 445, in handle_crawl_request
    "results": [result.model_dump() for result in results],
                                                  ^^^^^^^
TypeError: 'async_generator' object is not iterable

When I send a request with many URLs for a task , like this :

curl -X POST http://localhost:11235/crawl \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com","https://www.google.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"scraping_strategy":{"type":"WebScrapingStrategy","params":{}},"exclude_social_media_domains":["facebook.com","twitter.com","x.com","linkedin.com","instagram.com","pinterest.com","tiktok.com
","snapchat.com","reddit.com"],"stream":true}}}'

Is this reproducible?

Yes

Inputs Causing the Bug



Steps to Reproduce




Code snippets




OS


Linux


Python version


3.12.10


Browser


No response


Browser version


No response


Error logs & Screenshots (if applicable)


No response

      		
    

      
  





        



            
            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @zinodynn
zinodynn




            added
      

  🐞 Bug

  Something isn't working

      

  🩺 Needs Triage

  Needs attention of maintainers

  labels


      May 2, 2025

    

  



  

  

    
  


  

      

    @zinodynn
zinodynn



    linked a pull request
    
      May 5, 2025
    

    that will
    close
    this issue





  


  

      
   Open

  





  
  

      
        
      
      

  
    
    
  


    6 tasks
  





  













  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    🐞 Bug

  Something isn't working
  

    🩺 Needs Triage

  Needs attention of maintainers








      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    

      
        

          
    

    Development
  




            
  
Successfully merging a pull request may close this issue.





    


  
          
      Fix: request /crawl with stream: true  issue #1066

    
      zinodynn/crawl4ai
    





      
    

  




      

    

  
      

  

    

      1 participant
    

    

        
          @zinodynn 
    

  






    

    




    






  

          

        



    



  





    

  


  


          



    



  

    

    

    





    


    

    

  



0