Headless static (svg, png) image generation #538
Comments
|
@mattpap do you have any thoughts about what it would take to expose the phantomjs PNG generation we do with the test script to users? |
|
But using the phantom approach we have an image with the buttons and uncentered... so probably we need to pre-process the plot to render properly... just thoughts.... |
|
Right it won't work as-is. I can think of a few possible ways to use phantoms:
There may be others too. |
|
Update: it does look as though node-canvas may work for this, I am going to start some preliminary experimentation. |
|
Initial experiments on OS X are encouraging that |
|
Getting BokehJS working in node.js is a first prerequisite, I have made a separate issue for that task, |
|
we should rename this issue - there is no reason to involve the server at all here right? |
|
@hhuuggoo, the idea is to be able to produce plots on the server rather than having them rendered in the browser. This can be useful when you need to generate plots to be used in some context other than display in the browser (e.g., generation of a PDF report). |
|
@abastardi yes but in general you will be able to generate images from python, whether that's in the server, or in an interactive session or IPython notebook makes no difference. |
|
Got it. It sounded like @hhuuggoo meant this shouldn't happen on the server. |
|
I see that .8 has shipped but this issue is still open. Is there an update as to the disposition of this capability? |
|
@scherrey Unfortunately not. Here is a message I sent recently the Bokeh mailing list on 2015/03/04: Programmatic generation of static images is definitely something we would very much like to support. Unfortunately, it is also a surprisingly difficult proposition. The first observation is that browsers canvas implementation only generate raster images, and don't help with SVG or PDF. Worse, browser security models mean that getting the data out is also very problematic. You can follow this history of this topic in a few issues: MPL support is currently provided by a third party library, If the I wish I had some better news to give you. It is my earnest hope that we will have a solution in 2015 but I can't provide any better estimates at this time. |
|
I'm probably out of my depth here, but would it be easier and nearly as useful to generate SVGs via a javascript library in the browser? My main concern with adopting bokeh for analysis and plotting would be that I eventually need to produce publication-ready or presentation-ready graphics. To minimize file sizes and maximize quality, I prefer to work with vector formats (eps, pdf or possibly svg). But as I understand it, bokeh cannot currently produce vector graphics. This would be a barrier for me, but I don't mind much whether the impasse is solved on the server side or browser side. e.g., I would be satisfied if I could develop plots in an IPython notebook and then export them as vector graphics from there. In a quick search on this subject, I came across several javascript libraries that can generate SVG output using similar calls to drawing on an HTML5 canvas (which I assume is what bokeh does). I haven't investigated these very far, but I think they work either by accepting standard canvas drawing commands and then translating them into SVG elements, or they provide an alternative API which can be used both for drawing on the canvas and generating SVGs. Here are some of the key libraries I found: Fabric.js was recommended in a couple of stackoverflow posts on this subject (1, 2). It seems to be one of the most mature options. canvas-svg and canvas2svg were also mentioned in those stackoverflow posts but seem to be less mature. two.js offers what may be a slightly different path forward -- a single API for drawing either to the canvas or to SVGs. I imagine this would block the two-way communication needed by Bokeh. Another option, hinted at in yet another stackoverflow post might be to write graphs in the browser as SVG instead of canvas drawings. This would provide natural support for generating SVG files. This would also use a higher level interface than the canvas, which might simplify the interaction code (e.g., vertices could be native, clickable objects in the browser). Of course this would require a huge rewrite of the front end. |
|
Hi @mfripp thanks for the researched and informative post. The landscape has definitely changed and we are in fact already planning to look into some of those options mentioned. However, by themselves those libraries don't solve the problem. People want to be able to do For headless output with JS libraries to work, Bokeh would need to be able to be run in Node.js or similar. But last time we tried, BokehJS was too big and unwieldy and we could not get it to run. So a possibility is to split up BokehJS so that (hopefully) a minimal part can be to run in Node.js (just enough to generate . And in fact, this absolutely must happen for a variety of reason, and so work is under way. But that is a very difficult architectural refactor to make that happen in a way that is as transparent and as least disruptive as possible. It will probably not be complete for a few months, Completely unmentioned yet is that whatever solution we adopt of this has implications for being able to display math text in the saved output. Depending on the route we take, perhaps we have to give that up completely. To summarize, the core of the problem is that there are actually a few big pieces that interact in non-trivial ways. Some of the pieces have several possible options, none of which are very good or pleasant in some way. Or they may be blocked by major work in another area (splitting up BokehJS). |
|
Hi @bryevdv, thanks for your comments on this. Sorry for the slow response. I've been thinking a little more about this, but didn't get a chance to post until now. From the user's perspective I can see three new features here:
These are mostly independent, because any code written to provide features 1 and 2 in a web browser should also run in a server-side javascript environment (since you'll need a pretty robust javascript environment already). For the needs of scientific users, I would rank 1 as very important, 2 as moderately important and 3 as interesting but not particularly important. I would also say that feature 3 is not useful without feature 2, and feature 2 is not useful without feature 1 (other users may want to produce raster images of plots programmatically, but for scientific applications, users will immediately want vector graphics). Given that any code you write to implement features 1 and 2 in a full web browser would also be usable in a server-side environment, why not start with 1 and 2 in a web browser and then move them to the server environment along with the rest of BokehJS later? This would be an especially attractive approach since it will take an unknown length of time to get BokehJS running on the server side. i.e., the high-priority features must work in a browser, can be implemented more easily in a browser, and can be migrated eventually from the browser to the server, so why wait for the server implementation before developing them? |
|
@mfripp They are somewhat independent. However, BokehJS is already very large. Adding in yet another dependency and chunk of code to the current monolithic BokehJS, for a feature that is not actually the one that is most often requested [1] is not something I can justify. Once BokehJS is split up (we are about to have a kickoff on that front) it becomes more reasonable to think about adding an in-browser capability as an optional component, that users only have to pay for if they use. This is even more true if a new contributor wanted to help push on that front. (One of the biggest motivations for splitting up BokehJS is to enable outside contributions that can be developed independently and still integrated easily.) [1] by far most users ask us for option 2: programmatic, headless generation. This is by any metric I can think of, the higher priority feature. |
|
Hi bokeh! i've been working on nbpresent, and have made some decent progress on headless bokeh rendering in notebooks. Here's a sample output. Actually, it's the first time it worked, fully. I have given up on phantomjs in favor of ghost.py, which gives much finer-grained access to the underlying qt api. Glad for any feedback... and maybe some thoughts on how to crank up the resolution on bokeh renders, as I am really gunning for print-quality output (>600dpi). I am totally fine with adding bokeh-specific fix-all lines, or giving users ways to apply specific patches! |
|
We actually have an open PR already: #3047 This approach is a start to using node.js to create static raster images but it is hoped that we can leverage canvas-cairo to produce SVGs. If not, I think we will resort to implementing an alternative partial pycairo-based backed. Long term we would like to use MPL as a backend as well. Being completely honest, I am personally fairly negative on the long term utility of "screenshot" approaches like the one above, I think they will be fragile by nature and of course limited to raster images in any case. |
|
Great, glad that's working out! Will there be a way to access the vector format in the frontend, a la |
|
Brainstorming here but what about supporting png & jpeg creation by compiling those open source C libraries to asm.js and running in phantomjs? We've done a little bit of C++ coding that ran under asm.js with good result. Any ideas why this wouldn't work? |
|
@scherrey I think we want a solution that would also work without phantomjs - there's kind of a lot going on in this issue though, sowe may be talking about different things. |
|
@scherrey you may be right - I'm doing some experiments with phantomjs at the moment. |
|
@bryevdv pointed out that phantomjs won't be a solution until it supports TypedArrays. I'm poking around the currently in-development headless chrome. |
|
Hi Team, I really love bokeh and am currently using it to host a grid of plots 6x11. I can load the first 5 plots just fine, clicking 1 plot at a time but then my browser (chrome) crashes if I try to open any other plot. It would be really great to be able to export the plots to png without the need of the 'save' tool. At this point I can try to create a hack by trying to use some third party tool (phantomjs) or just switch to another package (matplotlib, etc), which I don't really want to do because I really really like Bokeh :( I see that this thread is over 2 years old... so it is safe to assume your team has no real interest in providing this feature? Thanks for your time. -- Grateful Bokeh User |
|
@tux2014, I'm confused, the last comment on this thread was 18 days ago. Also "your team" seems like a bit of a misconception. This is open source. You are part of the team too. We welcome your help in getting this feature done. |
No that is not a safe assumption at all. What is actually the case, is that it is extremely difficult from a technical perspective, with lots of potentially unpleasant but also unavoidable trade-offs, and that it competes with many other very high priorities for extremely limited human resources, from a project management perspective. |
|
+1 this would be so useful |
|
+1 The lack of static (SVG) image support is the main reason I'm not using bokeh right now. |
|
+1 Needed here as well |
|
Any updates to share? I would appreciate a sort of roadmap of what needs to be done before this is put into place. |
|
Please please please =) SVG and PNG output would be amazing |
|
I believe chrome headless is going to be the right way to do this and that project is moving very quickly with decent linux support already. There is a lack of resources on this at the moment, but hopefully that will be changing in the coming months. |
|
I don't know the current state of using headless chrome, but wkhtmltopdf (github) could be another option. It also may be useful in the short term for people to use as a workaround; I recently used it to programmatically generate a few hundred bokeh pdfs. I realize this issue is titled Current Limitations:
After downloading the alpha build, I was able to use the following to generate a pdf:
It looks like there's also a handy python wrapper called pdfkit (github). |
|
I have an open PR for generating PNGs that is seeking comment: #6189 |
|
This would be great for the purposes of unit testing. |
|
re-opening issue until SVG PR is also merged. |
|
Noting that current PRs support headless, static PNG, and static SVG (via toolbar button tool) A separate issue will be opened to extend SVG to headless programmatic usage. |
|
@bryevdv @canavandl @birdsarah This is great news, thank you so much for your effort! |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Enable generation of PNG images directly on the server, similar to the Vega.js headless mode (https://github.com/trifacta/vega/wiki/Headless-Mode).
The text was updated successfully, but these errors were encountered: