YANUB: yet another (nearly) useless blog

Six month ago, I wrote a lengthy post about my new computer hurting my eyes. I haven't made any progress with that, but I've accidentally upgraded my work computer from Kernel 4.4 to 4.8 and the nvidia drivers from 340.96-4 to 375.66-2. Well, my work computer now hurts, I've switched back to the previous kernel and drivers, I hope it'll be back to normal.

Any ideas of something specific that changed, either between 4.4 and 4.8 (kernel startup code, default framebuffer modes, etc ?), or between the 340.96 and the 375.66 drivers ? In any case, I'll try that specific combination of kernels/drivers home to see if I can get it to a useable state.

Update, July 23rd

I reverted to the earlier drivers/kernel, to no avail. But it seems in fact the problem with my work computer is linked to an allergy of some kind, since antihistamine drugs have an effect (there are construction works in my lab, perhaps they are the cause ?). No idea still for my home computer, for which the problem is definitely not solved.

The day has finally come again to release a new version of my plotting program, ctioga2. Version 0.14 features, among others:

• a new binner, useful to make histograms;
• a command to hide elements already presents, that can be very useful to make animations;
• a whole set of functionalities to make it easier to draw complex grids, including easy ways to style grid elements and a way to switch to the next grid element (picture);
• a whole series of command-file functions to obtain informations about loaded datasets;
• a way to draw legends separately by hand;
• and more, including bugfixes, better error reporting, such as when some of the numbers are infinite, and much much faster debug output.

As usual, the best way to update the package is running:

~ gem update ctioga2

An updated Debian package should find its way to the archive later on today. Have fun !

Quite a bit of work has gone into ctioga2 since my last post. Among the most interesting features, ctioga2 now features:

• full windows support, including proper file types associations
• fill syntax as powerful as gnuplot's
• an impulses plot style
• unclipped plot elements along with full control of the curves depth: in front of axes, behind background lines...
• control of symbol size through a Z axis

The latter feature proved actually instrumental in a research project under way in my lab (which is why I wrote it in the first place). It is demonstrated in the image above, made using this code.

I've also started the gnuplot versus ctioga2 gallery I was speaking about in my last post, where I try to reproduce most of gnuplot's examples using ctioga2; it can be found here. While I don't like so much the look of the gnuplot graphs, and I try to keep the look of ctioga2's as close as possible to that look, it proved invaluable in detecting small bugs, fixing small glitches and providing inspiration for new features.

In the meantime, my todo-list for ctioga2 doesn't seem to get any smaller. Let's hope I'll implement them fast enough ! Keep posted...

While in the process of polishing up a paper, I had to draw figures using a well-known proprietary program for drawing chemical formula whose name I will not write down, out of decency (but you know which one it is anyway). I'm using the mac version (yes, no version for Linux) which has a neat glitch in that the "Save As" button does not work at all: no way to export your chemical formula as a SVG/PDF/PNG picture or whatever ! So I resorted to the neat Print as PDF feature of macintosh. I had to edit the resulting PDF files by hand as some of the colors were wrong (great !), and when I tried to include the PDF, I realized they had a white background... No way !

So I dug in ImageMagick's convert documentation, and came up with the following code that converts the PDF file into a PNG with a transparent background:

convert -density 600 figure.pdf -channel alpha \
  -fx '((r == 1 && g == 1 && b == 1) ? 0 : 1)' figure.png

There are still some few points that look white but are probably not that white (due to antialiased rendering of the PDF file ?). There are many more things that can be done using the -fx operator, this page was helpful to me !

Edit: while the alpha channel seems to be on by default for PDF files, it is not necessarily the case for all images. If the above doesn't work, try adding -alpha Set before the -fx bit.

I've just released version 0.5 of ctioga2, my plotting program. While there is still quite a lot of improvements I'd like to get in some day, the past few days have seen a lot of activity. Probably the most important change is the new space-separated command file format, much simpler than the old one (which is still supported, although marked clearly as deprecated now). Another important change for the users is the large improvement of error reporting: in most cases, ctioga2 pinpoints the exact location where things go wrong; it also features reporting on LaTeX errors. The last important feature is also the introduction of real contour plots, such as the one below, coming from the documentation:

Now, I feel like ctioga2 is coming close to gnuplot in terms of features and ease of use. Part of my plans for the website is to create a gnuplot mockup gallery, in which I would take as many examples from the gnuplot website as possible (ie all the ones ctioga2 can reproduce) and compare the code and look of both.

Among features I'd like to implement for ctioga2, here are the ones that I think are the most important ones:

• plot styles (histograms, bars, impulses), in the spirit of the plot with instruction of gnuplot
• complex axes: broken axes and linked axes
• full control of the tick labels and positions (alternative formats, placing ticks at multiples of π, and labelling them as such...)
• full styling using CSS-like stylesheets
• a windows port would be nice, too

But maybe you have other ideas you'd like to see before that ?

I have been using xfig for ages to draw figures, and I appreciate it, especially combined with a small script I wrote based on an original idea by Seb Desreux at H&K, fig2ps, that allows painless integration of LaTeX code inside xfig figures.

But recently, it seems that I've hit limitations in xfig. Mainly, if it is great when you want to produce sober figures, whenever you want to do something fancier, including shadings and the like, frustration comes in quickly (since it is purely impossible). I've spent a while looking for a decent alternative, until I had a look again at pgf, which since turned into TikZ. That was amazing. It integrates painlessly within LaTeX and is simply great to work with. Drawing complex diagrams with noes and complex relations between them is trivial. It handles moving the nodes very gracefully, and styling can be done in a CSS-like fashion (though it definitely isn't CSS), which means you can first concentrate on structure and then turn your graph into something nice, while for all the other programs I know, you have to handle both structural and stylistic aspects at the same time.

Sure enough, you need to like the command-driven approach, as mice won't come in too useful here... As a side note, I wanted to express my admiration of Till Tantau's (TikZ's author) mastery of (La)TeX: TikZ can be seen as an interpreter of a simple graphical language written in pure TeX, which is, according to my humble experience with dirty trick in TeX, is simply amazing. Many thanks, Till !

First, some historical details: some 6 years ago, I discovered Ruby together with Tioga, a great library for producing plots. That immediately resulted in my first Ruby program, ctioga. But as years passed and more features were forced into ctioga, I was compelled to see my initial design mistakes, and think about a complete rewrite.

This is where ctioga2 was born, some time at the beginning of 2009. I completely stopped using the old ctioga a few month after starting to work on ctioga2, but it took ages before I finally came up with a decent enough documentation to release and announce publicly. But now, it's done, and the first public release of ctioga2 is out, still fresh !

Here are a few highlights of the differences:

• For me, it's day and night: adding new commands is painless: I only need to create an object with the appropriate code; registering is done automatically.
• Now, command-line switches also accept options looking like this /option value that allow many small tweaks I had never had the courage to write for the old ctioga as it was way too painful to add new commands there.
• ctioga2 can be driven both from command-line and using command files, in the style of gnuplot, which is the reason why I call it polymorphic.
• More importantly, I finally had the possibility (and more or less compelling reasons, to be found in a paper currently in press) to implement 3D data displays in ctioga2, in the form of color maps (that's what is shown here).
• There are still a few things that could be done by ctioga and that miss from ctioga2, such as histograms. When I have some time and/or motivation...

In any case, you can make yourself an opinion of ctioga2 by looking at the galleries.

Enjoy !

pdfinfo is one of the utility programs bundled with xpdf that gives useful information about PDF files, including its size. However, the size is written in Postscript points, and I can't say I'm that fluent in this unit to actually know how big that is. So here is a shell function that converts the output of pdfinfo into centimeters. Enjoy !

pdfinfom () {
    pdfinfo "$@" | perl -p -e \
 's#(\d+)\s*x\s*(\d+)\s*pts#(sprintf "%.2f", ($1*2.54/72))." x ".(sprintf "%.2f", ($2*2.54/72))." cm"#e'
}

I make heavy use of pdfcrop (the version by Heiko Oberdiek included in texlive) for printing, as it allows to remove all the white spaces around the text before nupping and printing. However, with recent ghostscript, the bounding box detection is dreadfully slow, anything from one to ten seconds for a single page ! When you have a 60 pages article to print, it can be quite tedious...

Well, I've just discovered that the --trim option of pdfnup can do very similar things too. It removes a given amount of margins on the side, with the syntax 'left bottom right top'. Most of the time, a well-thought pdfnup --trim ... will work as well as pdfcrop while in the meantime being much faster...

I haven't posted for a long while about ctioga, but that does not mean that development has stopped. It goes on, and features get added basically every single time I need them. Recent additions make it possible to tune a little more the look of the legends and add pure text lines in them.

I've however realized that now, what ctioga is missing the most is real flexibility over the coordinates. For now, all coordinate positions are either relative to the frame (such as for --inset) or to the graph (for --draw). What I really wish is a way to mix all kind of coordinates: real size (1cm), frame position and graph position. That will require quite a fair deal of work, of course, but that shouldn't be too difficult... I'll go on posting about it here.

I had some trouble finding out how to convert a color PDF to a grayscale one. I found a post on an Ubuntu forum giving the following recipe:

pdf2ps -sDEVICE=psgray original.odf
ps2pdf original.ps new.pdf

This works, but this has the unpleasant side effect to increase the size of the PDF and to lower the quality (for a reason which escapes me). So I looked again in the magic pdfwrite parameters, and I came up with the following solution, which works on my box (though the above page suggests it shouldn't):

gs -sOutputFile=biniou.pdf -sDEVICE=pdfwrite \
  -sColorConversionStrategy=Gray -dProcessColorModel=/DeviceGray \
  -dCompatibilityLevel=1.4 Plot.pdf < /dev/null

This works fine, does not lower the quality of the produced PDF and in the case I tried even made it smaller...

I use a combination of pdftk and pdfnup, from the pdfjam package to arrange (PDF) graphs produced by ctioga into an array for printing. However, I find that a simple

pdfnup --nup 3x3 concatenated_graphs.pdf

does not produce an A4 output (even with --paper a4paper), but a PDF who has the same size as individual graphs. Painful. It did take me long to realize it is a feature of pdfnup, who asks pdfTeX to have the paper size adjusted to that of the input when the layout is symmetric and no orientation has been provided. To disable that, and get a real A4 paper, specify the orientation:

pdfnup --orient portrait --nup 3x3 concatenated_graphs.pdf

Some time ago, I was delighted at the power of Ghostscript. Well, it saved the day again, with an article that didn't want to get printed. Quite naively, I always thought that doing first pdf2ps (or pdftops, based on Xpdf, that often produces smaller and nicer postscript) and then ps2pdf was a good way to deal with unfriendly PDF files. But, in real, nothing is better than working without intermediates. The following produces a pretty neat PDF file:

gs -sOutputFile=output.pdf -sDEVICE=pdfwrite \
  -dCompatibilityLevel=1.4 input.pdf < /dev/null

As side benefits, as this command also embeds postscript fonts, the output looks better in xpdf, and it is decently smaller !

I had to make some ID card pictures for my daughter, and the public machine that I usually use for that was down this morning. So, I decided that I would simply take a picture of her, arrange that into a 13x10cm picture, and have it developed (along with other pictures). My first intention was to use the gimp for that, but, I never learned how to use it, and I wouldn't have a simple way to do that again should I need to.

So I went for the command-line approach, with the tool montage from ImageMagick. The pictures should measure 35x45mm in the end. I used the following command-line options:

• -tile 3x2 6 small picture fit on the target paper
• -geometry 700x900 the size of an individual image (not the one of the whole montage); it is a multiple of 35x45 and gives a neat resolution in the end
• -border 8%x5% -bordercolor white to add a neat white border around the pictures. The numbers are computed so that the overall picture respects (around) the 13x10 aspect ratio

In short, that gives this:

montage -tile 3x2 -geometry 700x900 -border 8%x5% \
  -bordercolor white Morgane.jpg +clone +clone \
  +clone +clone +clone target.jpg

The +clone just tell montage to reuse the last picture used.

I've just been writing some cards for the birth of my daughter, and I've been looking for nice-looking fonts for LaTeX. I'm using the texlive distribution (Debian packages). So I've installed the texlive-fonts-extra package, and I had a quick look at all the pdf files therein:

for f in `dpkg -L texlive-fonts-extra | grep pdf`; do \
[ -h $f ] || xpdf $f; done

The [ -h $f ] || blurb is here to prevent reading the same file twice via a symlink. I found many interesting things, but not that many "fancy" fonts - apart from one package, aurical, that provides neat handwriting-like fonts. Attached is what I obtained with the \Fontlukas command. Hope you'll find this useful !

A little earlier in the evening, I had to send a document to my parents including pictures. Though the final version definitely should have high-quality pictures, I was forced to see that the version I sent was maybe slightly too big (5MB by email, when the receiver doesn't have a very fast connection, that is painful). After some experimentations with pdftops (from xpdf) and ps2pdf (from ghostscript) that didn't give satisfying results, I tried to use ghostscript directly:

~ gs -sOutputFile=biniou.pdf -sDEVICE=pdfwrite \
  -dCompatibilityLevel=1.4 FairePart.pdf  < /dev/null

What a surprise ! The output did shrink by a factor of 4:

~ ls -lh FairePart.pdf biniou.pdf
-rw-r--r-- 1 vincent vincent 1.1M 2007-10-10 22:56 biniou.pdf
-rw-r--r-- 1 vincent vincent 5.7M 2007-10-10 18:14 FairePart.pdf

A quick check with pdfimages combined with identify shows that all the images kept the same resolution. That must be a question of JPEG conversion, or something of this spirit... Then, another try is to add the -dPDFSETTINGS=/screen option to the command-line:

~ gs -sOutputFile=biniou.pdf -sDEVICE=pdfwrite \
  -dPDFSETTINGS=/screen -dCompatibilityLevel=1.4 
  FairePart.pdf  < /dev/null

Then, output file is minuscule:

~ ll biniou.pdf 
-rw-r--r-- 1 vincent vincent 34K 2007-10-11 00:09 biniou.pdf

The downside is, the output is pretty much ugly (well, you wouldn't have hoped anything good with a win of a factor of 150). Images went down from 2576x1932 to 322x241 or even smaller (depends on the physical size of the image). -dPDFSETTINGS=/ebook gave a slightly better output (for 70K), but still not good for my case... So I tweaked the pdfwrite parameters by hand:

 gs -sOutputFile=biniou.pdf -sDEVICE=pdfwrite \
  -dColorImageDownsampleType=/Bicubic -dColorImageResolution=300 \
  -dDownsampleColorImages=true -dCompatibilityLevel=1.4 \
  FairePart.pdf  < /dev/null

This gave me a pretty nice result. And this also shows that my image resolution was way too big anyway - 300 dpi is probably the best I'll get when printing... and the file produced is still ridiculously small (172K) ! I now start to realize the power of ghostscript, and I thank its authors for it !