SEGV during vips_shutdown #1792
Labels
Comments
|
Ah, how annoying. I wonder what's racing? I'll revert that init change locally and see if the gdb crash has been fixed. If it has, perhaps that's the simplest fix. nip2 needs that background thread -- it can have several thousand sinkscreens active at once, and there has to be a mechanism to make sure that only one is executing at any time. |
|
After further debugging this turns out to be a issue on our side, sorry for the noise. Ah, didn't know nip2 uses this too, I was grepping for |
jcupitt
added a commit
that referenced
this issue
Aug 28, 2020
we were starting the sinkscreen background thread during vips_init() -- instead, start it on first use see #1792
|
I'm seeing this on MacOS 10.15.6 with libvips installed via Homebrew. Any idea how long it will take to make it into the Homebrew cask? |
|
The change is in the head of the 8.10 branch, so it'll be in 8.10.1. But I don't know if it'll fix your crash. Do you have a testcase? |
|
Hrm actually the only way I can reproduce it is when Vips is required inside Spring in a Rails project. So here is a set of steps to repro from scratch (you'd need to have the Rails gem installed in your global Ruby env and Homebrew installed): However if I comment out the spring gem lines in Gemfile and load the console and exit then it exits cleanly: There is probably a way to isolate what Spring is doing outside of Rails but at least this gives you a reliable way to replicate the real world issue. Note that it's development mode only and only throws the error when exiting the console or server so it's not a blocking issue for us right now, just a little concerning as to what the memory leak is etc. UPDATE: I forgot to indicate lib/gem versions: From Gemfile.lock |
|
The issue I had was a call to
fwiw, this issue can be reproduced with: // compile with
// gcc -g -Wall issue-1792.c `pkg-config vips --cflags --libs` -o issue-1792
// run with
// ./issue-1792
#include <pthread.h>
#include <sys/types.h>
#include <vips/vips.h>
void *dummy_thread(void *data) {
return NULL;
}
int main(int argc, char **argv) {
pthread_t pid;
int rv;
if (VIPS_INIT(argv[0]))
vips_error_exit("vips_init() failed");
switch (fork()) {
case -1:
vips_error_exit("fork() failed");
case 0:
rv = pthread_create(&pid, NULL, dummy_thread, NULL);
if (rv != 0)
vips_error_exit("pthread_create() failed");
break;
default:
exit(0);
}
pthread_join(pid, NULL);
return 0;
} |
|
Yes, fork() is fundamentally incompatible with any threading. Even things like mutexes will break horribly. You can fork() and exec() just after, but that's all. |
|
Could you please make a new release? We are currently on v8.10.0, and it would be great to get the fix with a new version. |
|
Anyone have any idea if that fix will address the issue with Spring/Rails and libvips? |
|
OK, I'll do 8.10.1 this weekend. It might help spring (or might not :-() |
|
8.10.1 is up now: |
|
Thanks! Here it seems to be working fine, now. 👍🏻 |
|
Hey, that's great news! Thanks for reporting back. |
|
Now we just need homebrew to update the package on their end 😬 |
|
Has no one done it? Just fork homebrew-core, edit vips.rb and make a PR, it's pretty easy. |
|
Sorry @jcupitt I've been crazy busy but will test this against Ruby/Spring once it's updated in Homebrew! |
|
Looks like homebrew updated to 8.10.1 today, fwiw. |
|
This fixes the problem in Ruby/Spring as far as I can tell, great work @jcupitt ! 💯 |
|
Woo! That's good. @kleisauke suggested the fix, not me, heh. |
alon-ne
added a commit
to wix-playground/libvips
that referenced
this issue
Dec 21, 2020
* better webp load sanity checking see libvips@d93d9bb#r40846309 * block fuzz data over 100kb Many codecs can take a huge amount of time attempting to read large random objects. jpeg_read_header(), for example, can take ~10s on a 1mb of random data. Ignore fuzz objects over 100kb. See https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24383 * depr 6D40 ecate vips_popen() it didn't work well on win, and we no longer use it anyway * detect RLE overflow in radiance old-style radiance RLE could overflow harmlessly see https://oss-fuzz.com/testcase-detail/4918518961930240 * better dbg msg in spngload * try to work around some broken heic images see libvips#1574 * revise heic thumbnail workaround * Add C++ bindings for new_from_memory_steal() new_from_memory_steal() will create a new image with the input buffer and will "move" the data into the image. The buffer is then managed by the image, and will be freed when it goes out of scope. * Don't check g_signal_connect()'s return * Remove cast in free() call * Remove redundant part of comment * Add parameter name for unused image * fix typo see libvips/pyvips#198 (comment) thanks Tremeschin * fix write ICC profile to webp ooops, a typo broke ICC profile write to webp 18 days ago thanks augustocdias see libvips#1767 * put minimise support back into pdfload * docs fix pandoc changed the name of their top-level section node * start 8.10.1 following the doc generation fix * remove redefinition of typedefs We had this in a couple of places: typedef struct _A A; typedef struct _A A; Some old gccs (eg. centos6) throw errors for this. See libvips#1774 * add missing funcs to public C API The C API was missing vips_jpegload_source and vips_svgload_source. Thanks to augustocdias. See libvips#1780 * update wrapper script fixes "make check". * experiment with doxygen for C++ docs * fix regression in thumbnail of pyr tiff The new subifd pyramid thumbnail code broke the old page-based pyramid detector. Thanks tand826 See libvips#1784 * move vips-operators.h into the header doxy hates #include inside classes * Ensure magick buffer+file checks use consistent min length guard Prevents a zero-length buffer from crashing GetImageMagick It looks like the fix for magick7 in libvips#1642 is also now required for magick6 as the assertion appears to have been backported. * add optional params to doc strings * Ensure magick buffer+file checks use consistent min length guard Prevents a zero-length buffer from crashing GetImageMagick It looks like the fix for magick7 in libvips#1642 is also now required for magick6 as the assertion appears to have been backported. * prevent 0-length buffers reaching imagemagick im6 seems to have added an assert for this see libvips#1785 * add doxygen to the build system configure tests for it, make runs it, make install copies the generated html * revise cpp codegen again use f'' strings, polish formatting * add doc comments for VError and VInterpolate * a few more doc comments * convert no-profile CMYK to RGB on save Use the fallback cmyk profile to convert to RGB on save if the image has no embedded profile. Thanks augustocdias. See libvips#1767 * fix some unknown types We were missing VipsInterpolate and guint64. Add guint64 set() as well. see libvips@636e265#commitcomment-41589463 * improve C++ API Make VObject inheritance public, so we can have a single set() for all VObject-derived types. * note how to list interpolators * Ensure SVG loader skips input with chars outside x09-x7F range Add test with example valid WebP image that happens to contain the string '<svg' within its compressed image data. * Ensure SVG loader skips input with chars outside x09-x7F range Add test with example valid WebP image that happens to contain the string '<svg' within its compressed image data. * note svg fix * improve docs for arrayjoin see libvips/pyvips#202 * Fix a small memory leak in sinkscreen * start sinkscreen thread on first use we were starting the sinkscreen background thread during vips_init() -- instead, start it on first use see libvips#1792 * fix typo * better mask sizing for gaussmat We were calculating the mask size incorrectly for small masks. Thanks johntrunc see libvips#1793 * fix handling of "squash" param in tiffsave the deprecated param was not being detected correctly, breaking vips7 compat in some cases see libvips#1801 * fix jpegload autorotate thanks chregu see libvips/php-vips#105 * note render thread change in changelog * update magick metadata naming IM seem to have changed their rules for naming metadata chunks. They are now lowercase and ICM is renamed to ICC. Add a simple test too. See libvips/ruby-vips#246 * revise doxy flags to configure * add a README.md for cpp * add some more C++ docs * more C++ docs * better dint rules We had some special cases coded for dhint inheritance, but they could fail in some edge cases. Revert to something simpler and more predictable. see libvips#1810 * don't add generated latex to repo * finish C++ doc comments * integrate new C++ docs in main docs * more small doc tweaks * Ensure VImage::set uses class to determine type Prevents null GType and associated segfault * don't set JFIF res if we will set EXIF res Some JPEG loaders give priority to JFIF resolution over EXIF resolution tags. This patch makes libvips not write the JFIF res tags if it will be writing the EXIF res tags. See libvips/ruby-vips#247 * typo in recent cpp API improvements We had G_VALUE_TYPE instead of G_OBJECT_TYPE, oops. Thanks @lovell. see libvips#1812 * fix TIFF thumbnail of buffer and source We had dropped a couple of patches. see libvips#1815 * fix tiff thumbnail from buffer and source We were missing the new tiff thumbnail logic on the source and buffer paths. see libvips#1815 * add a .gitignore for the new cpp api to stop accidentally adding it to 8.10 * raise minimum libheif version to 1.3 We didn't compile with anything less than 1.3 anyway. see libvips#1817 * block doxy latex output too * libheif: expose speed parameter (currently AV1 compression only) Supports both aom and rav1e encoders by limiting to a 0-8 range. (The rav1e encoder accepts speed values of 9 and 10 but these use 64x64 blocks more suited to video than images.) * fix dzsave iiif dimensions dzsave in iiif mode could set info.json dimensions off by one thanks Linden6 see libvips#1818 * allow both dpi and scale to be set for pdfload pdfload didn't allow both dpi and scale to be set. This patch makes the two settings combine if both are given. thanks le0daniel see libvips#1824 * tiny thumbnail speedup thumbnail can skip premultiply/unpre if there's no residual resize * allow gaussblur sigma 0 meaning no blur (obviosuly) * oop typo * Verify ISO/3GPP2 signature in heifload is_a check * Verify ISO/3GPP2 signature in heifload is_a check * better heif signature detection * heifload: simplify is_a check of first 4 bytes Allow multiples of 4, up to 32, as chunk length * heifload: simplify is_a check of first 4 bytes Allow multiples of 4, up to 32, as chunk length * revise heif sniffing again * note VImage::new_from_memory_steal() in ChangeLog plus doxy commnets etc., see libvips#1758 * Fix test failure on ARM-based Windows The optional parameters of vips_gaussnoise were incorrectly passed within vips_fractsurf. This was discovered when running the libvips testsuite on Windows 10 IoT (ARM32). * Fix test failure on ARM-based Windows The optional parameters of vips_gaussnoise were incorrectly passed within vips_fractsurf. This was discovered when running the libvips testsuite on Windows 10 IoT (ARM32). * note fractsurf fix in changelog * heifload: prevent reading beyond end of source buffer * heifload: prevent reading beyond end of source buffer * heifload: prevent reading beyond end of source buffer * revise heifload EOF detection VipsSource used a unix-style model where read() returns 0 to mean EOF. libheif uses a model where a separate call to wait_for_file_size() beforehand is used to check thaht the read will be OK, and then the read() is expected to never fail. We were trying to put EOF detection into libheif read(), but that's not the right way to do it. Instead, test for EOF in wait_for_file_size(). see libvips#1833 * get docs building with goi 1.66+ It builds now, but some doc sections are missing. Fix this properly in 8.11. See libvips#1836 * note improvements to iprofile The docs had fallen behind a bit ... iprofile is no longer usually necessary. see libvips#1843 * update vipsthumbnail docs for --export-profile and --input-profile * add stdin, stdout to vipsthumbnail eg. vipsthumbnail stdin[page=2] -o .jpg[Q=90] mirror of syntax in new_from_file etc. * improve seek on pipes There were a few issues in VipsSource around seeking on pipes. With this patch, EOF detection is better, and pipe sources automatically turn into memory sources when EOF is hit. see libvips#1829 * revise pipe sources (again) Simplify and cleanup. * pdfload was missing a rewind on source * add a test for vipsthumbnail of stdin/stdout * libheif: expose speed parameter (currently AV1 compression only) Supports both aom and rav1e encoders by limiting to a 0-8 range. (The rav1e encoder accepts speed values of 9 and 10 but these use 64x64 blocks more suited to video than images.) * note new "speed" param in heifsave To help support the rapid move to AVIF. see libvips#1819 (comment) * fix a regression in the C path for dilate/erode A ++ had been dropped in the recent refactoring. Credit to kleisauke. See libvips#1846 * fix build with libheif save buit not load We had some definitions inside the #ifdef HEIFLOAD. Thanks estepnv libvips#1844 * heifload: expose heif-compression metadata * Speed up VIPS_ARGUMENT_COLLECT_SET By using G_VALUE_COLLECT_INIT, see: https://bugzilla.gnome.org/show_bug.cgi?id=603590 * fix heifload with libheif 1.6 heif_avif wasn't added until libheif 1.7 * Separate lock for PDFium * move the pdfium lock init move it inside the existing ONCE see libvips#1857 * better GraphicsMagick image write We were not setting matte or depth correctly, thanks bfriesen. * get pdium load working again It had bitrotted a bit. Thanks @Projkt-James. See libvips#1400 * fix pdfium mutex init We need to make the mutex in _class_init, not _build, since we can lock even if _build is not called. * add pdfium load from source * improve BGRA -> RGBA conversion * revise BGRA->RGBA * relax is_a heic test rules 32 was a little too small, see libvips#1861 * fix vips7 webp load webp load using the vips7 interface was crashing, thanks barryspearce see libvips#1860 * fix out of bounds exif read in heifload We were subtracting 4 from the length of the exif data block without checking that there were 4 or more bytes there. * only remove main image (ifd0) orientation tag we were stripping all orientation tags on autorot see libvips#1865 * fix two small bugs in test_connections.c We were passing NULL rather than argv0 to VIPS_INIT(), and we were not freeing the loaded file. thanks zodf0055980 see libvips#1867 * add mssing g_option_context_free() to vipsedit We were not freeing the argument parse context in vipsedit.c. Thanks zodf0055980 see libvips#1868 * Revert "Remove round-to-nearest behaviour" This reverts commit ac30bad * Round sum values to the nearest integer in *_notab * Fix centre convention * fix out of bounds read in tiffload libtiff can change the value of some fields while scanning a corrupt TIFF file and this could trigger an out of bounds read. This patch makes tiffload more cautious about rescanning a TIFF directory before reading out header fields. * fix tiff pyramid save region-shrink we'd forgotton to connect it up thanks imgifty see libvips#1875 * add tests for tiff pyr save region-shrink flag we were testing the flag before, but not that the result was correct see libvips#1875 * flush target at end of write we were missing end-of-write flushes on four save operations thanks harukizaemon see libvips/ruby-vips#256 * missing include * Update Examples.md Just some issues I found while testing the examples: - Reference on header for the file try255.py - Typo on parameter from bigtif to bigtiff - Use explicit call to python interpreter from command line * update examples to py3 * fix compiler warning * block annoying INFO messages on some older glibs Some old glibs can display INFO messages by default. Block these ourselves. See libvips#1876 * fix icc-profiles and dzsave --no-strip We were not copying metadata down pyramid layers in dzsave, s 8BB5 o --no-strip didn't allow icc profiles on tiles. Thanks altert See libvips#1879 * Ensure that streams are properly read in spngload * better GraphicsMagick image write We were not setting matte or depth correctly, thanks bfriesen. * add read loops to heif and ppm as well We were not looping on vips_source_read() in these loaders, so they could fail when reading from very slow pipes. See kleisauke/net-vips#101 * fix changelog after GM backport * note read fixes in changelog * forgot to advance the buffer pointers thanks kleis see kleisauke/net-vips#101 (comment) * add read loops to gifload and check for error in ppnmload. * add "seed" param to perlin, worley and gaussnoise see libvips#1884 * block 0 width or height images from imagemagick IM could return 0 width and/or height for some crafted images. Block these. Thanks @Koen1999. See libvips#1890 * oops typo in magick7 load * fix a possible read loop for truncated gifs * gifload: ensure total height of all pages is sanitised * reformat * backport gifheight check ensure gifheight can't oevrflow see libvips#1892 * oops typo in magick7 load * make ppm load default to msb first We has lsb first as the default, breaking 16-bit PPM load. Thanks ewelot. see libvips#1894 * byteswap on ppm save, if necessary this was missing, thanks ewelot see libvips#1894 * Ensure vipsload only byte swaps if necessary Prior to this commit, MSB-ordered vips images were always byte swapped on both little- and big endian systems. And LSB-ordered vips images were loaded without a byte swap. This works correctly on little endian systems, but will not work on big endian systems where the byte swap must be done vice versa. This commit ensures that the byte swap only takes place when needed. See libvips#1847. * Simplify MSB-ordered image check * Determine endianness at compile time * Port Ruby test case to Python * force binary mode for connections on win stdin / stdout (for example) are created in text mode by default on win. We need to flip them to binary mode for connection read and write. See https://stackoverflow.com/questions/65014352/pipe-libvips-cli-output-to-stdout-in-windows * better test for output to target We used to enable write to stdout if the first character of an output filename was ".", eg.: vips copy x.jpg .png But this will also enable write to stdout for things like: vips copy x.jpg ./y.png This patch also tests that the rightmost "." in a filename is also the first character. Thanks barryspearce See libvips#1906 * block deprecation warnings from libgsf with an uglu gcc progma * docs clarification libvips#1912 * don't add date in ppmsave if @strip is set see libvips#1913 * add is_a_source to ppmload ppmload_source was missing an ia_a test see libvips#1915 * note ppmload fix * fix ppmsave regression ppm strip dropped magic number * n comment * webpload: ensure first frame is not blended * make webp frame blend do doround to nearest see libvips#1918 * fix heif load fails with 0 length metadata fixes libvips#1901 * fix range clips for casts to and from int Fix two bugs: - clip in casts from int32 and uint32 could overflow -- do these as gint64 now - clip in casts from float to int could overflow since float32 can't represent the full range of int32 without losing precision -- do these as double And add some more tests. Thanks ewelot. see libvips#1922 * note HEIC fix in changelog see libvips#1921 * fix heif load fails with 0 length metadata fixes libvips#1901 * note HEIC fix in changelog see libvips#1921 * forgot changelog update * improve website link in docs it was being rewritten by the export script see libvips#1928 * fix spng detection This patch was dropped from 8.10.3 release 1, annoyingly. * start 8.10.4 with a dropped patch from 8.10.3 * allow spng.pc and libspng.ps for libspng discovery * webpload: prevent divide-by-zero when blending pixels Adds a test case to prevent regression - see commit 6eaf1ed * bump version for animated webp load fix Co-authored-by: John Cupitt <jcupitt@gmail.com> Co-authored-by: Kyle Schwarz <zeranoe@gmail.com> Co-authored-by: Lovell Fuller <github@lovell.info> Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl> Co-authored-by: DarthSim <darthsim@gmail.com> Co-authored-by: Antonio Martínez <amalbala@gmail.com> Co-authored-by: Daniel Dennedy <ddennedy@gopro.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm currently investigating a SEGV during
vips_shutdownafter the libvips 8.10 update. I had no luck reproducing this on a test server, but managed to get a GDB backtrace from a production server.Presumably this is due to commit 8840bc8, which spawns the
sink_screenthread duringvips_init. Prior to this commit, this thread was not spawned until a call was made tovips_sink_screen. As far as I know, we do not usevips_sink_screen(orvips_cache) within our codebase.As an aside,
For the WebAssembly bindings I completely removed the need of this thread with commit kleisauke@9111aaf (which builds upon #1492) since it seems to be used only when the
notify_fncallback is passed tovips_sink_screen. Although this callback is not used by libvips itself, it may break compatibility withvipsdisp.The text was updated successfully, but these errors were encountered: