CN118351299A - Image segmentation method and device based on open vocabulary segmentation - Google Patents

Abstract

The invention relates to an image segmentation method and device based on open vocabulary segmentation, which acquire a target image and a preset point grid; determining at least one text embedded vector corresponding to the target image based on the target image; determining a query embedding vector corresponding to each point aiming at each point in a preset point grid; determining a prediction mask area vector corresponding to each point based on the query embedded vector corresponding to each point; determining a prediction mask category corresponding to each point based on the prediction mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm; and the preset mask text matching algorithm is used for determining a text embedding vector matched with the prediction mask area vector corresponding to each point from at least one text embedding vector corresponding to the target image. The multi-functionality of image segmentation can be finished and the segmentation performance of the image segmentation can be improved without consuming a great deal of manual annotation cost.

Description

Image segmentation method and device based on open vocabulary segmentation

Technical Field

The present invention relates to the field of image segmentation, and in particular to an image segmentation method and device based on open vocabulary segmentation.

Background technique

To overcome the limitations of closed vocabulary segmentation, open vocabulary segmentation was proposed. Open vocabulary segmentation uses text embeddings of category names represented in natural language as label embeddings instead of learning them from a training dataset. By doing so, the model can classify a wider range of vocabulary, thereby improving the ability to handle a wider range of categories. To ensure that meaningful embeddings are provided, a pre-trained text encoder is usually used. This encoder can effectively capture the semantic meaning of words and phrases, which is critical for open vocabulary segmentation. Multimodal models such as (Contrastive language-image Pre-Training, CLIP) have shown potential in open vocabulary segmentation because they are able to learn aligned image-text feature representations from large-scale Internet data.

Currently, image segmentation based on open vocabulary segmentation usually relies on image-mask-text triples, but this method requires a lot of manual effort to annotate the correspondence between mask and text, which leads to expensive annotation costs. Although some weakly supervised methods have been proposed in the prior art, such as text supervision to reduce annotation costs, the incompleteness of supervision severely limits the versatility and performance. Among them, text supervision only uses image and text pairs for semantic segmentation, which is insufficient in capturing complex spatial details, which is not optimal for dense prediction. In addition, this type of supervision, text supervision, lacks location information, making it difficult for the model to distinguish different instances with the same semantic class. These problems severely limit the versatility and segmentation performance of existing weakly supervised methods.

Therefore, the existing techniques for image segmentation based on open vocabulary segmentation require expensive annotation costs, while limiting the versatility and segmentation performance of image segmentation.

Summary of the invention

The present invention provides an image segmentation method and device based on open vocabulary segmentation, which are used to solve the problem that in the image segmentation process based on open vocabulary segmentation in the prior art, expensive annotation costs are required, while the versatility and segmentation performance of image segmentation are limited. The method and device can achieve the versatility of image segmentation and improve the segmentation performance of image segmentation without consuming a lot of manual costs to annotate the relationship between images, masks and texts.

A method for image segmentation based on open vocabulary segmentation, the method comprising: obtaining a target image and a preset point grid; determining at least one text embedding vector corresponding to the target image based on the target image; determining a query embedding vector corresponding to each point in the preset point grid; and determining a predicted mask region vector corresponding to each point based on the query embedding vector corresponding to each point; wherein the query embedding vector corresponding to each point includes a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in the target image; determining a predicted mask category corresponding to each point based on the predicted mask region vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm; wherein the predicted mask category corresponding to each point includes a pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine a text embedding vector matching the predicted mask region vector corresponding to each point from the at least one text embedding vector corresponding to the target image.

In one of the embodiments, determining at least one text embedding vector corresponding to the target image based on the target image includes: determining at least one text embedding vector corresponding to the target image based on the target image and a preset enhanced image text extraction model; the preset enhanced image text extraction model is used to extract and enhance at least one text embedding vector that matches the target image description.

In one of the embodiments, the preset enhanced image text extraction model includes: a preset visual language model, a preset text language enhancement model and a preset text encoder, and the determining of at least one text embedding vector corresponding to the target image based on the target image and the preset enhanced image text extraction model includes: inputting the target image into the preset visual language model to determine an initial text embedding vector of the target image; the initial text embedding vector of the target image includes a feature representation of a descriptive text of the target image; the initial text embedding vector of the target image is input into the preset text language enhancement model to determine at least one enhanced text represented by the text features of the target image; and the at least one enhanced text represented by the text features of the target image is respectively input into the preset text encoder to obtain at least one text embedding vector corresponding to the target image.

In one of the embodiments, for each point in the preset point grid, determining the query embedding vector corresponding to each point includes: for each point in the preset point grid, based on a pre-trained visual cue encoder, encoding each point into two position embedding vectors and at least one real mask embedding vector corresponding to each point in the target image; based on the target image and a preset visual encoder, determining a content embedding vector corresponding to the at least one real mask embedding vector; for each point in the preset point grid, concatenating the two position embedding vectors corresponding to each point and the at least one real mask embedding vector corresponding to each point in the target image, and combining them with the preset query type corresponding to each point and the content embedding vector corresponding to the at least one real mask embedding vector to obtain the query embedding vector corresponding to each point.

In one of the embodiments, before determining the predicted mask area vector corresponding to each point based on the query embedding vector corresponding to each point, the method further includes: determining the multi-scale pixel feature vector corresponding to each point based on the target image, a preset visual encoder and a pre-trained multi-scale pixel decoder; determining the predicted mask area vector corresponding to each point based on the query embedding vector corresponding to each point includes: determining the predicted mask area vector corresponding to each point based on the multi-scale pixel feature vector corresponding to each point, the query embedding vector corresponding to each point and the pre-trained mask decoder.

In one of the embodiments, determining the multi-scale pixel feature vector corresponding to each point based on the target image, a preset visual encoder and a pre-trained multi-scale pixel decoder includes: for each point, inputting the image block corresponding to each point in the target image into the preset visual encoder to obtain the initial pixel feature vector of each point; inputting the initial pixel feature vector of each point into the pre-trained multi-scale pixel decoder to obtain the multi-scale pixel feature vector of each point.

In one of the embodiments, the method of determining the predicted mask category corresponding to each point based on the predicted mask region vector corresponding to each point, at least one text embedding vector corresponding to the target image, and a preset mask text matching algorithm includes: inputting the initial pixel feature vector of each point and the predicted mask region vector corresponding to each point into a pre-trained multi-scale feature adapter to obtain the pixel feature vector of each point; for each point, based on the arc cosine similarity between the pixel feature vector corresponding to each point and each text embedding vector corresponding to the target image, determining the cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to the target image; based on the cost matrix between the pixel feature vector of each point and all text embedding vectors corresponding to each image, and a bipartite graph matching algorithm, determining the best matching pixel feature vector and text embedding vector corresponding to each point; determining the best matching text embedding vector corresponding to each point as the predicted mask category of the best matching pixel feature vector corresponding to each point.

In one of the embodiments, before inputting the initial pixel feature vector of each point, the multi-scale pixel feature vector of each point and the predicted mask area vector corresponding to each point into a pre-trained multi-scale feature adapter to obtain the pixel feature vector of each point, the method further includes: updating the pre-trained multi-scale feature adapter based on a training sample set and a preset cosine similarity loss function; wherein the training sample set includes at least one image and a pixel-level category label corresponding to each image; and the preset cosine similarity loss function is used to determine the text embedding vector corresponding to the predicted mask area vector corresponding to each point from at least one text embedding vector corresponding to each image.

The present invention also provides an image segmentation device based on open vocabulary segmentation, the device comprising: an acquisition module, used to acquire a target image and a preset point grid; a first determination module, used to determine at least one text embedding vector corresponding to the target image based on the target image; a second determination module, used to determine the query embedding vector corresponding to each point in the preset point grid; and based on the query embedding vector corresponding to each point, determine the predicted mask area vector corresponding to each point; wherein the query embedding vector corresponding to each point includes the position embedding vector of each point and at least one pixel-level real mask embedding vector corresponding to each point in the target image; a third determination module, used to determine the predicted mask category corresponding to each point based on the predicted mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm; wherein the predicted mask category corresponding to each point includes the pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine the text embedding vector matching the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to the target image.

The present invention also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned image segmentation method based on open vocabulary segmentation.

The present invention also provides a storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned image segmentation method based on open vocabulary segmentation.

The above-mentioned image segmentation method and device based on open vocabulary segmentation, by providing a preset point grid and learning the query embedding vector of each point in the preset point grid separately, thereby determining the predicted mask area vector corresponding to each point, realizing pixel-level mask division of the target image, and then combining the preset mask text matching algorithm, matching the predicted mask area vector corresponding to each point with the text embedding vector corresponding to the target image, realizing entity alignment in the text description corresponding to the mask and the target image, thereby realizing pixel-level segmentation of the target image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, mask and text does not need to be expended a lot of manual costs, thereby realizing the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. Moreover, while realizing the decoupling of the mask and the text, it is possible not only to realize the semantic segmentation of the target image, but also to distinguish the instance of the target image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of an image segmentation method based on open vocabulary segmentation provided by the present invention;

FIG2 is a second flow chart of the image segmentation method based on open vocabulary segmentation provided by the present invention;

FIG3 is a third flow chart of the image segmentation method based on open vocabulary segmentation provided by the present invention;

FIG4 is a fourth flow chart of the image segmentation method based on open vocabulary segmentation provided by the present invention;

FIG5 is a fifth flow chart of the image segmentation method based on open vocabulary segmentation provided by the present invention;

FIG6 is a schematic diagram of the framework of an image segmentation method based on open vocabulary segmentation provided by the present invention;

FIG7 is a schematic diagram of the structure of a mask decoder layer provided by the present invention;

FIG8 is a schematic diagram of a framework of a multi-scale feature adapter provided by the present invention;

FIG9 is a comparison diagram of experimental results of the image segmentation method based on open vocabulary segmentation provided by the present invention and other methods;

FIG10 is a schematic diagram of the framework of an image segmentation device based on open vocabulary segmentation provided by the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

To facilitate understanding, the inventive concept of the present invention is described.

Currently, image segmentation based on open vocabulary segmentation usually relies on image-mask-text triples, but this method requires a lot of manual effort to annotate the correspondence between mask and text, which leads to expensive annotation costs.

Although some weak supervision methods have been proposed in the prior art, such as reducing annotation costs through text supervision, the incompleteness of supervision severely limits the versatility and performance. Among them, text supervision only uses image and text pairs for semantic segmentation, which is insufficient in capturing complex spatial details, which is not optimal for dense prediction. In addition, this type of supervision, text supervision, lacks location information, making it difficult for the model to distinguish different instances with the same semantic class, that is, this type is only suitable for ordinary semantic segmentation and is difficult to apply to instance segmentation or panoramic segmentation. These problems severely limit the versatility and segmentation performance of existing weak supervision methods.

Therefore, in view of the above problems, the image segmentation method and device based on open vocabulary segmentation provided by the present invention liberates the strict correspondence between mask and text by using independent image-mask and image-text pairs. These two types of pairs can be easily collected from different sources. And by providing a preset point grid and learning the query embedding vector of each point in the preset point grid separately, the predicted mask area vector corresponding to each point is determined, and the target image is masked at the pixel level. The predicted mask area vector corresponding to each point is matched with the text embedding vector corresponding to the target image in combination with the preset mask text matching algorithm, so as to realize the entity alignment in the text description corresponding to the mask and the target image, thereby realizing the pixel-level segmentation of the target image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, mask and text does not need to be expended a lot of labor costs, so as to realize the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. Moreover, while decoupling the mask from the text, it is possible not only to achieve semantic segmentation of the target image, but also to distinguish instances of the target image.

The image segmentation method and device based on open vocabulary segmentation provided by the present invention are described below with reference to the accompanying drawings.

Fig. 1 is a flow chart of an image segmentation method based on open vocabulary segmentation provided by the present invention. It can be understood that the image segmentation method based on open vocabulary segmentation can be performed by an image segmentation device based on open vocabulary segmentation. The image segmentation device based on open vocabulary segmentation can be a computer device.

As shown in FIG1 , in one embodiment, an image segmentation method based on open vocabulary segmentation is proposed, which may specifically include the following steps:

Step 110, obtaining a target image and a preset point grid.

The target image is an image to be semantically segmented or panoptically segmented, for example, an image in the field of medical imaging or autonomous driving. The preset point grid is used as a visual cue for the target image to determine the predicted mask region vector, mainly used to align the position information of the mask and the target image. The preset point grid includes a plurality of evenly distributed points. The preset point grid is a grid of size h×w, with a height of h≤4 and a width of w≤4, and the grid is aligned with the center of the pixel.

It is understood that after acquiring the target image, the target image can also be preprocessed according to the preset processing steps to obtain images of different resolutions. For example, a random horizontal flip can be applied to the target image. Subsequently, the image is randomly scaled to a resolution within a range of 716x 716 to 1075x 1075. Finally, a cropped image with a resolution of 896x 896 is extracted from the scaled image as input.

Step 120: Based on the target image, determine at least one text embedding vector corresponding to the target image.

Among them, the at least one text embedding vector corresponding to the target image is the text embedding vector of at least one description text corresponding to the target image. Among them, the at least one description text corresponding to the target image is used to describe the type of entity in the target image. For example, if the target image includes a person, a cat and a table, the description text is a person, a cat and a table. Combined with the aforementioned open vocabulary segmentation, it can be understood that in order to learn more semantic features in the image, the text embedding corresponding to the target image can be directly extracted by using a pre-trained text encoder as the label embedding of the target image, which can be used to classify the entities involved in the target image.

Specifically, at least one text embedding vector corresponding to the target image can be extracted through some multimodal models, such as CLIP.

Step 130: for each point in the preset point grid, determine the query embedding vector corresponding to each point; and based on the query embedding vector corresponding to each point, determine the prediction mask region vector corresponding to each point.

The query embedding vector corresponding to each point includes a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in the target image.

It can be understood that in order to use independent image-mask and image-text pairs to liberate the strict correspondence between mask and text, and at the same time to capture complex spatial details, obtain more location information, and distinguish different instances of the same semantic class in the target image, a preset point grid is set, and the query embedding vector of each point in the preset point grid is learned separately, so as to obtain the pixel-level true mask embedding vector corresponding to the target image, and realize pixel-level mask division of the target image, so that the predicted mask area vector corresponding to each point determined based on the query embedding vector corresponding to each point can not only realize semantic segmentation of the target image, but also instance segmentation and panoramic segmentation of the target image, thereby realizing the use of weak supervision methods, reducing the cost of manual annotation, and ensuring the versatility and segmentation performance of image segmentation.

It can also be understood that in actual applications, multiple points may be queried at a time, and a set of predicted mask region vectors corresponding to the multiple points can be determined based on the above method, and a set of predicted mask region vectors corresponding to the multiple points can correspond to multiple real mask embedding vectors. The relationship between the predicted mask region vector and the real mask vector can be one-to-one, one-to-many, many-to-one, or many-to-many.

Step 140: Determine the predicted mask category corresponding to each point based on the predicted mask region vector corresponding to each point, at least one text embedding vector corresponding to the target image, and a preset mask text matching algorithm.

Among them, the predicted mask category corresponding to each point includes the pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine the text embedding vector that matches the predicted mask area vector corresponding to each point from at least one text embedding vector corresponding to the target image.

It can be understood that the preset mask text matching algorithm is used here, combined with the predicted mask area vector determined previously, to achieve entity alignment between the mask and the corresponding text description of the target image, thereby achieving pixel-level classification of entities in the target image under the decoupling of the mask and the text, thereby achieving semantic segmentation of the target image without realizing it, and being able to distinguish instances of the target image, thereby ensuring the versatility and segmentation performance of image segmentation.

The image segmentation method and device based on open vocabulary segmentation provided by the present invention provide a preset point grid and learn the query embedding vector of each point in the preset point grid separately, so as to determine the predicted mask area vector corresponding to each point, realize the pixel-level mask division of the target image, and then combine the preset mask text matching algorithm to match the predicted mask area vector corresponding to each point with the text embedding vector corresponding to the target image, realize the entity alignment in the text description corresponding to the mask and the target image, so as to realize the pixel-level segmentation of the target image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, the mask and the text is not spent a lot of labor costs, so as to realize the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. In addition, while realizing the decoupling of the mask and the text, it is possible not only to realize the semantic segmentation of the target image, but also to distinguish the instance of the target image.

In one of the embodiments, determining at least one text embedding vector corresponding to the target image based on the target image includes: determining at least one text embedding vector corresponding to the target image based on the target image and a preset enhanced image text extraction model; the preset enhanced image text extraction model is used to extract and enhance at least one text embedding vector that matches the target image description.

In one embodiment, the preset enhanced image text extraction model includes: a preset visual language model, a preset text language enhancement model and a preset text encoder. As shown in FIG2 , the step of determining at least one text embedding vector corresponding to the target image based on the target image and the preset enhanced image text extraction model includes:

Step 210, input the target image into the preset visual language model to determine an initial text embedding vector of the target image; the initial text embedding vector of the target image includes a feature representation of the description text of the target image.

Among them, the preset visual language model is used to refine the text description and improve the quality of the text description. It can be understood that since the collected image-text pairs usually contain some text that does not match the image, which leads to incorrect correspondence between the mask and the entity, directly aligning these incorrect correspondences will lead to poor segmentation performance. Therefore, the present invention improves the quality of the text description by setting a corresponding preset visual language model, so that the initial text embedding vector of the target image is a feature representation of the text description that matches the target image more closely.

Preferably, the preset visual language model may be a Large Language and Vision Assistant (LLaVa) model. It is understood that the preset visual language model may also adopt other large visual language models, such as Wenxinyiyan, and the present invention is not limited thereto.

Step 220: input the initial text embedding vector of the target image into a preset text language enhancement model to determine at least one enhanced text represented by the text feature of the target image.

The preset text language enhancement model is used for accurate entity extraction. The at least one enhanced text represented by the text feature of the target image corresponds to at least one entity in the target image, such as tasks, scenes, and clothes worn by characters involved in the target image.

Preferably, the preset text language enhancement model may be, for example, a (Chat Generative Pre-trained Transformer, ChatGPT) parser. It is understandable that the preset text language enhancement model may also adopt other models, and the present invention is not limited to this.

It can be understood that the preset text language enhancement model combined with the preset visual language model can extract accurate entities from the target image, thereby obtaining accurate image-text pairs, laying the foundation for the subsequent alignment of the image text with the image mask.

Step 230: input at least one enhanced text represented by the text feature of the target image into the preset text encoder to obtain at least one text embedding vector corresponding to the target image.

The preset text encoder, for example, can be a CLIP model based on ConvNext. Specifically, the preset text encoder is constructed as a 16-layer Transformer, each layer has 768 neural units, and has 12 attention heads.

In one embodiment, as shown in FIG3 , determining, for each point in the preset point grid, a query embedding vector corresponding to each point includes:

Step 310, for each point in the preset point grid, based on a pre-trained visual cue encoder, encode each point into two position embedding vectors and at least one true mask embedding vector corresponding to each point in the target image.

Specifically, the position embedding vector of each point can be determined based on the sinusoidal encoding in the pre-trained visual cue encoder.

Wherein, the pre-trained visual cue encoder is used to encode the visual cue (point or bounding box). The pre-trained visual cue encoder is determined based on a first preset loss function, and the specific determination process is referred to the relevant description below. In the present invention, each visual cue (at least one real mask embedding vector corresponding to each point in the target image) is coupled with a position embedding vector.

Step 320: Determine a content embedding vector corresponding to the at least one true mask embedding vector based on the target image and a preset visual encoder.

Among them, the preset visual encoder is used to extract visual features in the target image, for example, it can be a CLIP model based on ConvNext. Specifically, the preset visual encoder can be configured as a ConvNext large model, including four stages. Each stage contains a different number of blocks: 3 in the first stage, 3 in the second stage, 27 in the third stage, and 3 in the fourth stage. Multi-scale features are extracted using the preset visual encoder. These features are represented as feature maps of different widths and scales: the 192-wide feature map is reduced by 4 times, the 384-wide feature map is reduced by 8 times, the 768-wide feature map is reduced by 16 times, and the 1536-wide feature map is reduced by 32 times.

Step 330, for each point in the preset point grid, concatenate the two position embedding vectors corresponding to each point and the at least one true mask embedding vector corresponding to each point in the target image, and combine them with the preset query type corresponding to each point and the content embedding vector corresponding to the at least one true mask embedding vector to obtain the query embedding vector corresponding to each point.

Specifically, the expression of the query embedding vector corresponding to the i-th point is: in, Represents the two position embedding vectors corresponding to the i-th point, q ^msk ∈R ^M×dim represents the M real mask embedding vectors corresponding to the i-th point in the target image, q ^type ∈R ^dim represents the preset query type corresponding to the i-th point, which can be a point or a box, and q ^feat ∈R ^dim represents the content embedding vector corresponding to the at least one real mask embedding vector.

In one of the embodiments, before determining the predicted mask area vector corresponding to each point based on the query embedding vector corresponding to each point, the method further includes: determining the multi-scale pixel feature vector corresponding to each point based on the target image, a preset visual encoder and a pre-trained multi-scale pixel decoder.

It can be understood that in order to effectively associate the mask with the visual cue (such as points and boxes), the visual cue encoder, the multi-scale pixel decoder and the mask decoder are combined to achieve it.

Among them, the pre-trained multi-scale pixel decoder is used to enhance the visual features in the target image extracted from the preset visual encoder, thereby reducing the inherent noise in the correspondence between the mask and the entity. The pre-trained multi-scale pixel decoder includes a feature pyramid network (FPN) architecture. First, a 6-layer multi-scale deformable transformer is used on the multi-scale features to aggregate contextual information. Then, the low-resolution feature map in the multi-scale pixel decoder is enlarged by 2 times and then combined with the corresponding resolution feature map from the backbone, which has been projected to match the channel dimension. The projection is achieved through a 1×1 convolutional layer, followed by group normalization (GroupNorm). Subsequently, the merged features are further processed by an additional 3×3 convolutional layer, supplemented by GroupNorm and ReLU activation. This process is applied iteratively, starting with a feature map that is 32 times smaller, until we obtain a final feature map that is 4 times smaller. In order to generate a pixel-by-pixel embedding, a single 1×1 convolutional layer is applied at the end. Throughout the multi-scale pixel decoder, all feature maps maintain a consistent dimension of 256 channels.

Correspondingly, the method of determining the predicted mask area vector corresponding to each point based on the query embedding vector corresponding to each point includes: determining the predicted mask area vector corresponding to each point based on the multi-scale pixel feature vector corresponding to each point, the query embedding vector corresponding to each point and a pre-trained mask decoder.

Specifically, the predicted mask region vector corresponding to each point can be obtained based on the matrix multiplication of the multi-scale pixel feature vector corresponding to each point and the query embedding vector corresponding to each point. The corresponding calculation formula can be: Among them, _qi represents the query embedding vector corresponding to the i-th point, The multi-scale pixel feature vector corresponding to the i-th point, sigmoid(·) is the sigmoid function that normalizes the mask value to [0,1].

It can be understood that in actual scenarios, multiple points can be queried at the same time, so the query embedding vectors corresponding to one or more points can be used as a group of query embeddings, that is, Q = q ₁ , q ₂ , ... q _n , and the corresponding calculation formula for the prediction mask region vector is: For example, it can be m _n = sigmoid (Q; x ^p ), m _n ∈ ^RM×H×W ,

In the present invention, a transformer decoder can be used as a mask decoder. Specifically, the present invention uses six mask decoder layers as shown in FIG7 , and the same loss function is applied after each layer. The specific loss function can refer to the first preset loss function below.

It should be noted that the visual cue encoder, mask decoder and multi-scale pixel decoder can all be updated based on the at least one image and the query embedding vector corresponding to each point in combination with a first preset loss function. Specifically, the first preset loss function is:

Where λ _bce and λ _dice are two hyperparameters used to balance the binary cross entropy loss and language segmentation loss The predicted mask region vector corresponding to each point _mi , It includes k real mask region vectors m _j and the category label c _j corresponding to each real mask region vector.

In one embodiment, as shown in FIG4 , the step of determining the predicted mask category corresponding to each point based on the predicted mask region vector corresponding to each point, at least one text embedding vector corresponding to the target image, and a preset mask text matching algorithm includes:

Step 410 , input the initial pixel feature vector of each point and the predicted mask region vector corresponding to each point into a pre-trained multi-scale feature adapter to obtain the pixel feature vector of each point.

Among them, the pre-trained multi-scale feature adapter is used to match the pre-trained multi-scale pixel decoder, and is used to process the feature maps of different scales output from the pre-trained multi-scale pixel decoder respectively, and refine and enhance the pixel features of various scales.

Specifically, the initial pixel feature vector of each point and the predicted mask region vector corresponding to each point may be sequentially subjected to mask pooling and a multi-scale feature adapter to obtain the pixel feature vector of each point. The expression corresponding to the pixel feature vector of each point is: r _i =Fv(P(x ^v ,m _i )), where P() represents mask pooling, Fv() represents a multi-scale feature adapter, x ^v represents the initial pixel feature vector of each point, and m _i represents the predicted mask region vector corresponding to each point.

Specifically, the specific structure of the multi-scale feature adapter can refer to FIG8 .

Step 420, for each point, based on the arc cosine similarity between the pixel feature vector corresponding to each point and each text embedding vector corresponding to the target image, determine a cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to the target image.

Among them, the expression corresponding to the cost matrix between the predicted mask area vector corresponding to each point and each text embedding vector corresponding to each image is: Wherein, δ'i _,j represents the arc cosine similarity between the pixel feature vector corresponding to the i-th point and the j-th text embedding vector corresponding to each image, _ri represents the pixel feature vector of the i-th point, and _tj represents the j-th text embedding vector corresponding to each image.

Step 430, based on the cost matrix between the pixel feature vector of each point and all text embedding vectors corresponding to each image, and a bipartite graph matching algorithm, determine the best matching pixel feature vector and text embedding vector corresponding to each point.

Step 440 , the best matching text embedding vector corresponding to each point is determined as the prediction mask category of the best matching pixel feature vector corresponding to each point.

In one of the embodiments, before inputting the initial pixel feature vector of each point, the multi-scale pixel feature vector of each point and the predicted mask area vector corresponding to each point into a pre-trained multi-scale feature adapter to obtain the pixel feature vector of each point, the method further includes: updating the pre-trained multi-scale feature adapter based on a training sample set and a preset cosine similarity loss function; wherein the training sample set includes at least one image and a pixel-level category label corresponding to each image; and the preset cosine similarity loss function is used to determine the text embedding vector corresponding to the predicted mask area vector corresponding to each point from at least one text embedding vector corresponding to each image.

Specifically, in one embodiment, as shown in FIG5 , the training process of the multi-scale feature adapter includes:

Step 510: for each image in at least one image in the training sample set, input the each image into a preset enhanced image text extraction model to determine at least one text embedding vector corresponding to the each image.

Step 520, input each image in the at least one image in the training sample set and the predicted mask area vector corresponding to each point into a preset visual encoder and a pre-trained multi-scale pixel decoder, and obtain the pixel feature vector of each point through the multi-scale feature adapter.

Step 530 , based on at least one text embedding vector corresponding to all images in the training sample set, the pixel feature vector of each point in the point grid and a preset cosine similarity loss function, a pre-trained multi-scale feature adapter is updated.

Specifically, step 530 includes:

Step 5301, for each point, based on the arc cosine similarity between the pixel feature vector of each point and each text embedding vector corresponding to each image, determine the cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to each image.

Among them, the expression corresponding to the cost matrix between the predicted mask area vector corresponding to each point and each text embedding vector corresponding to each image is: Wherein, δ'i _,j represents the arc cosine similarity between the pixel feature vector corresponding to the i-th point and the j-th text embedding vector corresponding to each image, _ri represents the pixel feature vector of the i-th point, and _tj represents the j-th text embedding vector corresponding to each image.

Step 5302, based on the cost matrix between the pixel feature vector of each point and all text embedding vectors corresponding to each image, and a bipartite graph matching algorithm, determine the best matching pixel feature vector and text embedding vector corresponding to each point.

Step 5303, input the best matching pixel feature vector and text embedding vector corresponding to each point into the preset cosine similarity loss function to obtain the loss function value corresponding to each point, and update the parameters of the pre-trained multi-scale feature adapter based on the loss function value corresponding to each point.

Among them, the preset cosine similarity loss function is:

Among them, _ri ' represents the best matching pixel feature vector corresponding to the i-th point, and _tk represents the best matching k-th text embedding vector corresponding to the i-th point.

FIG6 is a schematic diagram of a framework of an image segmentation method based on open vocabulary segmentation provided by the present invention. For image-mask pairs, a branch of mask generation, including a visual cue encoder, a pixel decoder, and a mask decoder, is used to predict a set of binary masks of the input image. For image-text pairs, mask-text bipartite graph matching is used to utilize confidence pairs between predicted masks and entities in text descriptions. Afterwards, a multi-scale feature adapter is used to enhance the mask visual embedding, which is further aligned with the relevant entity embedding based on confidence pairs. Since the collected image-text pairs usually contain some text that does not match the image, the correspondence between the mask and the entity is incorrect. Therefore, a large visual-text model can be used to improve the quality of the text description, and combined with a ChatGPT-based parser for accurate entity extraction. For mask-text alignment, mask-text bipartite graph matching is introduced to utilize confidence pairs of predicted masks and entities. The multi-scale feature adapter aims to enhance the visual embedding of the predicted mask and further align it with the text embedding of the corresponding entity. Since the mask and text are allowed to be unpaired in the present invention, multi-scale integration is introduced to improve the quality of visual embedding and effectively solve the inherent noise in the correspondence between mask and entity, thereby stabilizing the matching process.

As shown in Figure 7, each mask decoder layer includes a crisscross attention layer. This crisscross attention layer is crucial because it ensures that the final pixel features are enriched with basic geometric information such as point coordinates and bounding boxes. In addition, in each decoder layer, a crisscross attention layer is used, which ensures that the final pixel feature vector has access to key geometric information (e.g., point coordinates and boxes). Finally, the visual cue embedding (query embedding vector) is fed to the layer for normalization and then processed by a multi-layer perceptron (MLP). These processed query embedding vectors are then dot-producted with the pixel feature vector to ultimately generate a predicted mask (predicted mask region vector). The mask decoder layer updates the visual cue embedding and pixel features through a crisscross attention layer. The self-attention layer is used to update the visual cue. At each attention layer, the position encoding is added to the pixel feature, and the entire original visual cue (including the position encoding) is added to the updated visual cue.

FIG8 is a schematic diagram of the framework of the multi-scale feature adapter provided by the present invention. The multi-scale feature adapter shown in FIG8 is designed to refine and enhance pixel features of various scales. The adapter consists of multiple low-level adapters (LoRA adapters) [27], each of which is specially tailored to update features of a specific scale. Each LoRA adapter contains two linear layers, denoted as A and B, with a nonlinear activation function placed in the middle. The linear layers A and B are used to linearly transform the input data, while the activation function introduces nonlinearity, allowing more complex relationships in the data to be modeled. Each adapter is responsible for processing features of a specific scale, which suggests a hierarchical approach to refining features. This modular design allows features to be processed intensively and specialized according to their resolution and semantic complexity, which is particularly beneficial for intensive prediction tasks.

For ease of understanding, the effects that can be achieved by the present invention are illustrated below in conjunction with experiments. It is mainly compared with fully supervised and weakly supervised methods, and the specific comparison results are shown in Figure 9. Among them, "COCO S.", "COCO P." and "COCO C." represent the COCO dataset, the panoramic dataset and the subtitle dataset, respectively. "O365" represents the Object 365 dataset. "M.41M" represents the merged 41M image dataset. mIoU of all datasets of this application. Specifically, a series of benchmarks are used to comprehensively compare the method of the present invention with the existing methods. Among them, the datasets involved include ADE20K (including 150 and 847 variants), PASCAL Context (459 and 59 variants), PASCAL VOC (including 20 and 21 categories) and Cityscapes. Among them, the existing fully supervised methods involved can be referred to: (A simple baseline for openvocabulary semantic segmentation with pre-trained visionlanguage model, SimBaseline), (Decouplingzero-shot semantic segmentation, ZegFormer), (Language-driven semantic segmentation, LSeg+), (Open-vocabulary semantic segmentation with mask-adaptedclip, OVSeg), (Open-vocabulary panoptic segmentation with text-to-imagediffusion models, ODISE), (Generalized decoding for pixel, image, and language, X-Decoder), (A simple framework for open-vocabulary segmentation and detection, OpenSEED), (Openvocabulary universal image segmentation with maskclip, MaskCLIP), (Open-vocabulary segmentation with single frozen convolutionalclip, FC-CLIP). The existing weak supervision methods involved can be referred to: (Groupvit: Semantic segmentation emerges from text supervision, GroupViT), (Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs, TCL), (Learning open-vocabulary semantic segmentation models from natural language supervision, OVSeg), (Patch aggregation with learnable centers for pen-vocabulary semantic segmentation, SegCLIP), (Perceptual grouping in contrastive vision-language models, CLIPpy), (Mixreorg: Cross-modal mixed patcheorganization is a good mask learner for open-world semantic segmentation, MixReorg), (Sam-clip: Merging vision foundation models towards semantic and spatial understanding, SAM-CLIP). Compared with the existing weak supervision methods, the method of the present invention shows significant performance improvements on all evaluated datasets. Specifically, in the more challenging PASCAL Context-459 dataset, our method not only surpasses weakly supervised methods, but also outperforms the most advanced fully supervised methods, such as FC-CLIP. This shows that our method has superior ability in classifying various semantic categories. In addition, in the PASCAL VOC benchmark (20 and 21 categories), our method demonstrates significant enhancement over the most advanced weakly supervised methods, achieving 18.3% and 12.2% mIoU improvements, respectively, which shows that our method can capture fine-grained spatial structures. These results bring the practical applicability of weakly supervised open vocabulary segmentation to a new level.

The image segmentation device based on open vocabulary segmentation and the image segmentation method based on open vocabulary segmentation provided by the present invention are described below. The image segmentation device based on open vocabulary segmentation described below and the image segmentation method based on open vocabulary segmentation described above can be referred to each other.

As shown in FIG10 , in one embodiment, an image segmentation apparatus based on open vocabulary segmentation is provided. The image segmentation apparatus based on open vocabulary segmentation may include:

An acquisition module 1010 is used to acquire a target image and a preset point grid;

A first determination module 1020, configured to determine, based on the target image, at least one text embedding vector corresponding to the target image;

The second determination module 1030 is used to determine, for each point in the preset point grid, a query embedding vector corresponding to each point; and based on the query embedding vector corresponding to each point, determine a predicted mask region vector corresponding to each point; wherein the query embedding vector corresponding to each point includes a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in the target image;

The third determination module 1040 is used to determine the predicted mask category corresponding to each point based on the predicted mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm; wherein the predicted mask category corresponding to each point includes the pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine the text embedding vector that matches the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to the target image.

The image segmentation device based on open vocabulary segmentation provided by the present invention provides a preset point grid and separately learns the query embedding vector of each point in the preset point grid, thereby determining the predicted mask area vector corresponding to each point, realizing pixel-level mask division of the target image, and then combining the preset mask text matching algorithm, matching the predicted mask area vector corresponding to each point with the text embedding vector corresponding to the target image, realizing entity alignment in the text description corresponding to the mask and the target image, thereby realizing pixel-level segmentation of the target image under the condition of decoupling the mask and the text. In this way, the decoupling of the mask and the text is realized, and the relationship between the corresponding image, the mask and the text is not spent a lot of labor costs, so as to realize the use of independent image-mask and image-text pairs to liberate the strict correspondence between the mask and the text. In addition, while realizing the decoupling of the mask and the text, it is possible not only to realize the semantic segmentation of the target image, but also to distinguish the instance of the target image.

In addition, the present invention also provides a computer device, the computer device includes a memory, a processor and a computer program stored in the memory and executable on the processor, the processor executes the image segmentation method based on open vocabulary segmentation provided by the present invention, the image segmentation method based on open vocabulary segmentation includes: acquiring a target image and a preset point grid; based on the target image, determining at least one text embedding vector corresponding to the target image; for each point in the preset point grid, determining a query embedding vector corresponding to each point; and based on the query embedding vector corresponding to each point, determining a predicted mask area vector corresponding to each point; wherein each point The corresponding query embedding vector includes the position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in the target image; based on the predicted mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm, the predicted mask category corresponding to each point is determined; wherein the predicted mask category corresponding to each point includes the pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine the text embedding vector that matches the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to the target image.

On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image segmentation method based on open vocabulary segmentation provided by the present invention, and the image segmentation method based on open vocabulary segmentation includes: acquiring a target image and a preset point grid; based on the target image, determining at least one text embedding vector corresponding to the target image; for each point in the preset point grid, determining the query embedding vector corresponding to each point; and based on the query embedding vector corresponding to each point, determining the predicted mask area corresponding to each point vector; wherein the query embedding vector corresponding to each point includes the position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in the target image; based on the predicted mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm, the predicted mask category corresponding to each point is determined; wherein the predicted mask category corresponding to each point includes the pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine the text embedding vector that matches the predicted mask area vector corresponding to each point from the at least one text embedding vector corresponding to the target image.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the image segmentation method based on open vocabulary segmentation provided by the present invention, the image segmentation method based on open vocabulary segmentation comprising: acquiring a target image and a preset point grid; based on the target image, determining at least one text embedding vector corresponding to the target image; for each point in the preset point grid, determining a query embedding vector corresponding to each point; and based on the query embedding vector corresponding to each point, determining a predicted mask area vector corresponding to each point; wherein the query embedding vector corresponding to each point The vector includes a position embedding vector of each point and at least one pixel-level true mask embedding vector corresponding to each point in the target image; based on the predicted mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm, the predicted mask category corresponding to each point is determined; wherein the predicted mask category corresponding to each point includes the pixel-level category label corresponding to each point in the target image; the preset mask text matching algorithm is used to determine a text embedding vector that matches the predicted mask area vector corresponding to each point from at least one text embedding vector corresponding to the target image.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

It can be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (11)

1. An image segmentation method based on open vocabulary segmentation, the method comprising:

acquiring a target image and a preset point grid;

Determining at least one text embedding vector corresponding to the target image based on the target image;

Determining a query embedding vector corresponding to each point in the preset point grid; determining a prediction mask area vector corresponding to each point based on the query embedded vector corresponding to each point; the query embedding vector corresponding to each point comprises a position embedding vector of each point and a true mask embedding vector of at least one pixel level corresponding to each point in the target image;

determining a prediction mask category corresponding to each point based on the prediction mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image and a preset mask text matching algorithm; the prediction mask class corresponding to each point comprises class labels of pixel levels corresponding to each point in the target image; the preset mask text matching algorithm is used for determining a text embedded vector matched with the prediction mask area vector corresponding to each point from at least one text embedded vector corresponding to the target image.

2. An open vocabulary segmentation based image segmentation method according to claim 1 wherein the determining at least one text embedding vector corresponding to the target image based on the target image comprises:

Determining at least one text embedding vector corresponding to the target image based on the target image and a preset enhanced image text extraction model; the preset enhanced image text extraction model is used for extracting and enhancing at least one text embedded vector matched with the target image description.

3. An open vocabulary segmentation based image segmentation method according to claim 2 wherein,

The preset enhanced image text extraction model comprises the following steps: the method for determining the text embedding vector corresponding to the target image comprises the steps of:

Inputting the target image into the preset visual language model, and determining an initial text embedding vector of the target image; the initial text embedding vector of the target image comprises a characteristic representation of the descriptive text of the target image;

Embedding the initial text of the target image into a vector, inputting the vector into a preset text language enhancement model, and determining at least one enhancement text represented by the text characteristics of the target image;

And respectively inputting at least one enhanced text represented by the text characteristics of the target image into the preset text encoder to obtain at least one text embedded vector corresponding to the target image.

4. The method for image segmentation based on open vocabulary segmentation according to claim 1, wherein the determining, for each point in the preset point grid, a query embedding vector corresponding to the each point comprises:

For each point in the preset point grid, based on a pre-trained visual cue encoder, encoding each point into two position embedded vectors and at least one real mask embedded vector corresponding to each point in the target image;

determining a content embedding vector corresponding to the at least one true mask embedding vector based on the target image and a preset visual encoder;

And for each point in the preset point grid, splicing the two position embedded vectors corresponding to each point and at least one real mask embedded vector corresponding to each point in the target image, and combining the preset query type corresponding to each point and the content embedded vector corresponding to the at least one real mask embedded vector to obtain the query embedded vector corresponding to each point.

5. The open vocabulary segmentation based image segmentation method of claim 1 wherein prior to the determining the prediction mask region vector for each point based on the query embedding vector for each point, the method further comprises: determining a multi-scale pixel feature vector corresponding to each point based on the target image, a preset visual encoder and a pre-trained multi-scale pixel decoder;

the determining the prediction mask area vector corresponding to each point based on the query embedded vector corresponding to each point comprises the following steps:

And determining a prediction mask area vector corresponding to each point based on the multi-scale pixel characteristic vector corresponding to each point, the query embedded vector corresponding to each point and the pre-trained mask decoder.

6. The method for image segmentation based on open vocabulary segmentation according to claim 5, wherein the determining the multi-scale pixel feature vector corresponding to each point based on the target image, a preset visual encoder and a pre-trained multi-scale pixel decoder comprises:

Inputting an image block corresponding to each point in the target image into the preset visual encoder aiming at each point to obtain an initial pixel characteristic vector of each point;

And inputting the initial pixel characteristic vector of each point into the pre-trained multi-scale pixel decoder to obtain the multi-scale pixel characteristic vector of each point.

7. The method for image segmentation based on open vocabulary segmentation according to claim 6, wherein the determining the prediction mask category corresponding to each point based on the prediction mask region vector corresponding to each point, at least one text embedding vector corresponding to the target image, and a preset mask text matching algorithm comprises:

Inputting the initial pixel characteristic vector of each point and the prediction mask area vector corresponding to each point into a pre-trained multi-scale characteristic adapter to obtain the pixel characteristic vector of each point;

for each point, determining a cost matrix between the pixel feature vector of each point and each text embedding vector corresponding to the target image based on the inverse cosine similarity of the pixel feature vector corresponding to each point and each text embedding vector corresponding to the target image;

determining the best matched pixel characteristic vector and text embedding vector corresponding to each point based on a cost matrix between the pixel characteristic vector of each point and all text embedding vectors corresponding to each image and a bipartite graph matching algorithm;

and embedding the text with the best match corresponding to each point into the vector, and determining the prediction mask type of the pixel characteristic vector with the best match corresponding to each point.

8. The method of claim 7, wherein before inputting the initial pixel feature vector of each point, the multi-scale pixel feature vector of each point, and the prediction mask region vector corresponding to each point, the method further comprises:

Updating to obtain a pre-trained multi-scale feature adapter based on the training sample set and a preset cosine similarity loss function; the training sample set comprises at least one image and a pixel-level class label corresponding to each image; the preset cosine similarity loss function is used for determining a text embedding vector corresponding to the prediction mask area vector corresponding to each point from at least one text embedding vector corresponding to each image.

9. An image segmentation apparatus based on open vocabulary segmentation, the apparatus comprising:

The acquisition module is used for acquiring the target image and a preset point grid;

the first determining module is used for determining at least one text embedding vector corresponding to the target image based on the target image;

The second determining module is used for determining a query embedding vector corresponding to each point aiming at each point in the preset point grid; determining a prediction mask area vector corresponding to each point based on the query embedded vector corresponding to each point; the query embedding vector corresponding to each point comprises a position embedding vector of each point and a true mask embedding vector of at least one pixel level corresponding to each point in the target image;

A third determining module, configured to determine a prediction mask category corresponding to each point based on the prediction mask area vector corresponding to each point, at least one text embedding vector corresponding to the target image, and a preset mask text matching algorithm; the prediction mask class corresponding to each point comprises class labels of pixel levels corresponding to each point in the target image; the preset mask text matching algorithm is used for determining a text embedded vector matched with the prediction mask area vector corresponding to each point from at least one text embedded vector corresponding to the target image.

10. A computer device comprising a memory and a processor, wherein the memory has stored therein computer readable instructions which, when executed by the processor, cause the processor to perform the steps of the open vocabulary segmentation based image segmentation method of any one of claims 1 to 8.

11. A storage medium storing computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the open vocabulary segmentation based image segmentation method of any of the preceding claims 1-8.