From my previous blogs, we have seen an unconventional way of editing images by looking at their frequencies in the Fourier plane and blocking the unwanted ones. Now, we will look into another way processing image; via segmentation. Basically, image segmentation is the selection of a region(s) of a given image based on their similar features such that further processing might be done. For our purposes, the feature we will be focusing on is the color. Let us first study the image of a check below:
If we wanted to edit all the written text, we must find a way to separate them from the rest. To do this, we first take the histogram of the colors present in the check:
It is quite obvious that majority of the colors a present in the check are gray, which ranges from values 180 to 210 (remember that a gray value of 0 is black and 1 white). Since these background pixels occupy most of the image, we know that the text is of values lower than them. Let us apply a threshold of less than 125:
By doing this, we set the values of all the pixels less than the the threshold equal to 1 and the rest 0. Now that all the text is in white, it is much easier to edit these parts. If we then wanted to play with the background, we take the opposite of the threshold:
Quite simple eh? In a few lines, we were able to highlight areas of an image based on their pixel values. But what if the image is colored and its components aren’t distinguishable in grayscale, like this one for example:
For images like this one, segmentation must be done based on a set region having the same color of interest.
But before we delve into colored segmentation, let us first talk about the normalized chromaticity coordinates (NCC), which would aid us in performing colored segmentations. First, we know that any colored pixel is composed of three values; its red (R), green (G) and blue (B) components. If we let I = R + G + B, then the NCC are:
By doing this, we have r + g + b = 1 and b = 1 – (r + g) implying that the variables can only have values between 0 and 1 and the dependency of b to r and g. Thus, we can see that the color chromaticity can be represented using only the variables r and g, instead of R, G and B. The plot of r and g would look like:
where the x axis is g and the y axis r. Note that the image was tilted purposely to shift the origin accordingly with the default scilab origin for ease. At the origin, where r = g = 0, is blue while at r = 1 and g = 1 are the purely red and green points. With this technique, any color can be represented in the NCC space.
Now, let us look into the two ways of color segmentation; the parametric and non-parametric segmentation.
I. Parametric Probability Distribution Estimation
Basically, what we are going to do here is to compute the probability that a pixel belongs to a color distribution of interest. This color distribution is determined by the region of interest (ROI) we select. To start this process, we compute the normalized chromaticity coordinates of the red and green layers of the image to be processed. Next, we crop an area of the image with the colored desired (this will be our ROI). We also normalize the layers of this section and since there is an independent Gaussian distribution between r and g, we can obtain the mean and the standard deviation of the r and g values of the ROI. Lastly, we compute the probability that a pixel with chromaticity r and g belongs to the ROI using equations:
where σ(r) and σ(g) are the standard deviation of r and g in the ROI; μ(r) and μ(g) are their means; and r and g are the normalized chromaticity coordinates of the image processed. Therefore, the probability that a pixel belongs to the ROI is equal to the product of the two probabilities. Here is my actual scilab code for this part:
Lines:
1 – 4 > loading of the images and the ROI
6 – 20 > obtaining the NCC for both the image and the ROI
22 – 25 > calculation of the mean and standard deviation of r and g
27 – 29 > determination of the respective probabilities p(r) and p(g) and their product
Let us now apply this to an actual pattern:
Note that this pattern was hand drawn by Gia and uploaded by herself in this blogsite. I want to separate all the yellow parts in the pattern so selected the ROI from the yellow parts in the left pattern:
The segmented image will be:
The segmentation of the yellow areas in the left pattern weren’t complete even if the ROI was taken from that region. But what’s surprising is that the yellow portion in the right pattern was not detected at all, save some small amount of points. With this result, we can conclude that the yellow areas in both patterns are not actually the same when we talk about there r and g values. The similarity in their appearance must have been due to our eyes only. Of course, curiosity tells us to get another ROI from the right pattern this time:
The result is:
This time around, we see all of the yellow areas present in the picture, though the right pattern’s are not complete. We already have established that the two areas are of different values of yellow, and basing from this result, it seems that the yellow in the right pattern are of a wider amounts. This caused a broader Gaussian distribution that covered the yellow in the left pattern, ultimately including it in our segmentation. Here we see a limit of this technique 1) dependence on the range of values the ROI has.
Lastly, the incomplete segments in the right pattern is due to the difference in the values of the yellow in it (remember that the pattern is hand drawn).
II. Non – parametric Probability Distribution and Histogram backprojection
For this technique, the first thing we need to do is to obtain the 2D histogram of the ROI. We do this by converting the r and g values into integers and binning these values in a matrix. The location of peaks of the resulting image, as compared to Figure 7, should match the colors of the selected ROI. After verifying this, we perform histogram backprojection to the original image. Basically, we tag each pixel of the original image in the 2D histogram – if the pixel falls on the peak we give it a value equal to it, and zero if not. We implement this using the scilab code below:
Lines
1 – 20 > same as the code from the parametric part
22 – 32 > creation of the 2D histogram of the ROI
37 – 45 > histogram backprojection; evaluate each pixels (i, j) using the histogram
Using the code above and the ROI at figure 11, the 2D histogram will be:
From visual inspection, it is easily noticed that the peak is in the yellow – orange area of the normalized chromaticity space (NCS) just below the white spot in the middle, which is of the same color with our ROI. The segmented image is:
The segmented image contain both the yellow areas on both pattern, and it can be seen that we actually got the whole yellow portions in the left pattern while some minuscule parts are missing from the right one. Now for the second ROI:
Again, the peak falls on the yellow – orange area of the NCS. It is also noticeable that the area of the peak is larger than the first ROI shown in figure 16. The segmented image will be:
It is obvious from the figure that we were able to obtain all the yellow parts on both patterns. Since the ROI is obtained from the right pattern, I expected to see the whole yellow areas in right pattern segmented. The presence of all the yellow areas in the left pattern confirms my assumption in the previous section that the yellow areas in the right pattern has a wider range of yellow values. (I apologize for the repeated yellows)
From my results in the colored segmentation using the two techniques, I can say that the non-parametric was more effective since with the two ROI possible for the yellow segments, only that technique was able to return all the desired areas.
Technically speaking, this result might be due to the fact that we deal with the actual color values with the histogram (non parametric) while with the Gaussian estimation, we talk about the range of values. This means that a large number of high valued r or g in the ROI might cause a single low valued r or g to be overshadowed in the probability computation.
Let us further examine the techniques on a multi-colored example on the following section
III. Exploration of the Techniques + Application
To start with this section, let us focus on figure 20.
For an easier discussion, let me first name the pokemons on the right side of the figure; from top to bottom: Flareon(red), Jolteon(yellow), Leafeon(green), Glaceon(light blue), Vaporeon(dark blue), Espeon (purple) and Umbreon(gray). I specifically chose this figure because of the variety of colors present in it. I will start by showing the 2D histograms of the ROIs taken from each pokemon:
It can be easily seen from comparison with the normalized chromaticity space that the location of all peaks of the 2D histograms do fall to the corresponding colors of their ROI’s. That being said, I will not focus on explaining this part; rather, I will now show the result of the segmentations done. I will be discussing the results in three parts:
First set are the ones with unique colors:
The whole area of Jolteon was successfully segmented and there is no apparent difference for the segmented area of interest on both techniques. However, it is noticeable that some dots were included in the segmentation. These peaks are the yellow pixels from the other pokemons that weren’t noticeable in the original image.
The segmented areas of Leafeon for both techniques look exactly the same (for me at least), and this is because the ROI has a small range of colors causing a Gaussian curve of size that is just small enough to only cover the color ranges of Leafeon. Same goes for Espeon, with a slight difference on the smaller peaks above it.
The whole area of Flareon was segmented for both techniques, but it can be seen that there are extra areas of lower intensities segmented from the parametric. This is due to the multicolored ROI selected. The red-orange parts from the bottom of the ROI caused a wider Gaussian distribution taking a part of the yellow colored pixels in the whole image. Notice the similarity of the lower peaks from Jolteon’s segmentation in figure 27. For the Non-parametric segmentation, the back projection has only taken the areas with exact colors from the ROI.
Next we go to the blue colored Pokemons:
When I first used this image, I did not think that I would encounter such a cool problem: two different areas of interest with a quite similar color compositions. From figure 20, it can be seen that the body color of Glaceon is the color of the neck fur of Vaporeon while the ear color of Glaceon is similar to Vaporeon’s body. Therefore, even if we segment either of the two we should expect to see parts of the other, even if we only take a specific color. From figure 31, whose ROI is from Glaceon, we get its whole area as well as some parts of Vaporeon. On the other hand, the segmentation for Vaporeon has taken both its area as well as Glaceon’s due to the range of the ROI taken. Here we see the second limit of the segmentation techniques 2) we cannot separate the segmentation of two areas if they are composed of similar range of colors.
The last part is quite ugly (yeah, it really is) but is also interesting, nonetheless:
For Umbreon, which is gray colored, I really did not expect to be able to segment his area solely. Nevertheless, we see Umbreon’s area in the lower right part of the image. What’s quite interesting for me here is the verification that the black area we see in all images is not actually pure black, but a contour of gray black and white. It is also cool that for both types of segmentation, the white area of Eevee on the left-hand side was included even if there is no white areas present in the ROI. However, if we look closely on the 2D histogram of the ROI, we can see that it falls on the center of normalized chromaticity space, on the region of white – gray, which might be the reason why it was included in the segmentation.
Now that we have seen more examples of the two techniques, it is time to wrap things up and compare the two techniques. As you have seen from my figures, both segmentations work and we have successfully cropped parts of colored images. Now when is it the right time to use parametric or non-parametic?
Parametric segmentation, as I have discussed above, obtains a Gaussian distribution from the r and g values of the ROI and then evaluates the pixels of the original images if it falls within the range or not. Thus, a region of interest with a wide range of colors will give an erratic segmentation. Take for example figure 30, where I took an ROI from the body and neck of Flareon, which is colored red and orange respectively. Due to the 2 colors present in the ROI, we obtain a wider Gaussian distribution, which resulted to some minor peaks in the body of Jolteon which is colored yellow and the outline of Umbreon which is also a darker yellow. That being said, parametric segmentation should be used when the ROI is of a single color.
Non-parametric segmentation, if I may reiterate again, uses the 2D histogram of the ROI taken to backproject peaks. Essentially, the process takes a colored pixel from the original pixel, checks its location in the normalized chromaticity space and returns its corresponding value in the segmented image space. Since the algorithm reviews each pixel, the range of colors present in the ROI would not affect the color segmentation. Thus, non-parametric segmentation should be used when the ROI is of varied colors.
So there you have if friends and readers, a discussion of image segmentation. But before I wrap things up, remember how I mentioned that “image segmentation is the selection of a region(s) of a given image based on their similar features such that further processing might be done?” This means that segmentation part is only the half of the definition, how do we do know the further processing? Here, let me give you an example:
I shall leave the code to you, but I will discuss the process I did with the help of Gio Jubilo. (Thanks for the help mate! You can check his blog here.) I used parametric segmentation to separate all the red parts of spider-man, giving me a result similar to the ones I displayed above. I then applied a threshold on the resulting image, making all the peaks have a value of 1 and the rest 0. Next, I find the locations of the 1 in the image, get these locations in its R, G and B layer and play around with the colors. I present to you, the colored superior spider-mans:
and my personal favorite:
I believe that I have done what was asked and I was able to explain my algorithm and how the techniques I employed works; and additionally, I have applied the concepts I have learned to do something new. I could therefore, proudly, give myself a 12. 🙂
References:
[1] Soriano, M., “Activity 7 – Image Segmentation 2014.”
[2] Bulbapedia, “Bisharp,” Retrieved from: http://bulbapedia.bulbagarden.net/wiki/Bisharp_(Pok%C3%A9mon)
[3] “Gia’s Color Study,” Retrieved from: https://gia8.wordpress.com/2010/04/20/patterns-in-color
[4] Paper Hi. “Best wallpapers,” Retrieved from: http://www.paperhi.com/Toplist_Best_319
[5] Onore-Otaku. “Superior Spider-man v2.0,” Retrieved from: http://onore-otaku.deviantart.com/art/Superior-Spider-Man-v2-0-395553543
saveallred 