Building a Color-based Image Search Engine in Ruby

It is said that a picture is worth a thousand words. And in many ways the words in pictures are colors. Colors are an integral part of our life, and we can’t deny their importance.

While looking at an image, we have often tried to identify the color on a part of it. We have all tried doing it but never did it in details. When asked to identify colors from an image, we tend label them using specific color names, such as red, blue, and green. However, if we are asked to extract the 30 most prominent colors in an image our eye can’t detect or identify them quite as easily. Camalian is all about this. It helps you extract colors from images and then play with them.

In this article, we will take a peek at what color spaces are all about, what the Ruby gem Camalian has to offer, and how it can be used to make a simple image search engine that uses colors to identify and distinguish them.

Colors Space

Before we start, let’s first understand some basic concepts about colors. Images presented on our screens are represented using a two dimensional array of pixels. Although image files may be encoded in different ways, the crude representation after decompressing and decoding the data is same. In this 2d array based representation, each pixel in a color image has three components: red, green and blue. Although pictures printed on a paper are also a two dimensional surface of dots, the dots themselves are usually a mixture of four component inks: cyan, magenta, yellow and black. These among some other different techniques are used to represents colors are called color spaces. Some of the most popularly used color spaces are RGB, CMYK, HSL and HSV. CMYK is mostly used in the printing industry while all others are used in digital media.

RGB Color Space

Any physical electronic media like CRT screens, LCDs, or phones that transmit light produce color using three components: red, green, blue. The human eye can detect millions of colors by stimulating three types of colors receptors in eye. You can relate those receptors to R, G and B.

Ideally, each color component is stored in a byte whose values can range between 0 and 255.

HSL & HSV Color Space

Arranging RGB color space on a cube is rather challenging. The results of trying to represent it on a cube and or color wheel is poor. While working with million or colors, each tint of color can’t be aligned properly on RGB color space.

To overcome this problem, in the 1970s researchers introduced HSV (Hue, Saturation, Value) and HSL (Hue, Saturation, Lightness) color spaces. Both of these color spaces can be aligned on a color wheel properly and make it easier to identify various tints of color on it.

Ruby Gem for Colors

Camalian is all about colors. One of the simplest thing you can do with Camalian is identify each tint of color used in an image.

Let’s say we have an image with 15 distinct colors.

Identifying the colors from the swatch is certainly easier than identifying them from the image itself. Moreover, this is a simple image, and real photos captured are often far more diverse when it comes to their color palette. Extracting the color values from the image requires some pretty tricky bits of code, and that is where Camalian comes in. It does these tricky things for you so that you can extract color related information from an image with ease.

Getting Started

If extracting colors with Camalian is easy, installing with it is even easier. You can install the Ruby gem by executing:

gem install camalian

And to use this gem, you can require it directly into your Ruby script:

require 'camalian'

Extracting Colors

To extract colors from an image, we first need to load it into memory, and use methods on the image object:

image = Camalian::load( File.join( File.dirname(__FILE__), 'colormap.png') )
colors = image.prominent_colors(15)
puts colors.map(&:to_hex)

This snippet of code loads an image named “colormap.png” from the directory where the script resides and extracts the 15 most prominent colors from it.

To run it, save the file as “color_test1.rb” and run it in shell by ruby color_test1.rb. It should produce output similar to the following:

["#318578", "#41b53f", "#2560a3", "#359169", "#2154b1", "#4dda15", "#1d48bf", "#1530dc", "#296d94", "#193dcd", "#3da94d", "#45c131", "#3da84e", "#2d7986", "#193cce"]

And it’s that easy! We have just extracted 15 colors used in the above image. Can you imagine trying to do it using loopy code, or worse, with your eyes? Let’s dial things up by a notch. This time, we will try to use Camalian on an image with more details:

By running the same script on this image we get the following:

[“#210b03”, “#723209”, “#974d09”, “#ae5d08”, “#c77414”, “#d77f15”, “#ffea54”, “#94651f”, “#b66a15”, “#c25f06”, “#fdd94d”, “#d39a39”, “#efa540”, “#fffffe”, “#fff655”]

Attempting to visualize the array of color values produced above gives us something like this:

The palette is good, but there is not specific pattern in the colors extracted. Let’s sort the color values by similarity and see if that helps. All we need to do is call one more function before actually printing the array to console:

colors = image.prominent_colors(15).sort_similar_colors

But what if we wanted to extract colors that are relatively lighter? May be we want colors that are only 40% dark, or in other words have a lightness (in HSL color space) value between 0 and 40. All we need to do is:

colors = image.prominent_colors(15).light_colors(0, 40)

Making the Image Search Engine

Now that we know how easy it is to deal with colors using Camalian, let’s build a simple web application that allows you to upload images and have them indexed by color. For brevity, we will skip the various details involved in building a Ruby application. Instead, we will focus on specifics that deal with colors and Camalian usage.

As for the scope of this Ruby application, we will limit it to handling image uploads, extracting colors from the image before storing them, and searching for uploaded image based on chosen color and threshold.

Below is a model diagram to explain the structure of our application:

Every image uploaded is represented using a PortfolioItem object. Each Color object represents unique colors discovered through uploaded images, and finally PortfolioColor represents the relation between each image and colors found in it.

Most bits of the application are pretty standard, especially when it comes to handling image uploads, creating model entities and persisting them to database, etc. In case you are a Ruby developer, those should be no-brainer. Below is the method that is used to extract the color from the uploaded image:

after_save :extract_colors

private
  def extract_colors
    image = Camalian::load(self.image.path)
    colors = image.prominent_colors(self.color_count.to_i).sort_similar_colors
    colors.each do |color|
      unless c = Color.where(r: color.r, g: color.g, b: color.b).first
        c = Color.create(r: color.r, g: color.g, b: color.b, h: color.h, s: color.s, l: color.l)
      end
      self.colors << c
    end
  end

This helps by extracting the color palette and saving to the database. Notice how we extract only a specific number of prominent colors (something that the user can define when uploading the image).

When a user submits an image through the form on the web UI, the image is received through a post handler, and a new PortfolioItem is created for it. This method, extract_colors, is invoked whenever a portfolio item is persisted to the database.

To be able to render the color palette on pages, we use a simple helper:

module PortfolioItemsHelper
  def print_color_palette(colors)
    color_string = ''
    colors.each do |c|
      color_string += content_tag :span, ' ', style: "display: block; float: left;  width: 35px; height: 35px; background: #{c.to_hex}"
    end
    content_tag :div, color_string.html_safe, style: "display: inline-block;"
  end
end

It essentially creates a div with small square spans, each with their background color set to one of the extracted prominent colors.

Finally, to implement search we have to use some math and logic:

class PortfolioSearchForm
  include ActiveModel::Model
  attr_accessor :color, :similarity
  validates_presence_of :color, :similarity

  def color_object
    @color_object ||= Camalian::Color.new(self.color)
  end

  def color_range(color, level)
    (color_object.send(color) - level)..(color_object.send(color) + level)
  end

  def colors_by_rgb
    level = self.similarity.to_i * 255 / 100.0
    Color.where(r: color_range(:r, level), g: color_range(:g, level), b: color_range(:b, level))
  end

  def colors_by_hsl
    level = self.similarity.to_i
    Color.where(h: color_range(:h, (self.similarity.to_i * 30 / 100.0) ), s: color_range(:s, level), l: color_range(:l, level))
  end
end

With the colors_by_hsl method, we can fetch all the Color entities that match our query. And, with these we can identify all the uploaded images and render our search result page. The query itself is rather simple. Given a specific color, and a similarity value, a range of values is computed for each color component.

And that is pretty much all the hard parts.

Trying It Out

You can find the full code on GitHub. You can deploy an instance of this app to Heroku, or try it out locally:

git clone https://github.com/nazarhussain/camalian-sample-app.git
cd camalian-sample-app
bundle install
rake db:migrate
rails s

The final application looks something like this:

Once the application is running, point your web browser to http://localhost:3000. Using the form on screen, upload a few images of varying color palettes. Then, to search images by color, use the search field on the top right corner. The threshold dropdown allows you to specify the dissimilarity tolerance for matching the colors of the images.

What’s Next?

The demo application that we built in this article is rather simple, but the possibilities are endless! Some other practical uses of this library can include:

• Restricting users from uploading dark profile pictures
• Adapt the site’s color theme to some picture uploaded by the user
• For design competitions, automatically validate submissions against color palette requirements

You can further explore the library on GitHub and check out its source code. Feel free to report bugs by creating issues or contribute by sending pull requests.