Image Tagging Digital Image PowerPoint Presentation

Slide 1 -

Attaching textual meta-information or semantic linkages to images By Perry Rajnovic Image Tagging

Slide 2 -

What is a Digital Image? Digital Images are usually defined as an organized display of pixels, often called a bitmap. Each pixel is a numeric representation of the color intensities of that point.

Slide 3 -

What is a Digital Image? Each pixel may be explicitly defined or be the result rendered by a vector or graphics package functionality. These representations include no inherent textual elements or semantic description. Due to the above, images are not easily machine-readable.

Slide 4 -

Why machine-readability? Most searches are done via textual queries, thus there must be a mechanism to link applicable keywords or phrases to images. For blind persons, being able to convey information about the image in another medium would be good for accessibility.

Slide 5 -

Image Contents The contents of an image can be a full description written in prose (i.e. the adage “A picture is worth 1000 words”), or might simply have a few keywords describing spatial, temporal, or emotional aspects. In many cases, accurately identifying the content of images requires human intervention.

Slide 6 -

Identifying Image Contents Many good pattern recognition algorithms exist, however few are able to interpret the patterns extracted. Artificial Intelligence algorithms can learn recognized patterns, but such a system’s flexibility is limited by its predefined knowledgebase.

Slide 7 -

Identifying Text in Images CAPTCHAs (or Completely Automated Public Turing test to tell Computers and Humans Apart) are images which contain a distorted rendering of some text.

Slide 8 -

Identifying Text in Images Their goal is to provide an easy task for humans to do, but that is extremely hard for computer programs to perform equally. For this task, OCR is generally not sufficient enough to extract the text. This is a good example of why machine-readable information should be available.

Slide 9 -

Example Tag Contents As an example of what might be provided to tag an image, to the right is a list of words and phrases to describe this slide’s header. Navy Blue Squares Fade-out Horizontal Bar Minimalist Decorative

Slide 10 -

User Applications Many applications take advantage of image tagging, below are a few examples. Apple iPhoto Google Picasa Adobe Photoshop Elements Generally these programs use tagging for organization and user-defined searching.

Slide 11 -

Web Applications Several Web-based applications are now including tagging for images, as well as other non-image based features. Google ImageLabeler Flickr.com Facebook.com 23hq.com Fotki.com

Slide 12 -

Google Images Luis von Ahn developed the “ESP Game” which could be used to tag images. He presented a Google tech talk about the game as a form of human computation. Google later licensed the technology to create a similar web application called the Google Images ImageLabeler.

Slide 13 -

Google ImageLabeler The ImageLabeler game allows to random users to generate tags that accurately describe images. The tags should be accurate due to game constraints, and gain specificity after several rounds. The computed tags can improve searches.

Slide 14 -

Flickr Flickr is a “Web 2.0” photo hosting and sharing site. Users are encouraged to upload photos, then to name, describe, tag, annotate, geotag, comment on, and group their photos in collaborative ways.

Slide 15 -

Flickr - Tags Tags are words or phrases meant to act as keywords. They are searchable within the site, and can show popular topics. They improve search relevance.

Slide 16 -

Flickr - Geotagging Geotagging is a term for adding geospatial metadata to images such as the latitude, longitude and other directional indications of where a photo was taken at.

Slide 17 -

What are annotations? Wikipedia defines them as: Extra information associated with a particular point in a document or other piece of information. The US DoD defines them as: A marking placed on imagery or drawings for explanatory purposes or to indicate items or areas of special importance.

Slide 18 -

Annotating Images The use of annotations with images can provide several useful functions. Below are some examples: Point out a specific piece of content. Explain some icon or graphic. Summarize the meaning of some region. Provide additional information via text.

Slide 19 -

Flickr - Notes Flickr provides a feature called Notes. It uses a Flash-based implementation of an annotation system. You can dynamically size a rectangular region over a portion of the image, then attach a snippet of text to describe it.

Slide 20 -

FotoNotes FotoNotes is a data format for annotating images. Allows you to embed the metadata directly into the image files for portability. Flickr’s Notes feature is inspired by this standard and accompanying visualization implementation.

Slide 21 -

FotoNotes - More It was developed by Greg Elin. The homepage provides links to groups working with the standard. Additionally, an implementation which works in most browsers is provided as-is for customization.

Slide 22 -

Facebook Facebook.com has a tagging feature that is integrated with “My Photos”. It allows you to add a textual descriptor (tag or person’s name) to a specific point in the image. This allows the module to describe who or what are included in a specific album.

Slide 23 -

Facebook – Tag Display When the images are viewed, placing the mouse over a tag displays a fixed sized square indicating where the tag (person) is located within the image. This enables users to identify objects by visual inspection or by matching the list of contained objects with their tag displays.

Slide 24 -

Facebook – Links Another capability incorporates the site’s concept of friends. If the person you tag is identified as your friend on the site, their name will link to their profile. The site will also count this image in the “photos of” feature on their profile, allowing inclusion of photos added by other users.

Slide 25 -

Other Image Metadata: MPEG The MPEG-7 standard is a “Multimedia Content Description Interface” “MPEG-7 is not aimed at any one application in particular; rather, the elements that MPEG-7 standardises shall support as broad a range of applications as possible.”

Slide 26 -

Other Image Metadata: Adobe Adobe Systems created a new MetaData framework for images called XMP (Extensible Metadata Platform). It is publicly documented, based on W3C standards, built on XML, and is designed to eliminate growing incompatibility for metadata storage.

Slide 27 -

Other Image Metadata: IPTC The International Press Telecommunications Council created standards for the interchange of news data over a decade ago. These standards still persist in their IIM standard, as well as being usable in the newer XMP framework.

Slide 28 -

Joint Term Project By Perry Rajnovic and Mark Zalar Improving Clustering Search Interfaces

Slide 29 -

Term Project For my term project, I will be working with Mark Zalar to develop a new search engine interface It will draw inspiration from all of the top search engines today, along with the enhancements now possible using emerging technologies.

Slide 30 -

Project Goal The Goal of the project is to implement a search site that provides a highly usable interface for query refinement. Our backend will use clustering mechanisms to allow for easy refocusing of search topics Our frontend will use AJAX for flexibility.

Slide 31 -

Frontend Design The frontend will be designed with a technology known as Asynchronous JavaScript and XML (AJAX). This technology allows the site designer to run unseen requests to the server and parse XML-based results in the scripting language for interactivity.

Slide 32 -

Frontend Theory Most clustering based search solutions available today use minimally interactivity. Our theory is that making the ability to harness the power of clustering dynamically as you refine your search will improve results as well as time necessary to finish a search.

Slide 33 -

AJAX Functionality Our site will use AJAX to dynamically reconfigure the clustering menu. This allows a quicker browsing of clusters to identify the optimal range of pages to search within. The menu will also use a novel interface that shows sibling and parent clusters.

Slide 34 -

AJAX Functionality 2 The results will be displayed to the user with some animation. This will help to alert users when changes are made to the order or set of results. Another advantage of this is that users will be more aware of the difference between clusters as they browse them.

Slide 35 -

Search Target This search engine could target both websites and images. Valid keywords improve content knowledge. Clustering would be highly useful in finding an image with a desired scene or set of objects.

Slide 36 -

Example: search “creature” The engine might identify a general cluster of “animal” or “being”. Animal might have more results, so the medium level clusters are shown for that.

Slide 37 -

Example: search “creature” User wants a general discussion of mammals. Selects that cluster. The results change to focus on those related to mammals as a group and in specific.

Slide 38 -

Results Display To provide animation, a similar technology to that found in “TiddlyWiki” will be used. This interface allows topics to be added and removed dynamically with animation. Additionally, extra links can be attached to each topic for more functionality (open in new window, similar items, etc.)

Slide 39 -

Future Enhancements Our implementation will provide a basic mockup of the interface and refinement techniques made available. Several enhancements could be made to this interface that would improve its usability or functionality.

Slide 40 -

Enhancements in Search Taking advantage of a meta-search would allow the clustering algorithm to have a higher volume of data with which to generate data topologies to be explored. Using adaptive search (by userID or global optimization) would improve clustering by choosing ones more often used.

Slide 41 -

Enhancements in Interface Because the site will be AJAX based, a large amount of flexibility is possible with respect to changes in the interface. The browser window is similar to a canvas, with all of the site’s underlying Document Object Model available for addition, modification or deletion.

Slide 42 -

Thank You