Putting the G into GUI

Using graphics rather than text to represent data for superior user performance.

At the dawn of usability engineering, the UIs to databases were on dumb terminals, and they were slow, modey, cumbersome, and used ugly green character-cell displays like this:

Character-cell form

Today, the UIs to databases are on the web, and they are slow, modey, cumbersome, and use ugly multi-colored displays like this:

Same form as a web page.

Yay, progress. Big woot.

Thirty years of technical improvements, and what do we offer our users of business software? We’ve laptop PCs with more memory and speed than the mainframes those dumb terminals used to connect to, but how much does that translate into user performance? Not a whole lot, frankly. There’s little about the web version of the UI above that is more learnable than the character-based version. Nothing there that prevents more errors or makes error recovery easier. The information conveyed is about equally clear. Count the keypresses and mouse clicks, and the normative time to complete a task is about the same.

A bitmapped graphic display allows proportional font, which is more compact than fixed fonts. I suspect it is easier to read, but I haven’t seen any studies to confirm that for reading fields of data. The web UI allows color and graphics, but too often these are not used to make the user’s job easier, instead serving more as gratuitous decorations. At best they are used to group the data fields into a visual hierarchy. Remove the color and graphics from the web illustration above, and the interactions is no harder; they might even be a little easier.

Mouse-able links, buttons, and menus are great for a graphical user interaction, but most of the interaction for the UI illustrated above is reading and typing text, not viewing and manipulating graphics. A mouse does little good when your user’s fingers are on the keyboard. The truth is your typical graphical user interface for a business application doesn’t have a whole lot that is graphical. That is true for most thick client apps as well as for web-based apps. A GUI doesn’t do too much if all you use it for is to show a bunch of text fields.

And sometimes that’s okay. For many applications all you need to optimize user performance are well-arranged, well-labeled, and well-validating text fields. Yes, user performance won’t be a whole lot better than an app designed 30 years ago, even though computer performance has improved by a factor of 1 meg, but perhaps that just goes to show that technology changes but users don’t.

However, text isn’t always the best way to convey information. Sometimes a picture is worth a kiloword. Graphics such as maps, graphs, symbols, pictures, and diagrams are often better for certain tasks or users. If you’re limiting yourself to text fields in your designs, you might be missing out on the full potential a GUI can provide. This post covers how to use graphics not to visually group and rank data for the user but to represent  the data themselves to the user.

When to Get Graphic

Graphic data representations have disadvantages that should not be ignored. While there are cultural conventions for various graphic representations, often they must employ arbitrary visual codes that need to be taught to the user (e.g., through a key). This adds to the learning and memory burden necessary for users to use graphic representations. Graphic representations tend to require more screen real estate than equivalent text representations. For example, a table showing a schedule usually takes less space than a Gantt chart showing the same.

Table listing start and end dates.

Gantt charge showing start and end points.

The failure of icons as labels for commands is due largely to these limitations: inadequate real estate (e.g., 16×16 pixels) and arbitrary coding.

So graphic displays aren’t for everything. It’s all good fun to watch a geographic display of shoe purchases, but it doesn’t really help the user much, except for all those calceogeographers out there (both of them). Actually, even calceogeographers would probably want a different kind of map display than linked above. A graphic display can mean a serious investment, both of your developers and your users, so they should only be used when they have significant advantages. Graphic data representation should be considered in the following situations:

To communicate about concrete or quantitative objects and attributes. Language is generally better for concisely conveying free-form, abstract, and qualitative information. It’s easier to say, “I find meaning in my life through brutal domination of the weak, merciless revenge on my enemies, and cute kitties,” than to illustrate the same thing. On the other hand, graphics better handle structured, concrete, and quantitative information, such as what cute kitties, what kind of revenge, and how much domination.

To be consistent with task or user representations. The data representation should match the most task-effective mental representation. Otherwise, the user will need to mentally translate between text and graphic representations. For example, if the users’ task is to specify a geographic region around a collection of ground features, then the app should show a map of the ground features for the users to draw a region around, rather than requiring users enter as text the coordinates of the vertices of the region. Generally, the format of the user’s input and output should match the app’s. Graphic I/O needs graphic representations, while text I/O needs text representations. For example, if users use your app to identify faulty components in electronic devices they work on, then the app should show a picture of the device, highlighting the faulty component. On the other hand, if the users merely need to identify the name of the faulty components in order to order new ones, then it’s preferred to have a text field naming the component. If the user needs to know the location of a business in order to travel to it, then show it on a map. If the user needs to know the location of a business in order to mail something to it, then an address field is better.

To provide context for human judgment. For showing a single numeric value, a graphic like a bar chart consumes more space than a text field. However, the bar chart also provides additional implicit information that a text field lacks. By choosing a range of values for the bar chart you are communicating to your user the range of reasonable values. If a bar extends nearly the full length of the chart, then your user will infer that whatever the value being represented, it must be a lot. A graphic can provide a sanity check on a user’s input whether it was done graphically or by text. Did he just select Manhattan NY or Manhattan KS? A map would make the difference clear. A labeled graphic such as a bar chart shows other values in relation to the current value, allowing the user to see at a glance how close the current value is to them. A text field requires the user to recall the value of interest and perform mental subtraction. By marking task-significant points on a graphic (e.g., thresholds for actions), you can help your users anticipate things. For displays with values that change in real time (e.g., process control), a graphic representation also provides excellant indications of rate of change, allowing users to judge when a value will reach a significant level. Graphic representation of data brings the advantages of analog displays to the digital world.

To assist the user at pattern recognition. Graphic representation is essentially synonymous with data visualization. By converting large amounts of data to a single graphic, a users can spot overall patterns, such as central tendencies, outliers, and trends, that would be lost in a forest of text fields. Graphic representations work well for tasks involving searching or scanning for approximate magnitudes or frequencies of values. A text field allows the user to see a precise value at glance -out to however many decimal places you provide. However, graphic representations allow the user to see the big picture. Often the latter is more important.

To allow user parallel processing. Humans have a limited ability to conduct parallel processing of two tasks if one task is primarily verbal and the other task is spatial. If your users have to do two tasks at once and one is verbal (e.g., talking to a customer on the phone), you can improve user efficiency by making the other task as spatial as possible (e.g., allow the user to manipulate the user’s account by drag and drop rather than typing in commands or values). The more one task is separate from the other, the more graphic representations can help prevent embarassing spillover between tasks.

To overcome a language barrier. Graphic representations can help when you cannot rely on your users being able to read in your chosen language. However, it should be remembered that graphics have cultural baggage of their own. For example, you can’t always depend on users reading your graphics from left to right -some people tend to go right to left if that’s the convention for their language.

Scales of Measure

When it comes to creating graphic representations, cartography knows its stuff, having been making sophisticated graphic representations of the Earth for centuries. The content of this post draws heavily from cartographic traditions, particularly from Alan MacEachren’s How Maps Work (1995). From cartography, we learn of various graphical dimensions or ways to encode data graphically. The right graphic dimensions for an attribute depend firstly on the scale of measurement of the attribute. The possible scales of measurement are listed below in order of the amount of information available in comparing two values:

  • Categorical: Different values of the attribute indicate a qualitative difference with no implication of one value being “more” than the other. Also known as “nominal” scale of measurement, examples include gender, flavor, construction materials, color of hair (if any). Categorical data are usually stored as strings or as arbitrary numeric codes.
  • Ordinal: Values indicate different amounts or degrees, but the degree of difference between any two adjacent values is not necessarily constant. Examples include priority, rank, month, and general baldness (not bald, partially bald, ping-pong ball). Ordinal data are usually stored as integers, but characters or strings may also be used.
  • Numeric: Values indicate different degrees where the degree of difference between any two adjacent values is constant. Comprising interval and ratio scales of measurements, this includes distance, time (in constant sized units), money, speed, attitude (depending on how it’s measured), and percent of hair cover. These data are usually stored as integers or float/double.

When creating a data model for an app, you cannot use a scale a measurement higher than that of the underlying phenomenon you’re modeling. For example, you cannot impose a ordinal scale of measurement on gender -neither gender is superior to the other, no matter what my wife says. However, you can “downgrade” a scale of measurement, converting a numeric attribute to an ordinal one (e.g., changing the attribute of length in inches to simply small, medium, and large), or converting numeric or ordinal to categorical. Generally, you shouldn’t because this throws away information that a user may need. However, sometimes you can make users’ task simpler by downgrading the scale of measurement to make attributes fit the task better. For example, if all a user needs to know is, “should I discharge the Halon bottles now?” you should probably simply display the categorical indication of FIRE or not, rather than displaying the current smoke density in million particles per liter. For the purpose of choosing graphic dimensions to represent attribute values, what matters is the scale of measurement you present to the user.

Graphic Dimensions

Graphic representation of data usually means encoding the attribute values of a data object into perceptually distinguishable visual differences. As discussed in an earlier post, the human visual processing system distinguishes images using three basic dimensions: color, spatial frequency, and orientation. Any of these can be mapped to the values of an attribute. For example, one could map color to Super Weapon Status, with green meaning Good To Go.

Color

Color actually has three dimensions that can be each used to encode attribute values: hue, brightness, and saturation, the same basic components used to define screen colors on a monitor. Heck, add transparency (alpha), and you’ve four dimensions for color. Transparency may be a particular intuitive way to represent impermanence. However, using transparency requires you employ some sort of background, otherwise, it’ll be easy to confuse it with the other three components of color.

Brightness (or color value) works well for ordinal and sometimes numeric scales of measurement. It is especially good for indicating density, intensity, rates, and other ratios. Saturation is is best for ordinal scales but it’s rarely used. One of its problems is that humans, even those with good color vision, can only readily recognize a two or three different levels.

Brightness coding threat scale, saturation coding probability ranking.

Hue is best for categorical scales of measurement.

Hue encoding defense type.

Geeks may associate the colors of the spectrum with electromagnetic frequency and thus rank hues from blue to red, but mere mortal users don’t. To them, hue is a qualitative visual difference and thus should be used to encode qualitative information. If you want to see the failure of humans to rank hues, visit some karate dojos and check out how they rank their colored belts. Not only do the ranks of hues fail to be consistent with electromagnetic frequency (e.g., jumping from orange belt to purple belt), but they are not even consistent across karate styles (e.g., for some green belt is higher than blue, for others vice versa; for some purple is above blue and green, for others its below). Those 23-color maps of percipitation? DHS’s five-color National Threat Advisory? Not a good source of inspiration. That said, hue can be effective for ordinal or even numeric scales of measurement if you:

  • Use only a portion of the color circle (e.g., blue through purple to red).
  • Use culturally well-established color stereotypes (e.g., green for safe, yellow for not-so-safe, red for AAAAAUUGH!)

There are many other issues with using hue-coding, but that will have to wait for another post.

Spatial Frequency and Orientation

Orientation works is good for ordinal scales of measurements. Arrow imagery, for example, can be used to encode general trends as orientations. Sometimes even pictorial associations can used, for example to distinguish between closed, partially open, and wide-open.

Hue for defense type, orientation for secured-threatened-breached.

Spatial frequency has two dimensions for encoding, one being size, which corresponds to the lowest dominant frequency of an image. Size is great for numeric scales of measurement. In particular it’s good for indicating the total amount of something (as opposed rates, which work well with brightness). The attribute value can be encoded by the size of an object on two axes (height and width), but it’s usually better to use only one axis.

Size encoding time until arrival.

While the physical length of an image is linearly related to its perceived length, the same cannot be said for the area of an image.

Spatial frequency also has the less commonly used dimension of crispness, which corresponds to the higher “harmonic” spatial frequencies. Like saturation, its best for ordinal scales of measurements and only a few levels are likely to be readily recognizable by users. It can be used to suggest the certainty of the data:

Size for time until arrival, crispness for certainty of time.

You can use different shapes to represent data with a categorical scale of measurement. This is a type of encoding that emerges from orientation and spatial frequency since the human visual system distinguishes shapes by their differences on these dimensions. Shape encoding includes using icons, where the shape is a realistic or stylized picture intended to aid users in learning and remembering the graphic codes. Putting aside memorability, you make shapes maximally distinguishable by each having different combinations of orientation and spatial frequency.

Shape encoding prisoner type.

Ideally, a set of shapes should be both easily distinguishable and memorable. However, when the attribute is relatively abstract (such as the Prisoner Type attribute above), reliable pictorial associations are hard to come by, and you may get better overall human performance using arbitrary but highly distinguishable shapes.

Take a shape and repeat it, and now you have a pattern for encoding an attribute value. Differences in the rate of repetition in a pattern constitute density encoding, which can represent different attribute values on an ordinal scale of measurement. Like brightness, it tends to suggest rates or ratios (like, uh, density), rather than raw amounts.

Shape for prisoner type, pattern density for cell conditions.

The regularity of the repetition can represent a (binary) categorical-scaled attribute, perhaps whether conditions are organized or chaotic. The shape you use creates a pattern that can also be used for encoding categorical data, much like individual shapes. Add crispness, and you’ve a texture for encoding that can include an ordinal scale of measurement.

Position

There is another means of graphically encoding data, one so significant, so central to human vision, so blindingly obvious, that it’s easily forgotten. The position of a graphic can also represent an attribute value. The value of the attribute is correlated with the distance between the graphic from some visual reference point. The distance can be linear, such as in a bar graph, or radial, such as found in a rotary gauge.

Gauges using radial position coding.

Position coding is best for numeric scales of measurement, and is particularly suited for indicating the spatial position of objects. In other words, to make a map. However, it can be used for any arbitrary attribute, such as found in graphs. For example, it can represent time.

Timeline: position encoding points in time.

Time

There’s using a graphic dimension to represent time, then there’s using time as a graphic dimension. I almost hate to even mention this one, given the web’s frequent abuse of animation in gross violation of the Geneva Convention. But here goes.

If there’s space there must be time. There are several graphic dimensions that use animation and time to encode attribute values. The first is sequence of an animation, that is, the order of the frames in the animation cycle. Like position is good for encoding spatial position, sequence is good for encoding temporal position, representing changes of something over time. Just as maps have a scale of miles so that a large geographic distance fits within the user’s visual field, sequence encoding can have a scale of hours so that the progress of events fit within the user’s viewing time (e.g., to study developing precipitation on a weather map). In addition to representing literal time itself, sequence can encode the value of ordinal-scale events. Think animated representation of Top-Ten lists.

In addition to the sequence dimension, the rate of animation can encode a numeric-scale attribute. For example, one can imagine a browser throbber where the speed at which it completes its animation cycle is proportional to the bits per second transferred. Quite apart from the rate of animation is the step size or “jumpiness” of the animation. Jumpiness is probably a good for indicating the smoothness or efficiency of a process. For example, a browser’s throbber can use jumpier animation to represent a poorly responding server to help the user distinguish between slow downloads due to server demands from those due limited bandwidth (e.g., because of those seven movies the user is downloading simultaneously).

Finally there’s synchronization coding. If multiple items are animated, their synchronization can encode a (binary) categorical-scaled attribute. This should be a good for representing the coordination of processes.

As I alluded to above, because animation attracts (and distracts) attention so much, you have to be careful using animation to encode attribute values. Generally, it should be limited representing the attributes of an on-going process that the user must monitor closely. No animation of any kind should be used for tasks peripheral to the user’s primary concern, which is why I have no examples of time coding on this page.

Combining Graphic Representations

Multiple graphic representations can be combined redundantly and multivariately. In redundant combinations, the same attribute is represented by two graphic dimensions at once in order to make a stronger graphic representation. For example, combining saturation coding with brightness can make a very compelling code for intensity or rate.

Legend showing redundant combination of brightness and saturation coding.

Often other dimensions are combined with position coding. A bar graph combines position coding (the end of the bars) with size (the length of the bars). Needle gauges (or GUI simulations thereof) combine position with orientation. Line graphs have position coding, but the line segments in the graph also have orientations as emergent codes for trends and rates.

In multivariate combinations more than one attribute is encoded in the same graphic using different graphic dimensions. For example, brightness may be used to encode feedback rates and saturation may be used to encode thermal intensity.  The shape of a symbol can mean type of device, while the weight of the shape outline (size coding) can indicate if it’s on line or not, with crispness thrown in to encode the certainty of assessment. And while you’re at, use a patterned fill to represent destruction of the device via an impressive explosion.

Diagram of network, with each component using multiple multivariate graphic codes.

Okay, maybe that’s getting a little carried away. But combining two or three attributes in the same graphic can make for a very compact method to communicate a lot of information. Earlier in this post I knocked graphic representation for being real-estate hogging. Multivariate combinations are one way to give some real estate back.

As with redundant combinations, position coding works particularly well in multivariate combinations with other graphic dimensions, making position the overlord of graphic dimensions. This is what maps do: position encodes the location of geographic features while size, shape, and color indicate attributes that may be found in that feature.

Editing Graphic Representations

In addition to considering the attribute’s scale of measurement when selecting a graphic code, you should also consider the ease of editing of the attribute values. GUIs provide the means to directly manipulate orientation, size, and position of graphic representations through drag-and-drop. This is because any change in orientation, size, or position necessarily means transversing a distance across the display, which is what drag and drop does. If the user changes the orientation, size, or position of a graphic representation, there should be a corresponding change in the underlying attribute value.

The other graphic dimensions (color, crispness, density, regularity, texture, and the time dimensions) cannot be directly edited in a GUI. To modify these, the user has to work through a separate mechanism. This includes:

  • Enter the values as text in a text box or spinner (e.g., typing in a numeric value)
  • Selecting values from a menu such as a drop-down list, in which case it’s best to include in the menu both the graphic representation of the value along with the text representation.
  • Dragging and dropping using a separate graphic representation that encodes the possible values with size, orientation, or position (e.g., a slider control).

The upshot is that you should use position, size, or orientation (in roughly that order of preference) to encode attributes the user needs to change, especially if the user will select values by their look in the graphic representation. Values the user doesn’t change can be relegated to the other graphic dimensions. However, even when attributes are edited by direct manipulation of a graphic representation, it’s often a good to also include editable text representation of the associated value. This allows the user to copy the value in case it is needed elsewhere. Also, sometimes the user may have a precise numeric value in mind which they can type-in faster than they can drag-to. When they do this, the graphic representation should immediately change to reflect the new value, providing the user with unambiguous graphic feedback.

In general you may want to reinforce graphic representations with text representations of the same attributes, like I’ve been doing in these examples. This reduces the learning curve otherwise necessary for graphic representations. If space won’t allow text fields in addition to the graphic representation, then consider using tooltips to show the value as text. You can also follow the cartographic example and include a legend, possibly controlling its visibility with an option under Help, leaving it visible by default.

Bring us Graphic Controls

It’s time we make our graphical user interfaces more graphical. Not everything should be a boring form or table of text fields! With graphic representations, you and your app can take over the world!

By now you’re probably saying, “What an inspiring exhortation, Dr. Zuschlag, but you’re obviously mad.” Compared to tossing some combo and text boxes on a web page and linking their values to a database, graphic representations mean a tremendous and possibly prohibitive amount of additional work. Well, that may be true, but only because you already have combo and text boxes made for you. We need the equivalent for graphic representations, a set of controls or tools that allow you to easily bind database values to graphic dimensions like color, size, orientation, and position. We should be able to define the graphic codes for database values as easily as text lists. Only after that’s done can we say we’ve freed database apps of their 30-year old prison of green-glowing cells.

Summary Checklist, I Mean Table

Problem: Graphic representation of data.

Potential solution: When the task calls for graphic representation of data, choose graphic dimensions to match the data type as shown in the table below:

Dimension Numeric Ordinal Categorical Notes
Color
Brightness Fair Good Poor Good for rates, intensities, and ratios.
Saturation Fair Good Poor Limit to three levels.
Hue Fair Fair Good Numeric and ordinal performance depends on special circumstances
Transparency Poor Good Fair Good to suggest impermanance
Orientation &
Frequency
Orientation Fair Fair Good
Size Good Good Good Good for raw quantities. Generally use one axis.
Crispness Poor Good Poor Good to suggest uncertainty. Limit to three levels.
Shape/
Pattern
Poor Poor Good Can have pictorial associations for memorability.
Density Fair Good Poor Good for rates, intensities, and ratios.
Position Good Good Good Especially good for representing location.
Time
Sequence Poor Good Poor Especially good for temporal order
Rate Good Good Poor Good for speed and other rates.
Step size Fair Good Fair Good for efficiency
Synchronization Poor Poor Good Good for coordination

Curses! My Plot is Foiled.

30 kB and that’s near my limit for a post. But you haven’t heard the last of me! Watch this space for Graphic Data Representation II: This Time It’s Positional.

Comments are closed.