Real Time JPEG Corruption « Zach Archer Blog

23May/080

Real Time JPEG Corruption

Here's a new site that I worked on:

http://www.fashionbuddha.com/horaciosalinas/

The site seems typical at a first glance, until the glitches kick in... Then you're in for a treat. (You can adjust the glitch rate in the menu.) Here's how the effect works:

The concept + design were crafted by Fashionbuddha Studio. They handed me proof-of-concept code that loaded a JPEG, scrambled its bytes, and rendered it to the screen. Corrupted JPEGs look like you'd expect; some are beautiful, with distortions in their colors and textures; some were nasty or unrecognizable. A few wouldn't display at all, but more on that in a minute...

The site uses JPEG corruption as a method of animation. When the site is idle, a photograph will sometimes crackle to life and progressively corrupt itself for several frames. When you click a menu link, you see a similar effect, but it's reversed: the corrupted frames are quickly pre-rendered in memory, and then displayed in reverse order, materializing from the most corrupted frame to the least.

All the corruption effects are performed inside Flash. Use a URLLoader to load an image, then on Event.COMPLETE, you can access the raw JPEG data as a ByteArray:

var bytes:ByteArray = e.target.data;

You can traverse the ByteArray and change selective bytes, just like editing an Array. (Some people would call this "data bending".) Then you can display the corrupted image by using the Loader.loadBytes method:

_loader.loadBytes( bytes );

Matt Rooney of Impact Resist helped me with the next phase: sometimes, our corrupted ByteArrays weren't recognized as valid JPEGs, and Flash wouldn't display them. Matt dug into the JPEG file structure, and created a class that identified various markers inside the file.

I wrote a quick 'n dirty test harness (warning: contains drugs and boobs) that loads a couple images, and lets you change the value of one byte at one particular position. Using the harness, and the info gleaned from Matt's JPEGInfo class, I determined that there are three "hot zones" where you can garble the data and produce desirable effects:

Scan data. (This is the bulk of the file, containing the image data). These effects are usually relatively weak, and cause horizontal tearing, or sometimes color changes. Applying several corruptions is effective. To see a single corruption effect, drop these values into the harness: [10000/200]
Quantization tables. These are beautiful effects that can affect the entire image. The first half of the quantization data controls the visual complexity (Fourier constants) and the second half controls the overall color. Try these values: [60/40], [61/40], [64/75], [125/110]
Huffman lookup tables. This is the most drastic effect. Try these values: [366/28], [367/28], and explore the surrounding bytes.

It took lots of tuning to make the photo corruption look aesthetically "correct" -- not too intense, not too subtle. The final algorithm applies a number of scan data glitches every frame, and sometimes one quantization or Huffman glitch per frame (and always on the first frame, to emphasize that the glitch is occurring). Some byte values aren't safe to inject (or modify): 0, 1 and 255 have special functions in the JPEG file format, so the site avoids those completely.

If you want to scramble your own JPEGs, but don't want to write a whole Flash harness to do it, then you can open JPEGs using a text editor (such as TextMate) and edit the file as text (with unpredictable results). Have fun!

EDIT: The glitch effects have been removed from horaciosalinas.com, see this post for more info.