As with the other project sketches this is something that I plan on taking up myself, but am posting a running account here as practice for real-time project-blogging. Readers are encouraged to follow along themselves if they get interested or bored or inspired or whatever.

This is an exploration. Don’t know if it will work. It’s intended to be an approach for creating algorithmic images, or for artifying pre-existing images. There are innumerable variations, but I’ve decided on a simple one to try first, just to see if the outcome is anything like what I expect.

I’ve been thinking about this one, on and off, for about ten years. A few years ago somebody recommended I look at the Image Analogies project at NYU, which is really cool if you haven’t seen it. That’s related, but what I had in mind for this was something less practical. Through the years I’ve dabbled in Mathematica a bit and done it wrong, and played in R with neural implementations. All unsatisfactory. But I had an insight a few days back that clarifies it and makes it much more attainable.

And I was thinking of using it in the blog redesign, if it does what I suspect it will. cron jobs, of course.

I’ll sketch the project now, and then report the results of a planning session and all the iterations as I actually make my moves. I expect it wouldn’t take more than a few hours if I were hacking it; I don’t intend to hack it. I intend to do it with unit and acceptance tests, if I can convince myself unit testing is feasible in Mathematica or R or wherever I work it through. Maybe I’ll do it in Ruby or Python, for the testing.

I’ll also add diagrams as I work on it. Right now I’m just going to jot it down. Admittedly, as a project involving bitmaps and some geometry, diagrams would help. And so they will. Just not tonight.

In a nutshell: The project will take an image or set of images as exemplars. It will create a database of scaled color pyramids from the exemplar(s). A color pyramid of scale n with origin at pixel (i,j) is composed of a tuple of four “center” colors, and a tuple of eight “edge” colors. The four “center” colors are the average colors of the four n×n squares of pixels starting at (i,j), (i+n,j), (i,j+n), and (i+n,j+n), respectively—NW, NE, SW, and SE in that order. The eight “edge” colors are the equivalent averages, but for squares 2n×2n on a side, surrounding the four center squares, in the order NW, N, NE, E, SE, S, SW, W.

Now if (i,j) is too close to an edge of the image, clearly some of these square regions will overlap or lie outside the proper dimensions of the picture. We’ll assume that there’s a solid background color, defaulting to white, out there. So a pyramid (of any scale) with origin in the upper leftmost pixel of the image will have its NW, N, NE, SW, and W edge squares lying entirely outside the image, and composed entirely of the background color.

So a pyramid of scale 1 contains four one-pixel squares in the center, and eight 2×2 pixel squares on the edges. The center colors are the colors of those four pixels; the edge colors are the averages of the four pixels in each of the eight larger squares.

Like I said: diagrams will help. Just not now.

Anyway, to preprocess an exemplar image, we’ll be generating a database of all possible pyramids (where every center square contains only image data) of scale 1, 2, 4, 8, 16 and 32 (6×6, 12×12, 24×24, 48×48, 96×96 and 192×192 and pixels, respectively). Just keep a list, maintaining repeated samples if there are any.

I’m thinking a scan of text. Something black-and-white, to begin with at least.

Got the data? OK. Now create and train a Kohonen SOM from each scale’s samples based only on the four center colors. Ignore the nine edge colors for the training period; just cluster the pyramid data at each scale on the basis of the four center features.

All trained up? OK. Now for each pyramid in the dataset, assign it an index value based on the cell of the Kohonen network where it ended up being placed. Some unique identifier so that when a pattern is matched to the trained network on the basis of its four center colors, we can obtain the entire set of training patterns that map to that same cell.

Now we have an image [de]generator: I can specify the center colors of an arbitrary pyramid of some scale. We can look it up by finding the cell of the correct Kohonen network that’s the closest Euclidean match. We can then query the database and obtain a list of exemplar pyramids that match that same cell, and pick one of those at random.

Let’s grab a new image. Something unlike the exemplar, or maybe one just like it. Don’t care. Call this the active image.

Pick a random position in the active image, and a scale. Measure the colors of the pyramid in the active image at that scale, refer to the appropriate trained Kohonen network, and have the database spit out a related pyramid from the exemplars. Now change the colors of the pixels in all the edge squares of the pyramid, in the active image, so that they are closer to the colors in the sampled exemplar pyramid. Average them, weight them, whatever.

So for example if we’ve selected pixel (88,100) of the active image, at scale 2, the edge region that will be changed will span all 144 pixels from (84,96) to (95,107). All but the center 16 pixels of that square will become more like some pyramid in the database.

If a pyramid of a chosen scale and position overlaps the boundary, just imagine the surrounding pixels are fixed at the background color.

Iterate by picking a new pixel and scale. Not sure with what probabilities, but probably some kind of weight over the set of scales. Keep going until things get interesting.

The point, of course, is that the pyramids from the exemplar won’t exactly match the traits of the active image, and that it should get kindof munged up, or even surreal, as pyramids are replaced. One might consider applying pyramids from a scanned text page to a landscape; pyramids from a Dutch master’s still life to a snapshot; pyramids from a Persian rug to one’s cat picture.

We’ll see how it works out.