Our Material Virtualization Process

Hello and welcome to our ongoing series of short posts about the cool technology we’re developing at Primer.

My name is Brennan and I’m the 3D artist responsible for the textures and materials that ultimately end up on your walls in the app.

So let’s get started!

What a Good Material Does

First of all, it has to look like the thing it’s trying to be. And ‘looks’ are more than just pasting a picture on the wall: every surface has reflections, texture, smoothness, hardness, they’re made of physically different things, like a metal vs. a rubber will interact with light photons differently at a physical electron level.

As humans we are shockingly in tune with these differences even if we aren’t aware of them consciously. We can show you a picture of something and ask how heavy it is, or how squishable it might feel in the hand, or even how it might taste, and somehow our intuition and answers are actually fairly accurate.

So part of making a good material from scratch is knowing what makes something feel certain ways in the virtual ‘hand’ of our perception.

What We Use to Build These Layers of Perception

While we can’t codify absolutely everything a physical material does in a virtual sense, we can approximate it with a condensed handful of texture maps - graphic images that tell the device how to display a material.

The first and most obvious one is the diffuse map, which is a picture of the surface by itself.

By itself it’s flat. We could add lighting to this image directly but it would ultimately be like printing out a photo of a surface rather than being a surface. Like when you see a restaurant wallpaper a brick pattern instead of actually using real brick. In fact we go to great lengths to remove any lighting or shadows from this texture, and we’ll get into that further below.

In the early days of video games and 3D rendering we could apply a shine on this diffuse material with one value applied across the whole material. In some places this is enough, you can get the feeling across of a hard plastic vs. a soft rubber, for example, with just the shine - called a specular reflection - but materials aren’t often perfectly uniform and so adding realism is also in adding places where there is more or less shine across the surface. We call this a roughness map.

A roughness map looks like this:

It’s a greyscale image where the values from zero to one, black to white, represent the specular reflection of each individual pixel rather than the entire material as a whole.

In our case, for a tile, typically our maps will have a whiter grout (more rough) and a darker tile (more smooth) as it more accurately represents what the surface texture of those two materials are, and when the app renders the material the lighting will break up along the surface which already starts to make it feel less like a flat photo.

We can also vary the roughness per tile to create more natural variation - even without color in this example you can almost imagine some tiles are darker or lighter, smoother or rougher than others as you follow the lines of the reflection.

The most complex map type is a normal map which is full color: each pixel has an RGB value that represents the XYZ values of a surface. So we can take our completely flat geometry and give it the appearance of depth by giving each pixel a virtual angle of information.

For us and tiles, this is a huge help for giving the grout depth and the tile surfaces the natural dimples, bumps and texture that they would have in real life. When we move around a surface and pick up on all the reflection shimmer and depth, that’s the normal map doing heavy lifting.

Finally, we have a metalness map which describes how metallic something is. I mentioned at the top that metals behave physically different to non-metals with how light interacts at a molecular level and in the computer world it’s demonstrated by two different rendering modes.

We can represent this dichotomy with another texture map, also greyscale with values from zero to one, typically used as zero or one, since a material is either metal or not metal in a strict binary, but there’s some artistic license in between for giving a material a little visual juice that’s not strictly accurate. You can see here we’re faking it - this tile isn’t metallic at all, but adding a tiny bit of grey to the map creates a little more depth and contrast which looks and feels better.

Of course, for tiles and surfaces that are truly made of metal we also make maps that would look closer to 100% white - full metal - and use darkness to represent things like dirt or paint or wear that would occlude the metallic shine from showing through.

Putting it all together, we get these layers:

Turning a flat image into a 3D feeling material with shine and depth and reflection.

As an artist, my mental check is always to imagine scratching it with a fingernail, does it virtually feel like what it should really feel like?

I hope that users of the app get that feeling too, even if they never consciously think about it. We strive not only for visual realism but conceptual realism. No matter if you’re building a dream kitchen or renovating a bathroom floor, we want each tile to feel like a living example of how it might exist in real life - not just a picture or a color, a swatch or sample held up and guessed, but a true surface for you to touch and use every day.

The Process of Going From Photo to Material

Now that we know what each map does, what’s the process of making them?

Often it starts right here, with product images from the brand - they’re very useful for getting a feel for how shiny the product is, how it reflects the light in the environment, getting an idea of the colours and grout, sizing and spacing, that sort of thing.

This is the diffuse map we saw earlier that I make by hand using cuts of the tiles and then adjusting perspective to be perfectly square, then flipping, rotating, adjusting colors randomly to fill out the bigger pattern. Plus the grout: in this case it’s fairly thin but in some patterns the grout is substantially chunky so we’re making decisions about thickness, texture, color and adding that to the diffuse map to make one final image.

We started with just those four tiles but if you pattern them like that the looping becomes distractingly obvious and so ideally we try to generate a map with as much variation as we can fit into the texture size.

You’ll see repeats and adjustments in the above image as we turned four into 25 tiles, but when they’re on the wall the effect is relatively clean in big loops - while humans intuitively pick up on patterns being repeated, most people aren’t specifically looking for tiles that are shifted and so with a bit of cleanup in moving or healing dimples and identifying marks, it’s relatively easy to trick the brain into seeing unique-seeming squares.

The other big thing we have to do is de-light the entire image, which is to say: if the source textures have any sort of lights, shadows, reflections, etc. on them, we have to flatten those out as much as possible.

If the diffuse map includes lights and shadows baked into the texture, when we go to add new lights, shadows and reflections in AR they’ll clash and look funny.

Also as part of the pattern looping, we don’t want to see big gradients or lighting changes across the surface because when it loops the dark side will meet the light side and create an obvious contrast.

Finally, some tile patterns will loop across the seam so if you have half a brick on one side it has to match the other half on the other side. The above tile happens to be all squares which is convenient, but sometimes it can’t be helped:

Like looping across the screen in old arcade games, each tile at the top corresponds with ‘itself’ at the bottom, and each one that goes off the left comes in the right, etc.

So when you loop the pattern, the bricks are always seen as whole and don’t randomly change to a different colour halfway through:

At this point that’s pretty much a working diffuse map: we’ve built it up from corrected source images, randomized things, added grout and details, it tiles cleanly and it’s flat from lighting and gradients.

The roughness, metalness and normal maps can be generated from this image alone in the case of fancy photo-processing software, or more control can be had if you hold on to some of your layers.

Depending on the complexity of the tile I will sometimes make masks and alpha layers for just the grout such that we can isolate a different roughness (grout is almost always more rough than the tile itself).

We can also create a pre-normal map which gets converted into the actual normal map with slightly more specific values for the height differences between the grout layer and the tile layer, and even adding specific curvature and shape to ‘pillow’ tile surfaces where glazing or other surface treatments might create certain curvature to the face and edges.

Either way, in the end we have our four master maps that can be scaled up and down for a variety of uses in any rendering system from desktops to mobile AR.

Of course there’s always some tweaks and adjustments back and forth with implementation, things will look good on the computer but feel wrong in the ‘real life’ of augmented reality once you add lighting, reflections, room ambiance; we’re generating environment maps from the mobile cameras and so on, which we’ll talk about in our next post.

Until then, we’re super excited to be working on this pipeline and launching increasingly cool features with every new release.

Speaking of, we’re hiring!