How to upload Image data to Material

[xumo]

Hi, I hope you can give me some hint.

I'm working with MaxST, and augmented reality library and want to use it with filament.

What I need is to figure out how to send the data image from the camera, Max provides a byte[] array each frame to a filament Material.

I'm starting with the hello-camera example in filament but I don't know it stream.setAcquiredImage can work with an array of bytes.

Any hint will be highly appreciated.

PD: Thanks for his amazing library

[prideout]

android-stream-test shows how to connect pixels from ImageReader to setAcquiredImage, this could be helpful to you. I am not familiar with MaxST, I would hope that it proffers access to the camera image in a more efficient way than just an array of pixels. At first glance it seems to support ARCore, so you should be able to use a texture id, in which case, you would not use setAcquiredImage, instead you would use Stream.Builder().stream(texid). For an example, look for CANVAS_STREAM_TEXID in StreamHelper.kt.

[romainguy]

It would also be helpful to know what the pixel format of that byte array is. If it's RGB, you can use it to upload to a regular Texture object. But like @prideout said, this is not efficient.

[xumo]

I will try to look via ARCore, but is seems that they only use it when its available and its nice to have support for a bigger range of phones.

In their Android example they seem to expect the image format to be Yuv420_888 or YUV420sp. They provide an OpenGL approach where they upload the data like this:

GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
		GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureNames[0]);
		GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE, image.getWidth(), image.getHeight(), 0,
				GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, yBuffer);
		GLES20.glUniform1i(textureHandles[0], 0);

[hillesland]

That's not the efficient way @prideout was talking about, but it's generally usable. We built our AR experience well before ARKit/ARCore and that was sufficient for us; in fact, we still do that on iOS (but not through Filament).

It's possible you still want to use the textureId + stream interface for Filament in addition to the byte array for MaxST (which I'm not familiar with). In other words, you'd be asking the camera system to provide the same image twice - once as a byte array, and once as a texture. Although I've done each, I've never tried both at the same time. You don't actually need ARCore to do that (It's a camera interface)

Having said that, here's how I think you'd do a byte array version in Filament:

To use the byte array, you'd need to use filament::Texture::setImage. Remember to set a callback for releasing memory (I'm sure there's an example somewhere for that).

The code you show looks only like using "Y". You can write a material (shader) that will take the Y in one texture, and the UV in another (I think it's half size) and convert in the shader, if you want RGB to render.

Knowledge caveats:

1. I've only done the camera stream by texture id with Filament, and it's through our own java interface.
2. I've only used the byte array sort of thing for a camera on iOS, and it wasn't with Filament. It was a YUV420 sort of thing though (someone else wrote this part)
3. We are using Texture::setImage, but it's a lower frequency than a camera, used for something else, and hasn't shipped yet.

[romainguy]

@xumo Ah so looks like you need to upload two textures and do the YUV conversion yourself in the shader.

@hillesland Filament handles the YUV conversion for you on iOS when you use our external texture APIs. Under the hood we sample two textures, etc.