Scaling the CSM model resolution

[oleg-alexandrov]

I got a suggestion from some folks who are using LRO NAC and who found CSM so much faster than the ISIS camera model. One thing that CSM is lacking is being able to reduce the resolution of the cameras, in a way consistent with the ISIS "reduce" command. Being able to do things at lower-resolution can be quite useful. Note sure what it would take to use a scale factor with CSM. Scale the dimensions, and apply a scale factor each time the input or output are pixels?

[jessemapel]

There's a few obvious approaches here:

1. Add an optional scale keyword to the ISD
2. Modify the ISD parameters to "double" the pixel sizes

Either of this would require the image to also be down-sampled. What tools are users using to do this? Applications like reduce are outside of the CSM spec as it doesn't deal with any pixel values.

[steo85it]

Hi! I might be one of those "folks who are using LRO NAC" and indeed I would still be interested in trying this. Are you saying that using reduce and then modifying the json "manually" to adapt the pixel sizes (by changing the image_lines and image_samples values) would not work? That's the approach I meant to test...

[oleg-alexandrov]

It should work. Most of the model, including the bookkeeping for quaternions, positions, velocities, and times, will not change.

When it comes to what to modify, the key parameter in the model state (not in the original ISD which I don't know that well), is m_intTimes, which is the delta time per line width. Likely one need not change m_intTimeStartTimes (first line time), and m_intTimeLines (which I don't know what it does).

Also need to change m_nLines and m_nSamples, obviously.

The definition of model state is in: https://stereopipeline.readthedocs.io/en/latest/examples/csm.html#csm-model-state

ASP's cam_tool can be used for sanity checks as it can shoot rays from an original camera to the ground and back with a second camera.

[oleg-alexandrov]

Also need to change the optical center (m_detectorLineOrigin, m_detectorSampleOrigin).

The CSM model also has some weirdness in places when it comes to half a pixel offset. I am not sure they will have an effect. One can mapproject with original image/camera and modified image/camera and watch for subtle shifts.

[oleg-alexandrov]

Need to change focal length too, I believe, by same amount. (m_focalLength).

[steo85it]

Oh great, I didn't think about those at all. Thanks for the hints, I'll check this out.

[steo85it]

Your suggested "tweaks" seem to work (almost) perfectly, thanks @oleg-alexandrov!
Here below is the summary of a small test I did on LROC NAC image M129050120LE.

Original .cub vs .csm (cam_test result)

Image dimensions: 2532 52224
Number of samples used: 13598

cam1 to cam2 camera direction diff norm
Min:    5.66395e-10
Median: 1.97569e-07
Max:    7.79053e-06

cam1 to cam2 camera center diff (meters)
Min:    4.55051e-08
Median: 3.2549e-05
Max:    0.334139

cam1 to cam2 pixel diff
Min:    2.11018e-05
Median: 0.00969859
Max:    0.390455

cam2 to cam1 pixel diff
Min:    1.35774e-05
Median: 0.00977334
Max:    0.390295

Reduced .cub vs "tweaked" .csm (cam_test result)

Image dimensions: 253 5222
Number of samples used: 159

cam1 to cam2 camera direction diff norm
Min:    3.07541e-06
Median: 1.73445e-05
Max:    0.000145736

cam1 to cam2 camera center diff (meters)
Min:    0.00308957
Median: 2.00129
Max:    4.00384

cam1 to cam2 pixel diff
Min:    0.0132675
Median: 0.306688
Max:    0.817671

cam2 to cam1 pixel diff
Min:    0.0134031
Median: 0.306656
Max:    0.827721

There is still something off (most obviously in --> cam1 to cam2 camera center diff), but ASP.mapproject manages to process the reduced csm camera, and the resulting deformation is relatively small (when compared to the starting point w/o tweaks).

Still, I wonder what is missing to fine-tune the result (if at all possible).
I placed the original and tweaked model_state, along with the mapprojected images at this link.
Thanks for any further suggestions!

[oleg-alexandrov]

I took a quick look. My best guess is that the distortion would need scaling too. The LRO NAC distortion formula is here: https://github.com/DOI-USGS/usgscsm/blob/main/src/Distortion.cpp#L290 In your case, you have "m_opticalDistCoeffs": 1.81e-05 You can try to see if multiplying or dividing this number by 10 (which looks like your scale factor) brings one closer or further from the expected behaviour. Maybe one needs to divide it / multiply it by 10 * 10 instead, as the formula seems to be from the code above: double den = 1 + dk1 * dy * dy; ux = dx; uy = dy / den; I am not sure, really (and too late in the day to do the math), but since you went this far, maybe playing with this value a bit will shed some light.

…

On Mon, Jun 3, 2024 at 4:10 PM Stefano Bertone ***@***.***> wrote: Your suggested "tweaks" seem to work (almost) perfectly, thanks @oleg-alexandrov <https://github.com/oleg-alexandrov>! Here below is the summary of a small test I did on LROC NAC image M129050120LE. Original .cub vs .csm (cam_test result) Image dimensions: 2532 52224 Number of samples used: 13598 cam1 to cam2 camera direction diff norm Min: 5.66395e-10 Median: 1.97569e-07 Max: 7.79053e-06 cam1 to cam2 camera center diff (meters) Min: 4.55051e-08 Median: 3.2549e-05 Max: 0.334139 cam1 to cam2 pixel diff Min: 2.11018e-05 Median: 0.00969859 Max: 0.390455 cam2 to cam1 pixel diff Min: 1.35774e-05 Median: 0.00977334 Max: 0.390295 Reduced .cub vs "tweaked" .csm (cam_test result) Image dimensions: 253 5222 Number of samples used: 159 cam1 to cam2 camera direction diff norm Min: 3.07541e-06 Median: 1.73445e-05 Max: 0.000145736 cam1 to cam2 camera center diff (meters) Min: 0.00308957 Median: 2.00129 Max: 4.00384 cam1 to cam2 pixel diff Min: 0.0132675 Median: 0.306688 Max: 0.817671 cam2 to cam1 pixel diff Min: 0.0134031 Median: 0.306656 Max: 0.827721 There is still something off (most obviously in --> cam1 to cam2 camera center diff), but ASP.mapproject manages to process the reduced csm camera, and the resulting deformation is relatively small (when compared to the starting point w/o tweaks). Still, I wonder what is missing to fine-tune the result (if at all possible). I placed the original and tweaked model_state, along with the mapprojected images at this link <https://drive.google.com/file/d/1e4I8YQn3zhD8vEIYIsEqvHnOZbV1arE4/view?usp=sharing> . Thanks for any further suggestions! — Reply to this email directly, view it on GitHub <#421 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKDU3HYJADQZOFEVAB5PBLZFTZVFAVCNFSM57BHAPS2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJUGYZDONJTGM4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[steo85it]

I tried to play around with that (and with most other values, a bit blindly ^^), and it seems to impact the cam to cam pixel diffs (multiplying by 100 seemed to work best, reducing the differences by 50%).
Happy to see that a factor = scale_factor**2 would make sense, thanks.

Still, I can't find a parameter that has an impact on the camera center diffs (well, timing does, but usually in a negative way...). Will keep checking.

[oleg-alexandrov]

This is a tricky one. One can spend many days going through the fine implementation details for CSM, and some things I think are ad hoc and I could not figure out, like half pixel shifts. That you get about 1-4 pixels of disagreement surely suggests it is something subtle. You can try tweaking one of these by +/- 0.5, but this would be shooting in the dark: m_detectorLineOrigin m_detectorSampleOrigin m_intTimeLines

…

On Mon, Jun 3, 2024 at 5:38 PM Stefano Bertone ***@***.***> wrote: I tried to play around with that (and with most other values, a bit blindly ^^), and it seems to impact the cam to cam pixel diffs (multiplying by 100 seemed to work best, reducing the differences by 50%). Happy to see that a factor = scale_factor**2 would make sense, thanks. Still, I can't find a parameter that has an impact on the camera center diffs (well, timing does, but usually in a negative way...). Will keep checking. — Reply to this email directly, view it on GitHub <#421 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKDU3CAFVACWWJISF5N4V3ZFUEADAVCNFSM57BHAPS2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJUGYZTKNBRGAZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[steo85it]

While what I wrote above is true about the results of ASP.cam_test (comparing the reduced ISIS camera to the tweaked CSM camera), applying the factor (scale_factor**2) to m_opticalDistCoeffs (on top of all other modifications) removed the residual distortion. The new mapprojected image overlaps very well with the original csm camera (mapprojected using the same downsampling factor in the ASP.mapproject call)!

I wonder if the smal offset that I see between csm and isis mapprojected cameras can then be due to the mapprojection rather than to the downsampling of the csm camera?

In any case this looks great, and it's easy to automatize ... thanks for all the suggestions!

[oleg-alexandrov]

Glad there is an improvement.

I wonder if the smal offset that I see between csm and isis mapprojected cameras can then be due to the mapprojection rather than to the downsampling of the csm camera?

Not sure.

Your experiments show that a utility to scale a CM camera would be tricky. Particularly, code will be needed for every supported lens distortion model. For LRO NAC it was actually easy, as there was a single value squared. Other models have polynomials, and each coefficient would need appropriate adjustments, and for some model (radtan) no adjustment would be needed even if there are polynomials because it already applies some internal scaling.

[oleg-alexandrov]

After looking into this more closely, scaling a CSM camera to a reduced resolution is harder than it first appears. Most of the model scales cleanly (image dimensions, detector origins, line timing), but lens distortion does not. It would require strategically inserted code and a separate adaptation per distortion model. Across the 10 distortion models we support, these fall into roughly four distinct behaviors: self-normalizing models that need no change (radtan); single-coefficient squared models (LRO NAC, KPLO ShadowCam, Dawn FC); general polynomial models where each coefficient scales by its own power and some also carry additive terms (radial, transverse, Kaguya LISM, CAHVOR); and the rational model (CaSSIS, which additionally has a hardcoded focal-plane bound).

This is a real maintenance burden going forward: every new distortion model added later would need its own correctly-derived scaling rule, and a wrong rule fails silently rather than erroring out. The feature also does not appear to be in high demand, and the per-image state tweaks discussed earlier in this thread remain a workable manual route for anyone who needs it now.

I opened this issue, but I would suggest it be closed without implementing. If there is real interest (e.g. from those working with LRO NAC), it can be studied and pursued by an interested party who needs it. Leaving it open for now so that discussion can continue.

(Verbiage by bot, but edited and approved.)