Status of JUNOSW + Opticks : GPU ray trace accelerated optical photon simulation

Status of JUNOSW + Opticks :
GPU ray trace accelerated optical photon simulation

Open source, github.com:simoncblyth/opticks

Simon C Blyth, IHEP, CAS — JUNO Collaboration Meeting, Simulation AFG — 27 July 2025

Outline

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow
OJ : Opticks+JUNOSW Automated Gitlab-CI/CD Releases
Chi2 Comparison of photon histories between A:Opticks and B:Geant4
Common Input Photons in Opticks and Geant4 simulations
Input Photon Check Results
- 1M Input Photons targetting NNVT:0:1000 Chi2 History Comparison
- 1M Input Photons targetting Hama:0:1000 Chi2 History Comparison
- 1M Input Photons targetting Hama:0:0 Chi2 History Comparison
- 100k Input Photons targetting PMT_3inch:0:0 Chi2 History Comparison
- 1M Random Spherical Photons from CD center : History Chi2
- 100k Input Photons targetting PMT_20inch_veto:0:1000 History Chi2
- 100k Input Photons CircleXZ from WP top : Chi2 History Comparison
- 1M Input Photons shot up Chimney : TOTALLY DEVIANT : DIFF GEOMETRY
- TargetLS/LowerChimney/UpperChimney issues
- Summary of Input Photon Checks : Hit Count and Chi2
Recently Fixed Issues : Many Thanks to Dingyong, Haosen, Zhenning
Missing Features and Known Issues
OJ : Minimal Muon Script : ~/j/muon/muon_min.sh
muon_min.sh log : 121M photon kernel launch ~18 sec
Summary and Links, NEXT Development Steps, User Steps
Extra : Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7/8

Geant4 + Opticks + NVIDIA OptiX : Hybrid Workflow

https://github.com/simoncblyth/opticks

Opticks API : split according to dependency -- Optical photons are GPU "resident", only hits need to be copied to CPU memory

OJ : Opticks+JUNOSW Automated Gitlab-CI/CD Releases

Use Opticks+JUNOSW by sourcing envset.sh, eg:

source /cvmfs/opticks.ihep.ac.cn/oj/releases/J25.4.0_Opticks-v0.5.1/el9_amd64_gcc11/2025_07_17/envset.sh

Every day + when JUNOSW/Simulation changes => automated OJ build + test + release
For reproducibility, use dated reference releases eg 2025_07_17
tut_detsim.py --opticks-mode 1 is default (optical simulation on GPU)

A[blyth@localhost el9_amd64_gcc11]$ pwd
/cvmfs/opticks.ihep.ac.cn/oj/releases/J25.4.0_Opticks-v0.5.1/el9_amd64_gcc11

A[blyth@localhost el9_amd64_gcc11]$ ls -lt
1 lrwxrwxrwx. 1 cvmfs cvmfs   3 Jul 20 17:42 Latest -> Sun
1 drwxr-xr-x. 7 cvmfs cvmfs 181 Jul 20 17:36 Sun
1 drwxr-xr-x. 7 cvmfs cvmfs 181 Jul 19 17:33 Sat
1 drwxr-xr-x. 7 cvmfs cvmfs 181 Jul 18 17:33 Fri
1 drwxr-xr-x. 7 cvmfs cvmfs 181 Jul 17 17:36 Thu
1 lrwxrwxrwx. 1 cvmfs cvmfs  10 Jul 17 16:32 LastRef -> 2025_07_17
1 drwxr-xr-x. 7 cvmfs cvmfs 181 Jul 17 15:39 2025_07_17

Chi2 Comparison of photon histories between A:Opticks and B:Geant4

enum
{
    CERENKOV           = 0x1 <<  0,    // CK
    SCINTILLATION      = 0x1 <<  1,    // SI
    TORCH              = 0x1 <<  2,    // TO
    BULK_ABSORB        = 0x1 <<  3,    // AB
    BULK_REEMIT        = 0x1 <<  4,    // RE
    BULK_SCATTER       = 0x1 <<  5,    // SC
    SURFACE_DETECT     = 0x1 <<  6,    // SD
    SURFACE_ABSORB     = 0x1 <<  7,    // SA
    SURFACE_DREFLECT   = 0x1 <<  8,    // DR
    SURFACE_SREFLECT   = 0x1 <<  9,    // SR
    BOUNDARY_REFLECT   = 0x1 << 10,    // BR
    BOUNDARY_TRANSMIT  = 0x1 << 11,    // BT
    NAN_ABORT          = 0x1 << 12,    // NA
    EFFICIENCY_COLLECT = 0x1 << 13,    // EC
    EFFICIENCY_CULL    = 0x1 << 14,    // EX
    ...
};

Label every photon in both simulations, compare labels

Photon step point classified, 4 bits (16) => sseq.h
32*4 bits per photon => seq.npy array
Each photon has history string eg "SI BT BT BT BT EC"

sysrap/tests/sseq_index_test.sh

index histories with counts from A and B
form Chi2 comparing counts
large Chi2 contrib histories point to bugs

Chi2 sensitive to anything that changes photon histories

Opticks uses Gensteps collected from Geant4

=> both simulations have exact same photon counts
BULK_REEMIT is NOT considered to form new photon

More of one history implies less of other(s), A/BZERO points to bugs

JUNOSW/U4RecorderAnaMgr : passes G4Step to opticks/u4/U4Recorder recording into Opticks SEvt

Common Input Photons in Opticks and Geant4 simulations

Common input photons, two optical simulations

Control frame of input photons with envvar ====>
- target different geometry
- sysrap/stree.h stree::get_frame

ipc(){ source ~/j/InputPhotonsCheck/InputPhotonsCheck.sh $* ; }

tut_detsim.py
   --opticks-mode 3     ## both Geant4 and Opticks simulations
   --opticks-anamgr     ## record Geant4 photon steps points
   --no-anamgr-normal   ## still needed with opticks-mode 3
   ...
   opticks              ## use GtOpticksTool : Input Photons

https://code.ihep.ac.cn/blyth/j/-/blob/main/InputPhotonsCheck/InputPhotonsCheck.sh

Input Photon Check Results

Following pages cover:

1M Input Photons targetting NNVT:0:1000 Chi2 History Comparison
1M Input Photons targetting Hama:0:1000 Chi2 History Comparison
1M Input Photons targetting Hama:0:0 Chi2 History Comparison
100k Input Photons targetting PMT_3inch:0:0 Chi2 History Comparison
1M Random Spherical Photons from CD center : History Chi2
100k Input Photons targetting PMT_20inch_veto:0:1000 History Chi2
100k Input Photons CircleXZ from WP top : Chi2 History Comparison
- refining intersect precision, fixes deviation
1M Input Photons shot up Chimney : TOTALLY DEVIANT : DIFF GEOMETRY
TargetLS/LowerChimney/UpperChimney issues
Summary of Input Photon Checks : Hit Count and Chi2

1M Input Photons targetting NNVT:0:1000 Chi2 History Comparison

TEST=NNVT ipc run
SABHit  a_nhit : (  362580 +-  602 )  b_nhit:(   363183 +-  602 )  a/b: (  0.998 +-  0.003 )  b/a: (  1.002 +-  0.003)

TEST=NNVT ipc chi2
[sseq_index_ab::desc                           CHI2 sum  1393.8496 ndf 954.0000 sum/ndf 1.4611 
TO BT BT BT BT EC                                         :  312169 312612 :     0.3141 : Y :       1      6 :
TO BT BT BT BT SA                                         :  293808 294880 :     1.9521 : Y :       2      2 :
TO BT BT BT BT BT SR SA                                   :   43995  43757 :     0.6455 : Y :  103587 103712 :
TO BT BT BT BT BT SA                                      :   32244  32300 :     0.0486 : Y :   84121  84359 :
TO BT BT BT BT BR BT BT BT BT BT BT AB                    :   17363  16152 :    43.7572 : Y :    6638  39864 : DEVIANT
TO BT BT BT BT BT SR BR SR SA                             :   10870  10812 :     0.1552 : Y :  210140 210140 :
TO BT BT BT BT BR BT BT BT BT BT BT BT BT SA              :    8882   8551 :     6.2847 : Y :   73002  90912 :
TO BT BT BT BT BT SR BR SA                                :    8881   8635 :     3.4549 : Y :  202016 202009 :
TO BT BT AB                                               :    8706   8767 :     0.2130 : Y :      47    147 :
TO BT BT BT BT BR BT BT BT BT BT BT BT BT EC              :    8391   8289 :     0.6237 : Y :   79688  79729 :
TO BT BT BT BT BR BT BT BT BT AB                          :    8320   8001 :     6.2350 : Y :    1072    350 :
TO BT BT BT BT EX                                         :    7447   7330 :     0.9264 : Y :       0     90 :
TO BT BT BT BT BR BT BT BT BT BT BT BT BT BT BT BT BT EC  :    5996   5992 :     0.0013 : Y :  115560 115545 :
TO BT BT BT BT BT SR SR SA                                :    5973   5904 :     0.4009 : Y :  103570 103605 :
]sseq_index_ab::desc

Chi2 table : "seqhis" label, counts in A and B, Chi2 contrib, Y/N Chi2 inclusion, First index in A and B

NNVT:0:1000 : Consistent Hits, Chi2 OK, some deviants

Opticks+Geant4 SEvt saved as A000 B000 folders of NumPy arrays

TEST=NNVT ipc env_pdb  ## load SEvt AFOLD and BFOLD into ipython

 In [1]: a
 Out[1]: SEvt symbol a pid -1 opt  off [0. 0. 0.] a.f.base
 /tmp/blyth/opticks/GEOM/J25_4_0_opticks_Debug/InputPhotonsCheck/ALL1_DebugPhilox_3_NNVT:0:1000_RainXZ_Z230_1M_f8/A000

 In [2]: b
 Out[2]: SEvt symbol b pid -1 opt  off [0. 0. 0.] b.f.base
 /tmp/blyth/opticks/GEOM/J25_4_0_opticks_Debug/InputPhotonsCheck/ALL1_DebugPhilox_3_NNVT:0:1000_RainXZ_Z230_1M_f8/B000

 In [3]: a.f                                                               In [4]: b.f
 Out[3]:                                                                   Out[4]:
 CMDLINE:/home/blyth/j/InputPhotonsCheck/InputPhotonsCheck.py              CMDLINE:/home/blyth/j/InputPhotonsCheck/InputPhotonsCheck.py
 a.base:/tmp/blyth/opticks/GEOM/J25_4_0_opticks_Debug/InputPhotonsCheck    b.base:/tmp/blyth/opticks/GEOM/J25_4_0_opticks_Debug/InputPhotonsCheck
           /ALL1_DebugPhilox_3_NNVT:0:1000_RainXZ_Z230_1M_f8/A000                   /ALL1_DebugPhilox_3_NNVT:0:1000_RainXZ_Z230_1M_f8/B000

   : a.genstep       :            (1, 6, 4) : 3 days, 18:30:54.268232        : b.genstep           :            (1, 6, 4) : 3 days, 18:31:02.710375
   : a.photon        :      (1000000, 4, 4) : 3 days, 18:30:54.266232        : b.photon            :      (1000000, 4, 4) : 3 days, 18:31:02.708375
   : a.record        :  (1000000, 32, 4, 4) : 3 days, 18:30:54.130232        : b.record            :  (1000000, 32, 4, 4) : 3 days, 18:31:02.598375
   : a.seq           :      (1000000, 2, 2) : 3 days, 18:30:53.183233        : b.seq               :      (1000000, 2, 2) : 3 days, 18:31:01.623376
   : a.hit           :       (362580, 4, 4) : 3 days, 18:30:53.169233        : b.hit               :       (363183, 4, 4) : 3 days, 18:31:01.610377
   : a.inphoton      :      (1000000, 4, 4) : 3 days, 18:30:53.160233        : b.inphoton          :      (1000000, 4, 4) : 3 days, 18:31:01.599376
   : a.seqnib        :           (1000000,) : 3 days, 18:30:53.139233        : b.seqnib            :           (1000000,) : 3 days, 18:31:01.579376
   : a.seqnib_table  :              (33, 1) : 3 days, 18:30:53.137233        : b.seqnib_table      :              (33, 1) : 3 days, 18:31:01.578377
   : a.NPFold_index  :                 (9,) : 3 days, 18:30:53.137233        : b.NPFold_index      :                (11,) : 3 days, 18:31:01.578377
   : a.sframe        :            (4, 4, 4) : 3 days, 18:30:53.137233        : b.sframe            :            (4, 4, 4) : 3 days, 18:31:01.578377

Record (1000000, 32, 4, 4) : 1M photons, 32 step points of (4,4) param (pos,time,dir,wl,pol,id,...)

interpolation between step point positions gives position at any time
OpenGL geometry shader : animate photon positions with time as input

GEOM_J25_4_0_opticks_Debug_cxr_min_NNVT_20250704_190059.png

EVT=NNVT cxr_min.sh # geometry OFF, NNVT:0:1000

Snapshot from OpenGL photon record animation

GEOM_J25_4_0_opticks_Debug_cxr_min_NNVT_20250704_190101.png

EVT=NNVT cxr_min.sh # geometry OFF, NNVT:0:1000

Snapshot from OpenGL photon record animation, little later

1M Input Photons targetting Hama:0:1000 Chi2 History Comparison

TEST=Hama2 ipc run
SABHit  a_nhit : (  300462 +-  548 )  b_nhit:(   300163 +-  547 )  a/b: (  1.001 +-  0.004 )  b/a: (  0.999 +-  0.004)

TEST=Hama2 ipc run
[sseq_index_ab::desc                              [CHI2 sum   431.3700 ndf 439.0000 sum/ndf     0.9826 ] 
TO BT BT BT BT SA                                 :  375291 375423 :     0.0232 : Y :       2      2 :
TO BT BT BT BT EC                                 :  282246 281762 :     0.4153 : Y :       5      0 :
TO BT BT BT BT BT SA                              :  124915 125119 :     0.1664 : Y :   88947  88854 :
TO BT BT BT BT BT SR SA                           :   38523  38484 :     0.0198 : Y :  108919 108784 :
TO BT BT BT BT EX                                 :   24764  24932 :     0.5679 : Y :       0      5 :
TO BT BT BT BT BT SR SR SA                        :   20225  20052 :     0.7431 : Y :  108778 108774 :
TO BT BT AB                                       :    8758   8740 :     0.0185 : Y :      47     31 :
TO BT BT BT BT BT SR SR SR SA                     :    6015   5935 :     0.5356 : Y :  147290 147252 :
TO BT BT BT BT BR BT BT BT BT BT BT AB            :    4491   4337 :     2.6865 : Y :   38993  64544 :
TO BT BT BT BT AB                                 :    3410   3393 :     0.0425 : Y :     199     26 :
TO BT BT BT BT BT SR BR SA                        :    3137   3065 :     0.8359 : Y :  335672 335676 :
TO BT BT BT BT BR BT BT BT BT BT SA               :    3074   3099 :     0.1012 : Y :   10173  10179 :
TO BT BT BT BT BR BT BT BT BT AB                  :    2954   2841 :     2.2035 : Y :    2399   1014 :
TO BT BT BT BT BR BT BT BT BT BT BT BT BT EC      :    2618   2558 :     0.6955 : Y :   81470  81407 :

Hama:0:1000 : Consistent Hits, Chi2 OK

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Hama2_ipc_20250707_165110.png

EVT=Hama2 cxr_min.sh # geometry OFF, Hama:0:1000

Snapshot from OpenGL photon record animation, wider view

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Hama2_ipc_20250707_165111.png

EVT=Hama2 cxr_min.sh # geometry OFF, Hama:0:1000

Snapshot from OpenGL photon record animation, wider view

1M Input Photons targetting Hama:0:0 Chi2 History Comparison

TEST=Hama ipc run
SABHit  a_nhit : (  331799 +-  576 )  b_nhit:(   331701 +-  575 )  a/b: (  1.000 +-  0.003 )  b/a: (  1.000 +-  0.003)

TEST=Hama ipc chi2
[sseq_index_ab::desc                             [CHI2 sum  1107.2895 ndf 467.0000 sum/ndf     2.3711 ]
TO BT BT BT BT SA                         :  336783 336911 :     0.0243 : Y :       2     12 :
TO BT BT BT BT EC                         :  317582 317384 :     0.0617 : Y :       5      1 :
TO BT BT BT BT BT SA                      :  124915 125038 :     0.0605 : Y :   88947  88032 :
TO BT BT BT BT BT SR SA                   :   38523  38437 :     0.0961 : Y :  108919 108963 :
TO BT BT BT BT EX                         :   27933  28065 :     0.3112 : Y :       0      9 :
TO BT BT BT BT BT SR SR SA                :   20226  20230 :     0.0004 : Y :  108778 108786 :
TO BT BT BT BT BR BT BT BT BT BT SA       :   10668  10851 :     1.5563 : Y :   49243  49230 :
TO BT BT AB                               :    8758   8756 :     0.0002 : Y :      47      4 :
TO BT BT BT BT BT SR SR SR SA             :    6015   5933 :     0.5628 : Y :  147290 147241 :
TO BT BT BT BT AB                         :    3410   3293 :     2.0422 : Y :     199      6 :
TO BT BT BT BT BT SR BR SA                :    3136   3084 :     0.4347 : Y :  335672 335683 :
TO BT BT BT BT BR BT BT BT BT BT BT AB    :    2958   2465 :    44.8182 : Y :   64577  64662 : DEVIANT
TO BT BT BT BT BT SR SR SR BR SA          :    2391   2319 :     1.1006 : Y :  147741 147420 :
TO BT BT BR BT BT BT SA                   :    1961   2028 :     1.1253 : Y :   27950  38276 :
TO BT BT BT BT BR BT BT BT BT AB          :    1844   1832 :     0.0392 : Y :    2209   3824 :
TO BT BT BT BT BR BT BT BT BT SA          :    1778   1799 :     0.1233 : Y :      31      8 :
TO BT BT BT BR BT BT BT BT SA             :    1690   1635 :     0.9098 : Y :   42010  42229 :

Hama:0:0 : Consistent Hits, Poor Chi2 (Chimney?)

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Hama_20250704_215525.png

EVT=Hama ssst.sh # Hama:0:0 (LowerChimney to right)

ssst.sh # aka SGLFW_SOPTIX_Scene_test.sh: OpenGL photon record animation and triangulated wireframe geometry

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Hama_20250704_215528.png

EVT=Hama ssst.sh # geometry OFF, Hama:0:0 (LowerChimney to right)

ssst.sh # aka SGLFW_SOPTIX_Scene_test.sh: OpenGL photon record animation snapshot

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Hama_20250704_215530.png

EVT=Hama ssst.sh # geometry OFF, Hama:0:0 (LowerChimney to right)

PMT bathed in photons reflected from Chimney: likely source of discrepancy

100k Input Photons targetting PMT_3inch:0:0 Chi2 History Comparison

TEST=SPMT ipc run
SABHit  a_nhit : (   21893 +-  147 )  b_nhit:(    21919 +-  148 )  a/b: (  0.999 +-  0.013 )  b/a: (  1.001 +-  0.014)

TEST=SPMT ipc chi2
[sseq_index_ab::desc                                [CHI2 sum  2.6529 ndf  6.0000 sum/ndf  0.4422 ]
    TO BT BT EX                                  :   77287  77187 :     0.0647 : Y :       0      0 :
    TO BT BT EC                                  :   21819  21843 :     0.0132 : Y :       3      2 :
    TO AB                                        :     456    486 :     0.9554 : Y :      25     76 :
    TO BT BT AB                                  :     119    110 :     0.3537 : Y :    2121    261 :
    TO BT AB                                     :      36     41 :     0.3247 : Y :    3833    420 :
    TO BT BR BT BT BT AB                         :      30     38 :     0.9412 : Y :    2867   6191 :
    TO BT BR BT BT BT SC BT BT BT BT BT BT EC    :      16     14 :     0.0000 : N :    5229   7937 : C2EXC
    TO SC BT BT BT BT EC                         :      13     14 :     0.0000 : N :    8966  28955 : C2EXC
    TO SC BT BT BT BT SA                         :       8     12 :     0.0000 : N :    6692   9608 : C2EXC
    TO BT BR BT BT BT SC BT BT BT BT BT BT SA    :       9     11 :     0.0000 : N :   10946  34936 : C2EXC
    TO BT BR BT BT BT BT BT BT BT BT BT EC       :      10     11 :     0.0000 : N :   33607   7618 : C2EXC
    TO BT BR BT BT BT BT BT BT BT BT BT SA       :      10     10 :     0.0000 : N :    8104  22053 : C2EXC
    TO BT BR BT BT BT SC AB                      :       9      8 :     0.0000 : N :    5616   6787 : C2EXC
    TO BT BR BT AB                               :       6      6 :     0.0000 : N :    5173  19516 : C2EXC

PMT_3inch:0:0 : Consistent Hits, Chi2 OK

GEOM_J25_4_0_opticks_Debug_cxr_min_SPMT_cxs_20250722_155442.png

EVT=SPMT_cxs MOI=PMT_3inch:0:0 T0=0 T1=10 TN=1000 cxr_min.sh # geometry OFF

EVT select event

MOI select viewpoint

T0, T1 animation time range (ns)

TN animation steps

1M Random Spherical Photons from CD center : History Chi2

TEST=CD ipc run
SABHit  a_nhit : (  217101 +-  465 )  b_nhit:(   218082 +-  466 )  a/b: (  0.996 +-  0.004 )  b/a: (  1.005 +-  0.004)

TEST=CD ipc chi2
[sseq_index_ab::desc                       [ CHI2  sum  1838.7167 ndf 1652.0000 sum/ndf 1.1130 ]
TO AB                                                   :  111833 111239 :     1.5817 : Y :       0      9 :
TO BT BT BT BT BT BT EC                                 :   90826  91325 :     1.3670 : Y :      14     25 :
TO BT BT BT BT BT BT SA                                 :   87693  87782 :     0.0451 : Y :      11      2 :
TO SC AB                                                :   41715  41226 :     2.8830 : Y :      16     15 :
TO SC BT BT BT BT BT BT EC                              :   40527  40512 :     0.0028 : Y :       5     29 :
TO SC BT BT BT BT BT BT SA                              :   40240  39939 :     1.1300 : Y :      29     34 :
TO BT BT SA                                             :   23450  23550 :     0.2128 : Y :      56     21 :
TO SC SC BT BT BT BT BT BT EC                           :   15348  15450 :     0.3378 : Y :      37     14 :
TO SC SC BT BT BT BT BT BT SA                           :   15429  15397 :     0.0332 : Y :       3     13 :
TO SC SC AB                                             :   14643  14545 :     0.3290 : Y :      51    219 :
...
TO SC BT BT BT SA                                       :    4306   4471 :     3.1019 : Y :     190     82 :
TO BT BT BT BT BT BT BT EC                              :    4090   4050 :     0.1966 : Y :     675   1471 :
TO SC BT AB                                             :    3147   3440 :    13.0331 : Y :     508    763 : DEVIANT
TO BT BT BT BT AB                                       :    3259   3290 :     0.1467 : Y :     267    103 :
TO SC SC BT BT AB                                       :    3082   2941 :     3.3008 : Y :      36     35 :

sTarget:0:-2 : Consistent Hits, Chi2 OK

1M Random Spherical Photons from CD center : History Chi2 (continued)

AZERO: histories only present in simulation B (Geant4)
BZERO: histories only present in simulation A (Opticks)

Can be sign of geometry bugs : but low stats sprinkle expected

TEST=CD ipc chi2
[sseq_index_ab::desc u_size 129497 NUM_MAX 40 num 129497 opt AZERO mode 1
TO BT BT BT BT BR SR SR SR SR SR BT BT BT BT BT BT SA   :      -1     27 :     0.0000 : N :      -1  55880 : AZERO C2EXC
TO BT BT BT BT BR SR SR SR SR SR SR BR BR BR BR EX      :      -1     24 :     0.0000 : N :      -1  11230 : AZERO C2EXC
TO BT BT BT BT BR SR SR SR SR SR BT BT BT BT BT BT EC   :      -1     14 :     0.0000 : N :      -1  17109 : AZERO C2EXC
TO BT BT SC BT BT SC BT AB                              :      -1     12 :     0.0000 : N :      -1  23403 : AZERO C2EXC
TO BT BT BT BT BR SR SR SR SR SR SR BR BT BT BT BT AB   :      -1     11 :     0.0000 : N :      -1  16219 : AZERO C2EXC
TO BT BT BT BT BR SR SR SR SR SR SR BR BR BR BR EC      :      -1     11 :     0.0000 : N :      -1  53909 : AZERO C2EXC
]sseq_index_ab::desc

[sseq_index_ab::desc u_size 129497 NUM_MAX 40 num 129497 opt BZERO mode 2
TO BT BT BT BT BR SR SA                                 :     183     -1 :     0.0000 : N :    4232     -1 : BZERO C2EXC
TO BT BT BT BT BR SA                                    :      42     -1 :     0.0000 : N :   18984     -1 : BZERO C2EXC
TO SC BT BT BT BT BR SR SA                              :      25     -1 :     0.0000 : N :    7924     -1 : BZERO C2EXC
TO BT BT BT BT BT BT BR BT BT BT BT BT BT BT BT BT EC   :      19     -1 :     0.0000 : N :   20777     -1 : BZERO C2EXC
TO BT BT BT BT BR BR BR BT AB                           :      12     -1 :     0.0000 : N :  301580     -1 : BZERO C2EXC
...
]sseq_index_ab::desc

GEOM_J25_4_0_opticks_Debug_cxr_min_CD_ipc_20250706_113711.png

EVT=CD_ipc MOI=sTarget:0:-2 cxr_min.sh

ray trace render and OpenGL photon record animation

1M RandomSpherical photons from CD center

GEOM_J25_4_0_opticks_Debug_cxr_min_CD_ipc_20250706_113836.png

EVT=CD_ipc MOI=Hama:0:0 SDR=rec_line_strip cxr_min.sh: SDR : config record animation OpenGL shader

100k Input Photons targetting PMT_20inch_veto:0:1000 History Chi2

TEST=WP_PMT ipc run
SABHit  a_nhit : (   29562 +-  171 )  b_nhit:(    29735 +-  172 )  a/b: (  0.994 +-  0.012 )  b/a: (  1.006 +-  0.012)

TEST=WP_PMT ipc chi2
[sseq_index_ab::desc                                  [CHI2 sum 54.3135 ndf 59.0000 sum/ndf     0.9206 ]
TO BT BT BT BT SA                                            :   30563  30508 :     0.0495 : Y :       2      0 :
TO BT BT BT BT EC                                            :   29246  29398 :     0.3940 : Y :       1      5 :
TO BT BT BT BT BR BT BT BT BT SA                             :    9661   9850 :     1.8308 : Y :       6     45 :
TO BT BT BT BT BT SR SA                                      :    7688   7783 :     0.5833 : Y :   10622  10391 :
TO BT BT BT BT BT SR BT BT BT BT BT SA                       :    6782   6904 :     1.0875 : Y :   10830  10837 :
TO BT BT BT BT BT SA                                         :    2060   1940 :     3.6000 : Y :    8707   8511 :
TO BT BT BT BT EX                                            :    1982   1863 :     3.6830 : Y :       0      2 :
TO BT BT BT BT BT SR BR SR SA                                :    1739   1735 :     0.0046 : Y :   20717  20579 :
TO BT BT BT BT BT SR BR SR BT BT BT BT BT SA                 :    1552   1514 :     0.4710 : Y :   20911  20910 :
TO BT BT BT BT BT SR BR SA                                   :     911    876 :     0.6855 : Y :   10942  10869 :
TO BT BT BR BT BT SA                                         :     677    678 :     0.0007 : Y :      16     23 :
TO BT BT BT BR BT BT BT SA                                   :     594    595 :     0.0008 : Y :       4      4 :
TO BT BR BT SA                                               :     511    496 :     0.2234 : Y :      23     34 :
TO BT BT BT BT BT SR BR SR BR SR SA                          :     507    483 :     0.5818 : Y :   21826  21843 :
]sseq_index_ab::desc

PMT_20inch_veto:0:1000 : Hits Consistent, Chi2 OK

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_ipc_20250706_124505.png

EVT=wp_pmt_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh: line of 100k input photons incident on WP_PMT

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_ipc_20250706_124506.png

EVT=wp_pmt_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh: line of 100k input photons incident on WP_PMT

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_ipc_20250706_124509.png

EVT=wp_pmt_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh: line of 100k input photons incident on WP_PMT

XZ Semi-Circle of input photons targetting WP_PMT (clearance 0.05 mm)

TEST=WP_PMT_SEMI ipc env_sab
SABHit  a_nhit : (   26172 +-  161 )  b_nhit:(    26318 +-  162 )  a/b: (  0.994 +-  0.012 )  b/a: (  1.006 +-  0.012)

TEST=WP_PMT_SEMI ipc env_chi2
[sseq_index_ab::desc  [CHI2   sum    58.2633 ndf 64.0000 sum/ndf     0.9104  ]
TO BT BT BT BT SA                                           :   26918  26986 :     0.0858 : Y :       2      2 :
TO BT BT BT BT EC                                           :   25676  25837 :     0.5032 : Y :      15     27 :
TO BT BT BT BT BT SR SA                                     :   10281  10314 :     0.0529 : Y :      57    205 :
TO BT BT BT BT BR BT BT BT BT SA                            :    9972  10021 :     0.1201 : Y :    1668   1665 :
TO BT BT BT BT BT SR BT BT BT BT BT SA                      :    8187   8225 :     0.0880 : Y :    7251   7262 :
TO BT BT BT BT BT SA                                        :    2686   2623 :     0.7476 : Y :      32     19 :
TO BT BT BT BT BT SR BR SR SA                               :    2418   2400 :     0.0672 : Y :    7328   7099 :
TO BT BT BT BT BT SR BR SR BT BT BT BT BT SA                :    2047   1929 :     3.5020 : Y :    9325   9317 :
TO BT BT BT BT EX                                           :    1896   1862 :     0.3076 : Y :       7      3 :

                 Opticks (missing BT) histories near apex and equator 
[sseq_index_ab::desc BZERO
TO BT BT BT SA                                              :      45     -1 :     0.0000 : N :       3     -1 : BZERO C2EXC
TO BT BT BT EC                                              :      38     -1 :     0.0000 : N :   49757     -1 : BZERO C2EXC
TO BT BT BT BT SR SA                                        :      17     -1 :     0.0000 : N :   49855     -1 : BZERO C2EXC
TO BT BT BT BT BT SR SR SR AB                               :      17     -1 :     0.0000 : N :     706     -1 : BZERO C2EXC
TO BT BT BT BT SR BT BT BT BT BT SA                         :      15     -1 :     0.0000 : N :   49881     -1 : BZERO C2EXC
TO BT BT BT BR BT BT BT BT SA                               :      11     -1 :     0.0000 : N :   49853     -1 : BZERO C2EXC

Hits Consistent, Chi2 OK, BUT: Opticks misses BT intersects near apex and equator

XZ Semi-Circle of input photons targetting WP_PMT (clearance 0.10 mm)

TEST=WP_PMT_SEMI OLDNAME=1 ipc env_sab
SABHit  a_nhit : (   26175 +-  161 )  b_nhit:(    26314 +-  162 )  a/b: (  0.995 +-  0.012 )  b/a: (  1.005 +-  0.012)

TEST=WP_PMT_SEMI ipc env_chi2
[sseq_index_ab::desc  [CHI2   sum    60.5086 ndf 65.0000 sum/ndf     0.9309  ]
TO BT BT BT BT SA                              :   26964  26997 :     0.0202 : Y :       2      2 :
TO BT BT BT BT EC                              :   25721  25839 :     0.2701 : Y :      12     27 :
TO BT BT BT BT BT SR SA                        :   10292  10313 :     0.0214 : Y :      23    205 :
TO BT BT BT BT BR BT BT BT BT SA               :    9983  10018 :     0.0612 : Y :    1668   1664 :
TO BT BT BT BT BT SR BT BT BT BT BT SA         :    8210   8225 :     0.0137 : Y :    7311   7262 :
TO BT BT BT BT BT SA                           :    2689   2612 :     1.1185 : Y :       6     19 :
TO BT BT BT BT BT SR BR SR SA                  :    2420   2400 :     0.0830 : Y :    7328   7099 :
TO BT BT BT BT BT SR BR SR BT BT BT BT BT SA   :    2052   1929 :     3.8003 : Y :    9325   9317 :
TO BT BT BT BT EX                              :    1900   1853 :     0.5886 : Y :       1      3 :
TO BT BT BT BT BT SR BR SA                     :     824    821 :     0.0055 : Y :    3687   3693 :
TO BT BT BT BT BT SR BR SR BR SR SA            :     744    712 :     0.7033 : Y :    8209   8570 :
TO BT BT BT BT BT SR SR SA                     :     631    661 :     0.6966 : Y :      83    11

[sseq_index_ab::desc BZERO
TO BT BT BT BT BT SR SR SR AB                  :      17     -1 :     0.0000 : N :     489     -1 : BZERO C2EXC
]sseq_index_ab::desc

Hits Consistent, Chi2 OK, Missed intersects mostly avoided

export R12860OnlyFrontMaskManager__MAGIC_virtual_thickness_MM=0.10 # 0.05 def.

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_semi_ipc_20250721_111242.png

EVT=wp_pmt_semi_ipc MOI=PMT_20inch_veto:0:1000 cxr_min.sh: semi-circle of 100k input photons targetting WP_PMT

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_semi_ipc_20250721_111433.png

EVT=wp_pmt_semi_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh: semi-circle of 100k input photons targetting WP_PMT

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_semi_ipc_20250709_212248.png

EVT=wp_pmt_semi_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh # geometry OFF

export AFOLD_RECORD_SLICE="TO BT BT BT BT SA" # select one simple history

snapshot from photon record animation

virtual wrapper "touches" PMT at apex and equator

increase clearance 0.05 => 0.10 mm avoids missed intersects

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_ipc_20250701_200328.png

EVT=wp_pmt_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh: Speeding photons => investigation => fix of WP geometry overlap

far clipping used to remove other PMTs

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_wp_pmt_ipc_20250701_200337.png

EVT=wp_pmt_ipc MOI=PMT_20inch_veto:0:1000 ssst.sh: Speeding photons => investigation => fix of WP geometry overlap

GEOM_J25_4_0_opticks_Debug_cxr_min_poolcover_ipc_20250706_145250.png

EVT=poolcover_ipc cxr_min.sh # InputPhotons from XZ Circle near top of pool

"poolcover" test motivated by the speeding photons, reveals:

overlap in hierarchical WP geometry (cause of speeding) - FIXED

loss of precision for long distance intersects onto WP_PMT - FIXED

causing history Chi2 deviation from double-intersect

100k Input Photons CircleXZ from WP top : Chi2 History Comparison

500 mm radius Circle in XZ plane, center 3345.569,20623.73,21000 : near top of WaterPool

TEST=POOLCOVER ipc run
SABHit  a_nhit : (     140 +-   11 )  b_nhit:(      124 +-   11 )  a/b: (  1.129 +-  0.197 )  b/a: (  0.886 +-  0.154)

TEST=POOLCOVER ipc chi2
[sseq_index_ab::desc  [CHI2   sum    49.8171 ndf  9.0000 sum/ndf     5.5352  ]
TO SA                                      :   85301  85454 :     0.1371 : Y :       0      0 :
TO AB                                      :   13861  13716 :     0.7624 : Y :       1      1 :
TO BT BT BT BT SA                          :     154    224 :    12.9630 : Y :   71432  71433 : DEVIANT
TO BT BT SA                                :      79    131 :    12.8762 : Y :   71342  71342 : DEVIANT
TO BT BT BT BT EC                          :      96    117 :     2.0704 : Y :   71553  71555 :
TO BT BT AB                                :      63    108 :    11.8421 : Y :   71358  71344 : DEVIANT
TO BT BT BT BT BT SA                       :      64     -1 :     0.0000 : N :   71455     -1 : BZERO C2EXC
TO BT BT BT SA                             :      59     -1 :     0.0000 : N :   71352     -1 : BZERO C2EXC
TO BT BT BT BT AB                          :      40     59 :     3.6465 : Y :   71440  71432 :
TO BT BT BR BT BT SA                       :      30     40 :     1.4286 : Y :   71537  71534 :
]sseq_index_ab::desc

Discrepancy between 3/4/5 x BT intersects, for photons travelling from top to bottom of pool and clipping WP_PMT.

WaterPool : Bad Chi2, low stat hits

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_poolcover_ipc_20250703_211459.png

EVT=poolcover_ipc MOI=PMT_20inch_veto:0:$(( 51006 - 50000 )) ssst.sh

export AFOLD_RECORD_SLICE="TO BT BT BT BT BT SA"

Snapshot of record animation for virtual wrapper clippers with 5xBT (1 extra)

100k Input Photons CircleXZ from WP top : "TO BT BT BT BT BT SA"

Opticks tendency for extra intersect with "long" rays ?
select photons with 5 BT in Opticks(OK), 4 BT in Geant4, examine step point position/time:
HMM: 2nd Opticks(float) intersect is very close to 1st Geant4(double) intersect ?

TEST=POOLCOVER OLDNAME=1 ipc env_sab
In [1]: w54 = ab.q_and("TO BT BT BT BT BT SA", "TO BT BT BT BT SA") ; w54
Out[1]: array([71464, 71485, 71663, 71678, ... ])

In [2]: t54 = np.c_[a.f.record[w54,:7,0],b.f.record[w54,:7,0]] ; t54   ## compare positions and times between A and B
Out[2]:
array([[[  3235.416,  20623.73 ,  20512.285,      0.1  ,   3235.416,  20623.73 ,  20512.285,      0.1  ],
        [ -1188.089,  20623.73 ,    926.728,     92.668,  -1188.132,  20623.73 ,    926.537,     92.669],
        [ -1188.132,  20623.73 ,    926.537,     92.669,  -1251.53 ,  20623.73 ,    645.835,     93.995],
        [ -1251.53 ,  20623.73 ,    645.835,     93.995,  -1529.175,  20623.73 ,   -583.467,     99.806],
        [ -1529.175,  20623.73 ,   -583.467,     99.806,  -1567.19 ,  20623.73 ,   -751.785,    100.601],
        [ -1567.193,  20623.73 ,   -751.799,    100.601,  -6287.166,  20623.73 , -21650.   ,    199.373],
        [ -6287.166,  20623.73 , -21649.998,    199.373,      0.   ,      0.   ,      0.   ,      0.   ]],

       [[  3236.06 ,  20623.73 ,  20512.14 ,      0.1  ,   3236.06 ,  20623.73 ,  20512.14 ,      0.1  ],
        [ -1153.963,  20623.73 ,    954.78 ,     92.507,  -1154.008,  20623.73 ,    954.579,     92.508],
        [ -1154.008,  20623.73 ,    954.579,     92.508,  -1232.439,  20623.73 ,    605.175,     94.159],
        [ -1232.439,  20623.73 ,    605.175,     94.159,  -1507.233,  20623.73 ,   -619.022,     99.943],
        [ -1507.233,  20623.73 ,   -619.022,     99.943,  -1547.467,  20623.73 ,   -798.263,    100.79 ],
        [ -1547.467,  20623.73 ,   -798.263,    100.79 ,  -6228.037,  20623.73 , -21650.   ,    199.313],
        [ -6228.038,  20623.736, -21650.002,    199.313,      0.   ,      0.   ,      0.   ,      0.   ]],

        ...

Opticks : 2nd intersect from < 1mm "refining" precision of 1st intersect from ~20,000 mm ?

100k Input Photons CircleXZ from WP top : distances between intersects

Distances between BT (BOUNDARY_TRANSMIT positions) : A[1] -> B[1] and A[2] -> B[1]

TEST=POOLCOVER OLDNAME=1 ipc env_sab
In [16]: np.c_[ np.linalg.norm(a.f.record[w54,1,0,:3] - b.f.record[w54,1,0,:3], axis=1),
                np.linalg.norm(a.f.record[w54,2,0,:3] - b.f.record[w54,1,0,:3], axis=1) ]*1e3
Out[16]:
array([[195.158,   0.061],
       [205.677,   0.061],
       [ 62.536,   0.061],
       [ 77.175,   0.   ],
       [133.497,   0.   ],
       [ 51.962,   0.   ],
       [136.316,   0.061],
       [ 55.134,   0.061],
       [ 90.386,   0.061],
       [135.656,   0.061],
        ...

1st Opticks(float) intersect from 20,000mm is of lesser precision (~0.2 mm)
2nd Opticks(float) intersect from < 1mm very close to Geant4(double) (~1e-4 mm) intersect

GEOM_BigWaterPool_SGLFW_SOPTIX_Scene_test_poolcover_cxs_20250718_214233.png

GEOM=BigWaterPool EVT=poolcover_cxs ssst.sh # JUNO pool dimensions

GEOM_BigWaterPool_SGLFW_SOPTIX_Scene_test_poolcover_cxs_20250718_214551.png

GEOM=BigWaterPool EVT=poolcover_cxs ssst.sh # JUNO pool dimensions: compare cylinder intersects from range of distances : 20,000 -> 100 mm

TODO: auto-test intersect precision of all prim. + some CSG combinations

GEOM_BigWaterPool_cxt_precision_simtrace_input_photon_poolcover_and_refine_20250718_212024.png

cxt_precision.sh : intersect error max(median) : ~1(0.05)mm at 20,000mm

GEOM_BigWaterPool_cxt_precision_simtrace_input_photon_poolcover_and_refine_20250718_211808.png

export OPTICKS_PROPAGATE_REFINE=1
export OPTICKS_PROPAGATE_REFINE_DISTANCE=5000 # mm

How precise ? What cost ?

REFINE : call optixTrace twice for "long" rays, improving precision

CSGOptiX/CSGOptiX7.cu (added REFINE in Opticks-v0.5.1)

template<bool refine> static __forceinline__ __device__
void trace( OptixTraversableHandle handle,
   float3 ray_origin, float3 ray_direction, float tmin, float tmax,
   quad2* prd, unsigned visibilityMask, float refine_distance)
{
    ...
    optixTrace( handle, ray_origin, ray_direction, tmin, tmax, ... );

    float t_approx = 0.99f*prd->distance() ;
    if( t_approx > refine_distance )   //  For long rays take a second closer shot
    {
        float3 closer_ray_origin = ray_origin + t_approx*ray_direction ;
        optixTrace( handle, closer_ray_origin, ray_direction, tmin, tmax, ... );
        prd->distance_add( t_approx );
    }
 }

2nd optixTrace : less precision loss by avoiding FLOPS between very different values

export OPTICKS_PROPAGATE_REFINE=1
export OPTICKS_PROPAGATE_REFINE_DISTANCE=5000 # mm

RAY TRACING GEMS II, Chapter 34, IMPROVING NUMERICAL PRECISION IN INTERSECTION PROGRAMS (Ingo Wald, NVIDIA)

https://link.springer.com/content/pdf/10.1007/978-1-4842-7185-8.pdf (~860 page PDF)

fp0 (sign, exponent, mantissa)
fp1 (sign, exponent, mantissa)

FLOP : 1st shift mantissa bits of smaller value to bring to same exponent as larger

extinction/vanishing : shift looses mantissa bits of smaller value

more difference of exponents => more shift => more precision loss
reduce precision loss by intersect from closer_ray_origin

100k Input Photons CircleXZ from WP top : OPTICKS_PROPAGATE_REFINE

Refined intersect positions entirely removes history discrepancy.

TEST=POOLCOVER_REFINE ipc sab
SABHit  a_nhit : (     137 +-   11 )  b_nhit:(      124 +-   11 )  a/b: (  1.105 +-  0.194 ) b/a: (  0.905 +-  0.159)

TEST=POOLCOVER_REFINE HISWID=50 NUM_MAX=20 ipc env_chi2
[sseq_index_ab::desc  [CHI2   sum     2.3424 ndf  9.0000 sum/ndf     0.2603  ]
TO SA                                              :   85301  85453 :     0.1353 : Y :       0      0 :
TO AB                                              :   13861  13716 :     0.7624 : Y :       1      3 :
TO BT BT BT BT SA                                  :     218    224 :     0.0814 : Y :   71432  71433 :
TO BT BT SA                                        :     128    132 :     0.0615 : Y :   71342  71341 :
TO BT BT BT BT EC                                  :     131    117 :     0.7903 : Y :   71553  71555 :
TO BT BT AB                                        :     101    108 :     0.2344 : Y :   71358  71344 :
TO BT BT BT BT AB                                  :      57     59 :     0.0345 : Y :   71440  71432 :
TO BT BT BR BT BT SA                               :      40     40 :     0.0000 : Y :   71537  71534 :
TO BT BT BT BT BR BT BT BT BT SA                   :      31     35 :     0.2424 : Y :   71583  71580 :
]sseq_index_ab::desc

FIXED : Discrepancy between 3/4/5 x BT intersects, for photons travelling from top to bottom of pool and clipping WP_PMT.

WaterPool : Chi2 OK with REFINE

1M Input Photons shot up Chimney : TOTALLY DEVIANT : DIFF GEOMETRY

[sseq_index_ab::desc  [CHI2   sum 32216.2266 ndf 58.0000 sum/ndf   555.4522  ]
TO BT SA                      :  885432     -1 :     0.0000 : N :       1     -1 : BZERO C2EXC
TO BT BT BT BT SA             :      -1 493337 :     0.0000 : N :      -1      0 : AZERO C2EXC
TO BT AB                      :   95945  41370 : 21690.4972 : Y :       0      7 : DEVIANT
TO BT BT BT SC SA             :      -1  65842 :     0.0000 : N :      -1      2 : AZERO C2EXC
TO BT SC SA                   :      -1  57241 :     0.0000 : N :      -1      1 : AZERO C2EXC
TO BT BT BT AB                :      -1  36192 :     0.0000 : N :      -1      9 : AZERO C2EXC
TO BT BT BR BT SA             :      -1  28419 :     0.0000 : N :      -1     43 : AZERO C2EXC
TO BT BT BT SC DR SA          :      -1  23389 :     0.0000 : N :      -1     14 : AZERO C2EXC
TO BT SC DR SA                :      -1  19439 :     0.0000 : N :      -1     61 : AZERO C2EXC
TO BT BT BT RE SA             :      -1  14856 :     0.0000 : N :      -1     76 : AZERO C2EXC
TO BT BT BT BR BT BT BT SA    :      -1  13635 :     0.0000 : N :      -1    359 : AZERO C2EXC
TO AB                         :   13250   6045 :  2690.4392 : Y :      25     50 : DEVIANT
TO BT RE SA                   :      -1  13014 :     0.0000 : N :      -1     15 : AZERO C2EXC
TO BT BT BT SC DR DR SA       :      -1   8319 :     0.0000 : N :      -1     98 : AZERO C2EXC
TO BT SC DR DR SA             :      -1   6640 :     0.0000 : N :      -1     21 : AZERO C2EXC
TO BT BT BT RE DR SA          :      -1   5035 :     0.0000 : N :      -1    275 : AZERO C2EXC
TO BT RE DR SA                :      -1   4140 :     0.0000 : N :      -1    135 : AZERO C2EXC
TO BT BT BT SC BT BT SA       :      -1   3846 :     0.0000 : N :      -1     40 : AZERO C2EXC

sChimneyLS:0:-2 : Terrible Chi2 : different geometry ?

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Chimney_ipc_20250707_135938.png

EVT=Chimney_ipc MOI=sChimneyLS:0:-2 ssst.sh

A : Opticks

Photons reflected at top of LowerChimney

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Chimney_ipc_20250707_135924.png

EVT=Chimney_ipc MOI=sChimneyLS:0:-2 ssst.sh

B : Geant4

Photons escape between Lower and UpperChimney

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Chimney_ipc_20250707_135926.png

EVT=Chimney_ipc MOI=sChimneyLS:0:-2 ssst.sh

B : Geant4

Photons escape between Lower and UpperChimney

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_Chimney_ipc_20250707_135928.png

EVT=Chimney_ipc MOI=sChimneyLS:0:-2 ssst.sh

B : Geant4

Photons escape between Lower and UpperChimney

cxt_min_LowerChimney_overview.png

MOI=sWaterTube:0:-2 cxt_min.sh ## simtrace 2D slice view of LowerChimney

fake boundaries apparent

TargetLS/LowerChimney/UpperChimney issues

LowerChimney and UpperChimney not joined : problem for Geant4 and Opticks
implemented as three separate volumes : fake coincident boundaries between them
many coincident surfaces : problem for Opticks translated geometry

Fakes and coincident boundaries

Geant4 volume-based geometry model gets away with many sins
Opticks surface-based geometry model cannot handle

MAJOR GEOMETRY RE-IMPLEMENTATION NEEDED

Unified geometry : TargetLS LowerChimney UpperChimney
within heirarchical WaterPool geometry

Major geometry issue : BUT little impact on hits

Summary of Input Photon Checks : Hit Count and Chi2

Target	Hit Count A/B	Chi2/ndf	Note
NNVT:0:1000	0.998 +- 0.003	1394/954 = 1.46
Hama:0:1000	1.001 +- 0.004	431/439 = 0.98
Hama:0:0	1.000 +- 0.003	1107/467 = 2.37	near Chimney
PMT_3inch:0:0	0.999 +- 0.013	2.65/6 = 0.44
sTarget:0:-2	0.996 +- 0.004	1839/1652 = 1.11	RandomSph
PMT_20inch_veto:0:1000	0.994 +- 0.012	54/59.0000 = 0.92
3345.569,20623.73,21000	1.129 +- 0.197	50/9 = 5.55	no REFINE
3345.569,20623.73,21000	1.105 +- 0.194	2.34/9 = 0.26	with REFINE
sChimneyLS:0:-2	0.033 +- 0.002	32216/58 = 555	TERRIBLE

Hit counts consistent
Chi2 OK, other than close to Chimney
long distance intersect precision issue fixed with REFINE

Recently Fixed Issues : Thanks to Dingyong, Haosen, Zhenning

Water Pool Related

long distance intersect precision loss caused history Chi2 deviation
speed discrepancy for some photons reflected from WP PMT
- caused by overlapping vetoWater-Tyvek-DeadWater
WP : no Geant4 intersects due to theLV WaterPool injection bug
WP PMT : ~10% more Opticks hits than Geant4 due to qeScale bug in opticks/sysrap/SPMT.h
WP PMT : no Opticks hits

Muon Related

assert on first muon with no photons
crash on 5th muon : non-optical particles ProcessHits corrupted Geant4 Track Info

excess Opticks hits due to lack of theta dependent collection efficiency
excess SPMT hits due to qeScale bug

GPU Cluster Related

different compute capability on cluster and workstation required CMake rejig for CUDA kernels to work across both
SIGINT from unwritable HOME eg: /afs/ihep.ac.cn/users/${USER:0:1}/$USER

Missing Features and Known Issues

Missing Features

Not supported --pmt-hit-type 2
--opticks-mode 3 cannot be used with other anamgr =>
- mode 3 does both Opticks and Geant4 simulations
GPU node gitlab runner for automated validation

To Investigate

validity and performance implications of truncation eg:

export OPTICKS_MAX_BOUNCE=63
export OPTICKS_MAX_TIME=1000 # (ns)

cost of refine:

export OPTICKS_PROPAGATE_REFINE=1
export OPTICKS_PROPAGATE_REFINE_DISTANCE=5000 # mm

Known Issues

LowerChimney/UpperChimney very discrepant : major re-implementation needed
- Expt. needed to find translatable Geant4 geometry

OJ : Minimal Muon Script : ~/j/muon/muon_min.sh

https://code.ihep.ac.cn/blyth/j/-/blob/main/muon/muon_min.sh

#!/bin/bash
#SBATCH --partition=gpu
#SBATCH --qos=normal
#SBATCH --account=junogpu
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=16000
#SBATCH --gpus=v100:1

source /cvmfs/opticks.ihep.ac.cn/oj/releases/J25.4.0_Opticks-v0.5.1/el9_amd64_gcc11/2025_07_17/envset.sh

export CUDA_VISIBLE_DEVICES=0      # pick GPU
export OPTICKS_MAX_BOUNCE=63       # configure truncation
#export OPTICKS_MAX_TIME=1000      # (ns) another way to truncate

tut_detsim.py
    --no-gdml --pmtsd-merge-twindow 1.0
    --event-split
    --output oj_edm.root --user-output oj_user.root
    --evtmax 10 --seed 42
    hepevt
    --exe Muon

Use OJ like J, (slurm sub to GPU cluster/workstation)

muon_min.sh log : 121M photon kernel launch ~18 sec

Welcome to SNiPER v2.1.0
Running @ localhost.localdomain on Mon Jul  7 20:52:04 2025
...

2025-07-07 20:58:15.036 INFO  [2212188] [QSim::simulate@432]  eventID      6 igs (491517, 6, 4, ) MaxSlot 262000000 MaxSlot/M 262 sslice::Desc(igs_slice)
sslice::Desc num_slice 1 TotalPhoton  121298919 TotalPhoton/M 121.298919

2025-07-07 20:58:15.036 INFO  [2212188] [QSim::simulate@450]    0 : sslice {       0,  491517,         0, 121298919}  121.298919
2025-07-07 20:58:33.266 INFO  [2212188] [QSim::simulate@516]
SEvt__MINTIME
 (TAIL - HEAD)/M  18.280804 (head to tail of QSim::simulate method)
 (LEND - LBEG)/M  18.229832 (multilaunch loop begin to end)
 (PCAT - LEND)/M   0.000018 (topfold concat and clear subfold)
 (TAIL - BRES)/M   0.000006 (QSim::reset which saves hits)
 tot_idt/M        17.171942 (sum of kernel execution int64_t stamp differences in microseconds)
 tot_dt           17.171923 int(tot_dt*M)     17171922 (sum of kernel execution double chrono stamp differences in seconds, and scaled to ms)
 tot_gdt/M         1.038782 (sum of SEvt::gather int64_t stamp differences in microseconds)

...
############################## SniperProfiling ##############################
Name                     Count       Total(ms)      Mean(ms)     RMS(ms)
GenTools                 10          2.06400        0.20640      0.38575
DetSimAlg                10          435848.58088   43584.85809  87427.73331
Sum of junotoptask       10          435850.66813   43585.06681  87427.66931
#############################################################################
Terminating @ localhost.localdomain on Mon Jul  7 21:00:42 2025

~8 min running time, ~18 s kernel time for 121M photons

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105550.png

EVT=muon_cxs ssst.sh #0

export AFOLD_RECORD_SLICE="[:1000000]" ## 1M from 69G record.npy

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105556.png

EVT=muon_cxs ssst.sh #3

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105609.png

EVT=muon_cxs ssst.sh #4

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105610.png

EVT=muon_cxs ssst.sh #5

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105620.png

EVT=muon_cxs ssst.sh #6

GEOM_J25_4_0_opticks_Debug_cxr_min_muon_cxs_20250707_112242.png

EVT=muon_cxs cxr_min.sh #12

GEOM_J25_4_0_opticks_Debug_cxr_min_muon_cxs_20250707_112243.png

EVT=muon_cxs cxr_min.sh #13

GEOM_J25_4_0_opticks_Debug_cxr_min_muon_cxs_20250707_112244.png

EVT=muon_cxs cxr_min.sh #14

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105700.png

EVT=muon_cxs ssst.sh #8

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105701.png

EVT=muon_cxs ssst.sh #9

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105702.png

EVT=muon_cxs ssst.sh #10

GEOM_J25_4_0_opticks_Debug_SGLFW_SOPTIX_Scene_test_muon_cxs_20250707_105704.png

EVT=muon_cxs ssst.sh #11

Summary and Links

Opticks : state-of-the-art GPU ray traced optical simulation integrated with Geant4, with automated geometry translation into GPU optimized form.

Opticks+JUNOSW : ready for testing, despite Chimney issue.

Please report issues/findings/performance

https://github.com:simoncblyth/opticks.git	day-to-day code repository
https://simoncblyth.github.io	presentations and videos
https://groups.io/g/opticks	forum/mailing list archive
email: `opticks+subscribe@groups.io`	subscribe to mailing list
`simon.c.blyth@gmail.com`	any questions

NEXT Development Steps, User Steps

investigate WP_PMT side intersect difference
remove --no-"other"-anamgr requirement for --opticks-mode 3
re-implement TargetLS/LowerChimney/UpperChimney
setup GPU Gitlab runner to automate (SEvt/NumPy based):
- history Chi2 + hit consistency testing
- low stats histogram comparison : time, position, ...
production experience, is development needed to decide CPU/GPU ?

muon running, validity + performance comparison
higher stats hit histogram comparison between OJ and J (root based)
- script to automate comparison, for gitlab CI inclusion

Opticks+JUNOSW NEEDS BRAVE USER(S): Experience Cutting Edge of Optical Photon Simulation

Extra Slide for Context

Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7/8

Geometry Model Translation : Geant4 => CSGFoundry => NVIDIA OptiX 7/8

Geant4 Geometry Model (JUNO: 400k PV, deep hierarchy)

PV	G4VPhysicalVolume	placed, refs LV
LV	G4LogicalVolume	unplaced, refs SO
SO	G4VSolid,G4BooleanSolid	binary tree of SO "nodes"

Opticks CSGFoundry Geometry Model (index references)

struct	Notes	Geant4 Equivalent
CSGFoundry	vectors of the below, easily serialized + uploaded + used on GPU	None
qat4	4x4 transform refs CSGSolid using "spare" 4th column (becomes IAS)	Transforms ref from PV
CSGSolid	refs sequence of CSGPrim	Grouped Vols + Remainder
CSGPrim	bbox, refs sequence of CSGNode, root of CSG Tree of nodes	root G4VSolid
CSGNode	CSG node parameters (JUNO: ~23k CSGNode)	node G4VSolid

NVIDIA OptiX 7/8 Geometry Acceleration Structures (JUNO: 1 IAS + 10 GAS, 2-level hierarchy)

IAS	Instance Acceleration Structures	JUNO: 1 IAS created from vector of ~50k qat4 (JUNO)
GAS	Geometry Acceleration Structures	JUNO: 10 GAS created from 10 CSGSolid (which refs CSGPrim,CSGNode )

JUNO : Geant4 ~400k volumes "factorized" into 1 OptiX IAS referencing ~10 GAS