Integration of JUNO Offline + Opticks
Integration of JUNO Offline + Opticks :
GPU Optical Simulation with NVIDIA® OptiX™
Open source, https://bitbucket.org/simoncblyth/opticks
Simon C Blyth, IHEP, CAS — JUNO Collaboration Meeting, 6 July 2020, Virtual
Outline
- Introduction (pp 2-12)
- Optical Photon Simulation Problem...
- Rasterization and Ray Tracing
- Ray Traced Rendering ~ Photon Simulation
- Turing Built for RTX ; Boundary Volume Heirarchy (BVH) algorithm
- NVIDIA OptiX Ray Tracing Engine
- Geant4 + Opticks Hybrid Workflow
- Opticks : Translates G4 Optical Physics to CUDA/OptiX
- G4Solid -> CUDA Intersect Functions for ~10 Primitives
- G4Boolean -> CUDA/OptiX Intersection Program Implementing CSG
- Opticks : Translates G4 Geometry to GPU, Without Approximation
- Necessary JUNO Geometry Changes (pp 15-22) [SVN trunk r3990]
- Remove all Torus to enable translation to GPU
- "tubs-torus" -> polycone : --pmt20inch-polycone-neck
- HamamatsuR12860 & NNVTMCPPMT : neck comparisons
- Integration of JUNO Offline + Opticks (pp 23-26) [SVN trunk r3990]
- Opticks is now a JunoENV External
- JUNO Offline : Where WITH_G4OPTICKS used ?
- Using Offline + Opticks : tut_detsim.py options
- Next Step : Theta dependent Collection Efficiency -> GPU
- Opticks + Geant4 News : Two Geant4 Task Force R&Ds Based on Opticks (p27)
- Summary and Links (p28)
JUNO Optical Photon Simulation Problem...
Ray-tracing vs Rasterization
Optical Photon Simulation ≈ Ray Traced Image Rendering
Much in common : geometry, light sources, optical physics
- simulation : photon parameters at PMT detectors
- rendering : pixel values at image plane
- both limited by ray geometry intersection, aka ray tracing
Many Applications of ray tracing :
- advertising, design, architecture, films, games,...
- -> huge efforts to improve hw+sw over 30 yrs
- Up to "10 GigaRays per second" per GPU
https://nvidianews.nvidia.com/news/nvidia-reveals-the-titan-of-turing-titan-rtx
Spatial Index Acceleration Structure
Geant4OpticksWorkflow
Opticks : Translates G4 Optical Physics to CUDA/OptiX
OptiX : single-ray programming model -> line-by-line translation
- CUDA Ports of Geant4 classes
- G4Cerenkov (only generation loop)
- G4Scintillation (only generation loop)
- G4OpAbsorption
- G4OpRayleigh
- G4OpBoundaryProcess (only a few surface types)
- Modify Cherenkov + Scintillation Processes
- collect genstep, copy to GPU for generation
- avoids copying millions of photons to GPU
- Scintillator Reemission
- fraction of bulk absorbed "reborn" within same thread
- wavelength generated by reemission texture lookup
- Opticks (OptiX/Thrust GPU interoperation)
- OptiX : upload gensteps
- Thrust : seeding, distribute genstep indices to photons
- OptiX : launch photon generation and propagation
- Thrust : pullback photons that hit PMTs
- Thrust : index photon step sequences (optional)
G4Solid -> CUDA Intersect Functions for ~10 Primitives
- 3D parametric ray : ray(x,y,z;t) = rayOrigin + t * rayDirection
- implicit equation of primitive : f(x,y,z) = 0
- -> polynomial in t , roots: t > t_min -> intersection positions + surface normals
G4Boolean -> CUDA/OptiX Intersection Program Implementing CSG
Complete Binary Tree, pick between pairs of nearest intersects:
| UNION tA < tB |
Enter B |
Exit B |
Miss B |
|---|
| Enter A |
ReturnA |
LoopA |
ReturnA |
| Exit A |
ReturnA |
ReturnB |
ReturnA |
| Miss A |
ReturnB |
ReturnB |
ReturnMiss |
- Nearest hit intersect algorithm [1] avoids state
- sometimes Loop : advance t_min , re-intersect both
- classification shows if inside/outside
- Evaluative [2] implementation emulates recursion:
- recursion not allowed in OptiX intersect programs
- bit twiddle traversal of complete binary tree
- stacks of postorder slices and intersects
- Identical geometry to Geant4
- solving the same polynomials
- near perfect intersection match
- [1] Ray Tracing CSG Objects Using Single Hit Intersections, Andrew Kensler (2006)
- with corrections by author of XRT Raytracer http://xrt.wikidot.com/doc:csg
- [2] https://bitbucket.org/simoncblyth/opticks/src/master/optixrap/cu/csg_intersect_boolean.h
- Similar to binary expression tree evaluation using postorder traverse.
Opticks : Translates G4 Geometry to GPU, Without Approximation
G4 Structure Tree -> Instance+Global Arrays -> OptiX
Group structure into repeated instances + global remainder:
- auto-identify repeated geometry with "progeny digests"
- JUNO (SVN trunk) : 9 distinct instances + 1 global
- instance transforms used in OptiX/OpenGL geometry
instancing -> huge memory savings for JUNO PMTs
j1808_top_rtx
j1808_top_ogl
JUNO Geometry : Remove Torus to enable translation to GPU
tut_detsim.py options (in SVN trunk r3990)
- --no-guide_tube
- skip the guide tube torus
- --pmt20inch-polycone-neck
"cylinder - torus" -> polycone
- NNVT_MCPPMT_PMTSolid
- Hamamatsu_R12860_PMTSolid
- TODO: --pmt20inch-simpler-csg
- simpler+faster CSG, same geometry
"tubs-torus" -> polycone : --pmt20inch-polycone-neck
gpltcf_HamamatsuR12860lMaskVirtual_SubPlotsFig
gpltcf_HamamatsuR12860lMaskVirtual_SubPlotsFig_Zoom
gpltcf_HamamatsuR12860lMaskVirtual_combiZoom_SubPlotsFig
gpltcf_NNVTMCPPMTlMaskVirtual_SubPlotsFig
gpltcf_NNVTMCPPMTlMaskVirtual_SubPlotsFig_Zoom
gpltcf_NNVTMCPPMTlMaskVirtual_combiZoom_SubPlotsFig
Integration of Offline + Opticks : Opticks is now a JunoENV External
Add Opticks to an existing installation (SVN trunk r3990+):
bash junoenv libs all opticks
- pre-requisites : NVIDIA Driver, CUDA, OptiX
Four mandatory envvars :
export CMTEXTRATAGS=opticks
# -> Opticks ON in JunoENV + CMT
export OPTICKS_COMPUTE_CAPABILITY=70 # -> nvcc flags
export OPTICKS_CUDA_PREFIX=/usr/local/cuda-10.1
export OPTICKS_OPTIX_PREFIX=/usr/local/OptiX_650
# location of CUDA and OptiX installs
Build all JUNO externals, including Opticks (takes ~1hr, 13G) :
jlibs(){
mkdir -p $JUNOTOP && cd $JUNOTOP
svn co https://juno.ihep.ac.cn/svn/offline/trunk/installation/junoenv && cd $JUNOTOP/junoenv
local libs=$(bash junoenv libs list | perl -ne 'm, (\S*)@, && print "$1\n"' -)
for lib in $libs ; do
bash junoenv libs all $lib || return 1
done
}
Integration of Offline + Opticks : Where WITH_G4OPTICKS used ?
Offline Opticks : #ifdef WITH_G4OPTICKS ... #endif + if(m_opticksMode > 0){ ... }
cd $JUNOTOP/offline/Simulation/DetSimV2 ; find . -type f -exec grep -l WITH_G4OPTICKS {} \+
./DetSimOptions/src/DetSim0Svc.cc ## initialize/finalize
./DetSimOptions/src/LSExpDetectorConstruction.cc ## translate geometry to GPU
./PhysiSim/src/DsG4Scintillation.cc ## collect Scintillation gensteps, excluding reemission
./PhysiSim/src/LocalG4Cerenkov1042.cc ## collect Cerenkov gensteps
./PhysiSim/src/DsPhysConsOptical.cc ## use LocalG4Cerenkov1042 rather than G4Cerenkov
./PMTSim/src/PMTSDMgr.cc ## setup separate PMTHitMerger m_pmthitmerger_opticks
./PMTSim/src/junoSD_PMT_v2.cc ## bulk hit creation+collection at EndOfEvent
- junoSD_PMT_v2::Initialize
- setup extra hitCollection_opticks (temporarily doing both GPU+CPU sim, with separate hits + merger)
- junoSD_PMT_v2::EndOfEvent_Opticks
- invoke G4Opticks::propagateOpticalPhotons with the gensteps collected
- bulk populate hitCollection_opticks
Integration of Offline + Opticks : tut_detsim.py options
tut_detsim.py options relevant to Opticks usage
- --opticks-mode 0
- do not use Opticks GPU propagation, the default
- --opticks-mode 1 (currently SVN trunk r3990)
- performs both CPU and GPU optical simulations
put GPU hits into separate hitCollectionOpticks
- --opticks-mode 10 # TODO: IMPLEMENT AFTER FURTHER VALIDATION
- switch off CPU optical photons,
put GPU hits into standard hitCollection
- --no-guide_tube
- change geometry skipping the guide tube torus
- --pmt20inch-polycone-neck
- change geometry of LPMT necks avoiding use of torus
tds : Example Bash Function :
tds-(){ python $JUNOTOP/offline/Examples/Tutorial/share/tut_detsim.py $* ; }
tds(){ tds- --opticks-mode 1 --no-guide_tube --pmt20inch-polycone-neck --evtmax 2 gun ; }
Integration of Offline + Opticks : Next Step : Theta dep. CE -> GPU
(1) Complete CPU side bulk hit creation in PMTSim: junoSD_PMT_v2::EndOfEvent_Opticks
Hits currently missing:
- local_pos : position in PMT frame
- theta dependent collection efficiency culling
- pAcrylic/pInnerWater boundary_pos[1] : is this essential ?
(2) CUDA port: junoSD_PMT_v2::get_ce
- interpolation of : 9 collection efficiency theta samples 0->90 deg
- PMTParamSvc info (PMTType[2], QEType) -> GPU
- Port G4DataInterpolation::CubicSplineInterpolation ?
- direct port difficult + lots of flops for every detected photon (perhaps [3])
- instead "cheat" with GPU texture "tables" (similar to reemission_texture approach)
- 9 theta samples -> G4DataInterpolation -> 90 theta "samples" -> theta_texture
- theta_texture(theta) -> ce : GPU hardware linear interpolation
[1] set in NormalTrackInfo by NormalAnaMgr::UserSteppingAction
[2] PMTID int distinguishes NNVT/Hamamatsu !
[3] https://github.com/DannyRuijters/CubicInterpolationCUDA "GPU Prefilter for Accurate Cubic B-spline Interpolation"
Opticks + Geant4 News : Two Geant4 Task Force R&Ds Based on Opticks
Full integration of Opticks with Geant4 : LAr TPCs, DUNE
Creation of a Geant4 extended example G4OpticksTest
Advised Hans on Opticks use, adding Wavelength Shifting
Prototype of gamma or electron transport using GPU vendor library : HL-LHC ECAL
Examine the Opticks simulation of optical photons, to identify similarities/differences ... for gamma and/or electron interactions
Agreed to act in a consulting role for this task ; email exchanges with John Apostolakis
[1] HSF : HEP Software Foundation
Summary and Links
Opticks : state-of-the-art GPU ray traced optical simulation
integrated with Geant4. Now integrated with JUNO Offline
Pioneer users welcome
- Drastic speedup -> better detector understanding -> greater precision
- any simulation limited by optical photons can benefit
- more photon limited -> more overall speedup (99% -> 100x)
scan-pf-1_Opticks_vs_Geant4 2
| JUNO analytic, 400M photons from center |
Speedup |
|---|
| Geant4 Extrap. |
95,600 s (26 hrs) |
|
| Opticks RTX ON (i) |
58 s |
1650x |
PMTID physvol/@copynumber -?-> PMTtype Hamamatsu/NNVT -- GDML:
014743 <physvol name="pLPMT_NNVT_MCPPMT0x329e330"> <-- zero is special ? NOT A GOOD ID
14744 <volumeref ref="NNVTMCPPMTlMaskVirtual0x32a56c0"/>
14745 <position name="pLPMT_NNVT_MCPPMT0x329e330_pos" unit="mm" x="-930.2976" y="-111.8724" z="19365"/>
14746 <rotation name="pLPMT_NNVT_MCPPMT0x329e330_rot" unit="deg" x="180" y="0" z="180"/>
14747 </physvol>
14748 <physvol copynumber="1" name="pLPMT_Hamamatsu_R128600x329e3b0">
14749 <volumeref ref="HamamatsuR12860lMaskVirtual0x3290b70"/>
14750 <position name="pLPMT_Hamamatsu_R128600x329e3b0_pos" unit="mm" x="-492.5657" y="-797.0872" z="19365"/>
14751 <rotation name="pLPMT_Hamamatsu_R128600x329e3b0_rot" unit="deg" x="180" y="0" z="180"/>
14752 </physvol>
14753 <physvol copynumber="2" name="pLPMT_NNVT_MCPPMT0x329e4b0"> <-- forced to use name OR PMTParamSvc -> PMTType
14754 <volumeref ref="NNVTMCPPMTlMaskVirtual0x32a56c0"/>
14755 <position name="pLPMT_NNVT_MCPPMT0x329e4b0_pos" unit="mm" x="316.0782" y="-882.0791" z="19365"/>
14756 <rotation name="pLPMT_NNVT_MCPPMT0x329e4b0_rot" unit="deg" x="180" y="0" z="180"/>
14757 </physvol>
...
102774 <physvol copynumber="17610" name="pLPMT_Hamamatsu_R128600x3a2a9d0"> Simple fix : separate NNVT/Hamamatsu offsets
102775 <volumeref ref="HamamatsuR12860lMaskVirtual0x3290b70"/>
102776 <position name="pLPMT_Hamamatsu_R128600x3a2a9d0_pos" unit="mm" x="-468.5" y="811.4658" z="-19365"/>
102777 </physvol>
102778 <physvol copynumber="17611" name="pLPMT_NNVT_MCPPMT0x3a2aa80">
102779 <volumeref ref="NNVTMCPPMTlMaskVirtual0x32a56c0"/>
102780 <position name="pLPMT_NNVT_MCPPMT0x3a2aa80_pos" unit="mm" x="-926.5345" y="139.6526" z="-19365"/>
102781 </physvol>
102782 <physvol copynumber="300000" name="PMT_3inch_log_phys0x3a307c0">
102783 <volumeref ref="PMT_3inch_log0x3a2e430"/>
102784 <position name="PMT_3inch_log_phys0x3a307c0_pos" unit="mm" x="-3734.24733197939" y="1835.06563378074" z="18932.1786116475"/>
102785 <rotation name="PMT_3inch_log_phys0x3a307c0_rot" unit="deg" x="-174.463707019134" y="-11.1072339347848" z="-26.7089289722046"/>
102786 </physvol>
102787 <physvol copynumber="300001" name="PMT_3inch_log_phys0x3a30890">
102788 <volumeref ref="PMT_3inch_log0x3a2e430"/>
102789 <position name="PMT_3inch_log_phys0x3a30890_pos" unit="mm" x="-3470.82479867782" y="1705.61581563788" z="18994.3071973244"/>
102790 <rotation name="PMT_3inch_log_phys0x3a30890_rot" unit="deg" x="-174.86882108069" y="-10.3147892102431" z="-26.6336342093065"/>
102791 </physvol>
PMTParamSvc relies on large std::map<int,int> m_pmt_categories
- unsigned long long pmtdat
- 64-bit bitfield can encode all PMTParamSvc info in CUDA friendly way
- label sensitive volumes with pmtdat
- bit twiddling then provides all the info, no big PMTParam buffers needed
43 class PMTParamSvc: public SvcBase { // Encode all this into : 64 bit pmtdat (unsigned long long)
44 public: //
45 PMTParamSvc(const std::string& n) //
.. // 16 bits : which PMT identifier (enough for 1..65,536)
53 bool isCD(int pmtid); // 16 bits : 8 bits of type flags + 8 bits spare
54 bool isWP(int pmtid); // 32 bits : float qe
56 bool is20inch(int pmtid); // ----------------------------------------------
57 bool is3inch(int pmtid); // 64 bits : unsigned long long pmtdat
59 bool isNNVT(int pmtid); //
60 bool isHamamatsu(int pmtid); //
61 bool isHighQE(int pmtid); // Adhere to hex 4bit boundaries:
62 bool isHZC(int pmtid); //
// Which PMT 16 bits
63 float getPMTQE(int pmtid); // |
.. // 0x|....|....|..00|....|
74 std::map<int, int> m_pmt_categories; // |
75 std::map<int, float> m_pmt_qe; // float qe type flags
76 }; //
// unsigned long long pmtdata[npmt] ;
- pmtdat will simplify bringing junoSD_PMT_v2::get_ce to GPU.
