Week 8: HPC Cluster Benchmarking on Fugaku & Final Testing
Following up on the local performance benchmarking from Week 7, our primary goal for Week 8 was to test the PETSc backend in a true High-Performance Computing (HPC) environment. We deployed our benchmarks on the Fugaku Cluster to measure MPI strong scaling beyond the constraints of local machines and to identify the true performance ceiling of our implementation. Additionally, we codified our testing framework, adding robust unit tests for the PETSc backend into the standard QuTiP test suite.
1. Fugaku Cluster Benchmarking
Our local tests showed that PETSc significantly outperforms the traditional QuTiP csr backend for very deep hierarchies. However, local strong scaling degraded beyond 4 MPI ranks due to hardware limitations and communication overhead.
Running on the Fugaku cluster allowed us to utilize a high-bandwidth interconnect and many more physical cores. We reproduced our benchmarks scaling the Hierarchy Depth and the MPI Ranks.
1.1 Scaling the Hierarchy Depth (N=2, 1 Bath)
We fixed the system size to N=2 with 1 bath and scaled max_depth up to 22 (80,730 ADOs). This time, we used 32 MPI ranks on the Fugaku cluster and compared it against our best local PETSc run (4 ranks) and the single-core Traditional solver.
| Depth | ADOs | Traditional (1 Rank) | PETSc Local (4 Ranks) | PETSc Cluster (32 Ranks) | Speedup (Cluster vs Trad) |
|---|---|---|---|---|---|
| 6 | 462 | 1.55 s | 3.39 s | 3.29 s | 0.47x |
| 10 | 3003 | 29.71 s | 19.65 s | 6.88 s | 4.31x |
| 14 | 11628 | 174.89 s | 70.46 s | 20.24 s | 8.64x |
| 18 | 33649 | 644.50 s | 181.89 s | 52.70 s | 12.22x |
| 22 | 80730 | 1985.08 s | 397.87 s | 115.00 s | 17.26x |
Key Insights: - Massive Performance Gains: With 32 ranks on the cluster, PETSc reduces the evolution time of the 80k ADO system from 33 minutes (traditional) to under 2 minutes, an incredible 17.2x speedup. - Overhead Masking: Even at smaller depths (3000 ADOs), the 32-rank cluster significantly outperforms the traditional solver, whereas on the local machine the gap was marginal.
1.2 MPI Strong Scaling (Depth 22, 80,730 ADOs)
To see how well our implementation distributes work, we ran strong scaling benchmarks up to 64 MPI ranks on Fugaku.
| MPI Ranks | PETSc Local (s) | PETSc Cluster (s) |
|---|---|---|
| 1 | 677.71 | 3486.70 |
| 2 | 424.15 | 1696.40 |
| 4 | 397.87 | 854.60 |
| 8 | 486.92 (Degraded) | 471.76 |
| 16 | - | 233.00 |
| 32 | - | 115.40 |
| 64 | - | 59.10 |
(Note: Traditional Baseline = 1985.08 s)
Key Insights: - Near-Linear Scaling on Cluster: The Fugaku cluster exhibits near-perfect linear scaling from 1 rank all the way to 64 ranks. Halving the ranks roughly doubles the time. - Single-Core Cluster Deficit: An interesting observation is that a single Fugaku core (3486 s) is slower than a single local core (677 s). However, because Fugaku scales linearly without communication bottlenecks up to 64 ranks, it ultimately finishes the task over 6 times faster than the best local configuration. - Breaking the 1-Minute Barrier: By using 64 ranks, we managed to solve a system with 80,730 ADOs in just 59.1 seconds.
2. Integration and Testing
With the performance capabilities proven, we solidified the implementation into the QuTiP framework: - Test Suite (qutip/tests/solver/heom/test_petsc_backend.py): We wrote a comprehensive suite of unit tests for the PETSc backend to ensure long-term stability. - Bug Fixes: Resolved an edge case where QuTiP 5 generated Dense components for L_sys which PETSc failed to digest. We added automated fallback conversions to CSR format prior to assembly. - Accuracy Verification: The PETSc backend implicitly utilizes Backward Differentiation Formulas (BDF), which has different numerical characteristics than the classic Adams method. Tests confirmed fidelity scores > 99.99\% with appropriately adapted absolute tolerances (atol=1e-2).
Conclusion
Week 8 successfully demonstrated that our PETSc integration is not just a theoretical improvement, but a highly scalable solution capable of leveraging world-class supercomputers like Fugaku. We have successfully broken past the memory constraints and execution time limits of the traditional solver, opening the door for QuTiP users to simulate massive open quantum systems.