Repository navigation
Block-structured adaptive mesh refinement (AMR) - #2117
Open
sbryngelson wants to merge 12 commits into
Open
sbryngelson wants to merge 12 commits into
sbryngelson wants to merge 12 commits into
Conversation
|
Claude Code Review Head SHA: f08f835 Files changed:
Findings:
Not reviewed in depth: the bulk of the |
sbryngelson
force-pushed
the
amr
branch
8 times, most recently
from
November 4, 2026 04:36
be1f45c to
2dd3cca
Compare
sbryngelson
force-pushed
the
amr
branch
2 times, most recently
from
November 5, 2026 08:05
34cbc88 to
ac5becc
Compare
sbryngelson
marked this pull request as ready for review
November 5, 2026 16:21
|
Claude Code Review Head SHA: ac5becc Files changed:
Findings:
Residual: only partial review of the m_amr_*.fpp, riemann solver and rhs hunks; no defect was confirmed there. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #2117 +/- ##
==========================================
+ Coverage 64.27% 64.61% +0.34%
==========================================
Files 89 109 +20
Lines 23733 28860 +5127
Branches 3402 4326 +924
==========================================
+ Hits 15255 18649 +3394
- Misses 5863 6784 +921
- Partials 2615 3427 +812 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Lines of Code
|
…L/HLLC/LF/HLLD, copy primitives only for readers; case-optimized builds compile only the selected Riemann solver
…ir input; fail format on ffmt structure warnings; Cray debug bounds checking; Open MPI oversubscribe; stage-guard lint; validator fixes
…ry helpers, restart I/O
…M, IGR, physics, startup, output)
…iction/reflux, clustering, nesting, regrid, subcycling, restart)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2116 (solver performance) and #2118 (small fixes). Until they merge this diff includes their commits;
the AMR changes are the last 8 commits.
Contribution Policy
All contributions are expected to demonstrate:
Summary
Adds block-structured adaptive mesh refinement to
simulation, off by default (amr = F). With AMR off, results arethose of #2116 (every existing golden passes unchanged).
amr_max_grid_sizecells, refinement ratio 2 (or 4 for one level), up to
amr_max_levelproperly nested levels, regrid everyamr_regrid_intsteps.amr_subcycle, two fine steps per coarse step, up to two refinedlevels) with time-interpolated ghosts. Flux registers reflux every coarse/fine face, so the scheme is conservative.
amr_prolong = 2) with a fallback to linearwherever the quartic state would have non-positive density or internal energy; conservative restriction.
with
amr_place_owner). Exchanges are nonblocking waves with keyed tags, so results do not depend on message timing.Lagrangian bubbles (lock-step, cloud on the base grid), acoustic sources (base grid), 1D MHD, IGR (single rank), and
immersed boundaries (fixed, prescribed or force-driven motion).
docs/documentation/amr.md(user guide),amr_implementation.md(internals), one AMR section incase.md.The solver runs on a block by swapping its grid globals (
m,n,p, index ranges, coordinates) in and out;amr_implementation.mdstates the contract for module state that depends on the grid. AMR validation fails closed: withamr = Tthe validator rejects any logical switch not inAMR_VETTED_FLAGS, so new physics is refused until it meets thatcontract. Explicitly rejected with a message:
model_eqns = 1, Euler-Euler bubbles, QBMM, phase change, surface tension,synthetic turbulence, cylindrical or stretched grids,
bc = -4withnull_weights; subcycling with IGR, chemistry,Lagrangian bubbles or moving bodies.
Verification
Regression
-c frontier_amd, 4 shards)--no-mpi)-c frontier, 4 shards)-c frontier, 4 shards)The full lanes ran before the last restructure, which splits
m_amr_ghostandm_amr_seamout ofm_amr_exchange,folds seven copies of the limited-linear stencil into
f_amr_linand two admissibility checks intof_amr_cons_ok, andfuses the ghost fill, quartic guard and volume-fraction closure into one per-cell kernel (same operations per cell). On
the head: the AMR cases pass on Frontier amdflang OpenMP (52/52), and
simulationbuilds with NVHPC 25.5 and 25.11 OpenMPin the CI container. CI runs the full matrix on the head. The 7 CCE misses are multi-rank AMR cases (np = 2 to 8) off by 1.5e-12 to 1.6e-9
against tolerances of 1e-12 to 1e-10: the goldens come from amdflang, and CCE's multi-rank sums differ in the last digits.
The same 7 cases miss by the same amounts before this PR's cleanup commits. Tolerances are unchanged.
New tests: 56 AMR cases (multi-rank, multi-level, subcycling, restart, IB, quartic prolongation, and AMR variants of the
chemistry, IGR and Lagrangian-bubble cases) and one
rank_time_wrtdiagnostics case. No existing golden changes.Compiler issues found and handled here:
m_amr_regridinto host loops it flattened and dropped stores in them(regrid read garbage block origins and faulted).
m_amr_regridjoins the existing per-file-Oipa0list inCMakeLists.txt.anyway; both are now automatic arrays.
-mp=gpu:fort2segfaults in itshlscrubstore-forwarding pass. The pass reads theaddress constant of the CPU fallback's argument pack as an ILI index (
is_alloc), which faults when that symbol indexexceeds the routine's ILI table. MFC's large kernels keep the table far above the symbol count; the AMR ghost fill was
three one-kernel routines (about 3.5k symbols against a 2-4k table), and source edits only moved the crash. Fusing them
into one kernel removes it with no compiler flag: CI-container builds of 25.5 and 25.11 pass, and every AMR kernel file
compiles under 48 perturbations each (2 versions, 3 paths, 8 line offsets).
gpuParallelization.mdrecords the rule:keep a routine's kernel work in one kernel rather than splitting it into small routines.
-O3: the 1D acoustic-source case under dynamic regrid drifts 3e-12 from the golden, like its static-blockparent, and gets the same 1e-8 tolerance.
Accuracy and conservation
1D periodic Euler, WENO5, SSP-RK3, refinement ratio 2:
L1 density error after one period. Third order is kept; the error constant is larger because the coarse/fine ghost fill is
second order. Total mass, momentum and energy change by 1.4e-16, 4.2e-16 and 1.9e-16 (relative) over 200 steps with regrid
every 4 steps. The same run on 1 and 4 ranks is bitwise identical.
Performance (weak scaling, blast lattice, 8M base cells per GPU, one refined level, subcycled)
s/step (efficiency vs 8 GPUs):
MFC refines about 1.3x more volume than AMReX and PeleC here (40.5% vs 30.3%). The table is from an earlier commit of this
work; at 64 GPUs this PR's tree gives 0.686 and 0.687 s/step in two runs (table: 0.699), timed as
(t240 - t40)/200 of the step-loop wall so startup and the first regrids are excluded. Regrid cost
grows faster than the step beyond 512 GPUs; that is a follow-up.
Use
Mach 2.5 shock through 297 fixed random spheres: 384 x 256 x 256 base grid plus one level, 128 GPUs, 1350 steps, up to
about 5300 blocks.
Known limitations and follow-ups
migrated), which costs accuracy in multi-level runs that regrid often.
the swap but touches the whole solver.
m_amr_stateholds many module variables; a block-table derived type would be cleaner.