PSyclone: [LFRic] [PSyAD] Missing halo exchange and immediate cleaning of halos when using setval_random with redundant calculations

We are testing some LFRic branches that run the adjoint tests in parallel. We find that when using setval_random there is a missing halo exchange in the PSy layer code after the field values have been randomised. This causes some failures in adjoint tests that would otherwise pass if the halo is properly initialised.

With redundant calculations, the halo exchange would be missed because after the field is randomised, the halo is set to dirty but then immediately set to clean. This isn’t present when using setval_random normally. However, there is also no halo exchange after setval_random is used before it enters the forward kernels, which when run in parallel causes floating invalid errors.

Ultimately, we need to perform a halo exchange between the use of setval_random and the forward kernel in order to get the tests to work properly, whilst still randomising the annexed dofs.

As an example, looking at atlt_vorticity_advection_alg_mod we have the following invoke:

call invoke( setval_random(r_u), setval_x(r_u_input, r_u), &
             setval_random(wind), setval_x(wind_input, wind), &
             setval_random(vorticity), setval_x(vorticity_input, vorticity), &
             setval_random(ls_wind), setval_random(ls_vorticity), &
             tl_vorticity_advection_kernel_type(r_u, wind, vorticity, ls_wind, ls_vorticity, chi, panel_id, qr_xyoz), &
             x_innerproduct_x(r_u_inner_prod, r_u), &
             x_innerproduct_x(wind_inner_prod, wind), &
             x_innerproduct_x(vorticity_inner_prod, vorticity) )

which when compiled using LFRic parallel optimisations and the options described above yields

      !
      ! Set-up all of the loop bounds
      !
      loop0_start = 1
      loop0_stop = r_u_proxy%vspace%get_last_dof_halo(1)
      loop1_start = 1
      loop1_stop = r_u_input_proxy%vspace%get_last_dof_annexed()
      loop2_start = 1
      loop2_stop = wind_proxy%vspace%get_last_dof_halo(1)
      loop3_start = 1
      loop3_stop = wind_input_proxy%vspace%get_last_dof_annexed()
      loop4_start = 1
      loop4_stop = vorticity_proxy%vspace%get_last_dof_halo(1)
      loop5_start = 1
      loop5_stop = vorticity_input_proxy%vspace%get_last_dof_annexed()
      loop6_start = 1
      loop6_stop = ls_wind_proxy%vspace%get_last_dof_halo(1)
      loop7_start = 1
      loop7_stop = ls_vorticity_proxy%vspace%get_last_dof_halo(1)
      loop8_start = 1
      loop8_stop = ncolour
      loop9_start = 1
      loop10_start = 1
      loop10_stop = r_u_proxy%vspace%get_last_dof_owned()
      loop11_start = 1
      loop11_stop = wind_proxy%vspace%get_last_dof_owned()
      loop12_start = 1
      loop12_stop = vorticity_proxy%vspace%get_last_dof_owned()
      !
      ! Call kernels and communication routines
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop0_start,loop0_stop
        CALL RANDOM_NUMBER(r_u_data(df))
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL r_u_proxy%set_dirty()
      CALL r_u_proxy%set_clean(1)
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop1_start,loop1_stop
        r_u_input_data(df) = r_u_data(df)
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL r_u_input_proxy%set_dirty()
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop2_start,loop2_stop
        CALL RANDOM_NUMBER(wind_data(df))
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL wind_proxy%set_dirty()
      CALL wind_proxy%set_clean(1)
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop3_start,loop3_stop
        wind_input_data(df) = wind_data(df)
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL wind_input_proxy%set_dirty()
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop4_start,loop4_stop
        CALL RANDOM_NUMBER(vorticity_data(df))
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL vorticity_proxy%set_dirty()
      CALL vorticity_proxy%set_clean(1)
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop5_start,loop5_stop
        vorticity_input_data(df) = vorticity_data(df)
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL vorticity_input_proxy%set_dirty()
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop6_start,loop6_stop
        CALL RANDOM_NUMBER(ls_wind_data(df))
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL ls_wind_proxy%set_dirty()
      CALL ls_wind_proxy%set_clean(1)
      !
      ! End of set dirty/clean section for above loop(s)
      !
      !$omp parallel default(shared), private(df)
      !$omp do schedule(static)
      DO df=loop7_start,loop7_stop
        CALL RANDOM_NUMBER(ls_vorticity_data(df))
      END DO
      !$omp end do
      !$omp end parallel
      !
      ! Set halos dirty/clean for fields modified in the above loop(s)
      !
      CALL ls_vorticity_proxy%set_dirty()
      CALL ls_vorticity_proxy%set_clean(1)
      !
      ! End of set dirty/clean section for above loop(s)
      !
      IF (chi_proxy(1)%is_dirty(depth=1)) THEN
        CALL chi_proxy(1)%halo_exchange(depth=1)
      END IF
      !
      IF (chi_proxy(2)%is_dirty(depth=1)) THEN
        CALL chi_proxy(2)%halo_exchange(depth=1)
      END IF
      !
      IF (chi_proxy(3)%is_dirty(depth=1)) THEN
        CALL chi_proxy(3)%halo_exchange(depth=1)
      END IF
      !
      IF (panel_id_proxy%is_dirty(depth=1)) THEN
        CALL panel_id_proxy%halo_exchange(depth=1)
      END IF
      !
      DO colour=loop8_start,loop8_stop
        !$omp parallel default(shared), private(cell)
        !$omp do schedule(static)
        DO cell=loop9_start,last_halo_cell_all_colours(colour,1)
          !
          CALL tl_vorticity_advection_code(nlayers, r_u_data, wind_data, vorticity_data, ls_wind_data, ls_vorticity_data, &
&chi_1_data, chi_2_data, chi_3_data, panel_id_data, ndf_w2, undf_w2, map_w2(:,cmap(colour,cell)), basis_w2_qr_xyoz, ndf_w1, &
&undf_w1, map_w1(:,cmap(colour,cell)), basis_w1_qr_xyoz, ndf_aspc9_chi, undf_aspc9_chi, map_aspc9_chi(:,cmap(colour,cell)), &
&basis_aspc9_chi_qr_xyoz, diff_basis_aspc9_chi_qr_xyoz, ndf_adspc3_panel_id, undf_adspc3_panel_id, &
&map_adspc3_panel_id(:,cmap(colour,cell)), np_xy_qr_xyoz, np_z_qr_xyoz, weights_xy_qr_xyoz, weights_z_qr_xyoz)
        END DO
        !$omp end do
        !$omp end parallel
      END DO
      !
      ! Set halos dirty/clean for fields modified in the above loop
      !
      CALL r_u_proxy%set_dirty()
      !
      ... ...

where I have omitted the inner product calculations for brevity. Running this as-is causes a floating invalid error in the forward code.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 33 (21 by maintainers)

Most upvoted comments

Thanks Iva, from discussion with Terry, the problem seems to be that this is the first time anyone has tried to run the PSyAD test harness in parallel.

Thanks @DrTVockerodtMO, it’s possible fixing #2992 would solve our problems rather than switching COMPUTE_ANNEXED_DOFS back on.

I am suggesting that in the meantime that we switch COMPUTE_ANNEXED_DOFS (CAD) back on until this #2992 problem is resolved, as it breaks the adjoint model. Turning CAD true will break the adjoint tests, but importantly the model itself will not be broken. For local testing we’ll just need to add halo exchanges manually after setval_random until that is resolved. When a solution is found for either #2992, or using the setval_random with CAD=true, then the tests will be fixed.

This was causing some floating invalid errors when running the adjoint code, and it helped uncover a bug in our patches. Essentially the metadata for kernels with multiple code implementations inherits from the forward code, which makes sense because sometimes multiple routines in the code can correspond to helper functions etc. where one would not be able to derive the metadata. I made an issue previously about it here: https://github.com/stfc/PSyclone/issues/2495.

What I focused more on was the naming of the adjoint kernel, but in fact it’s the entire metadata that is inherited. This includes, for instance, having GH_READ for fields that should be GH_WRITE or GH_INC! The main difference there is that the vital halo exchanges are not called when running the application which cause the floating point errors. After modifying the kernel metadata to be correct I now get these particular kernels with floating point errors to pass! I’ll include the patch changes on my LFRic apps branch.

I had a discussion with @arporter and it looks like setting COMPUTE_ANNEXED_DOFS to false in the psyclone.cfg fixes this. I will do some more testing to see the differences with it on and off.

I have a branch in LFRic apps open which plumbs in a custom psyclone.cfg specifically for the adjoint_tests application since this is the only thing currently that uses setval_random. The adjoint itself by default will use the global config so that the other builtins are not slowed down by unnecessary halo exchanges.

I won’t close the issue yet as I would like to add the diffs of the PSycloned adjoint tests so we have a better understanding as to what is going on.

It would seem to me that setval_random is a special case here - each MPI PE is not guaranteed to generate values which match the “owned” dofs on another partition, due to the order in which the random numbers are generated (this will naturally change as the partitioning & indexing changes). Perhaps there is a clever way to account for this somehow? But if not then the only way these match would be with a halo exchange, otherwise the halo will contain “dirty” halo information. Sorry that this doesn’t help address the main issue here much…

That’s a very good point which I hadn’t considered: essentially it’s not possible to do redundant ‘computation’ when creating pseudo random numbers (well, I guess it is possible but it would be quite tricky). Have you tried turning off the ‘annexed dofs’ configuration option?

There must be something special going on here as, from a PSyclone point of view, there’s nothing special about setval_random c.f. the other setval_* builtins. They’ve been in use for a long time with no problems reported about missing halo exchanges. However, you mention ‘annexed dofs’ and that might be relevant. If you are running PSyclone with the ‘annexed dofs’ option set to True (https://psyclone.readthedocs.io/en/latest/dynamo0p3.html#annexed-dofs) then all builtins will (or should) compute out to the L1 halo to ensure that annexed dofs are clean. Does that shed any light on the situation?