pimd: Multicast routes don't always get their iif updated based on unicast routing changes

The network setup we have is:

S+Rn_.y = multicast source and receiver n, 
                    where .y is the least significant octet of the IPv4 address

Cisco_n = Cisco PIM Router
pimd_n = pimd PIM Router (pimd v2.3.2)

The RP is the router at the top of the picture.

              Cisco_1 --------------------  <BGP>
                 |                        |
S+R1_.165 -------| <ospf>                 |
                 |                     Cisco_2 ------ S+R2_.46
               Cisco_3                    |
                 |                        | <BGP>
S+R3_.117 -------| <ospf>                 |
                 |                        |
                 | <eth0>                 |
               pimd_4                     |
                 | <ndl0>                 |
                 |                        |
                 | <ospf>                 |
                 |                        |
                 | <ndl0>                 |
               pimd_5                     |
                 | <eth0>                 |
                 |                        |
                 | <ospf>                 |
S+R4_.101 -------|------- Cisco_6         |
                 |           |            |
                 |           | <ospf>     |
                 |           |            |
               pimd_7      Cisco_8        |
                 |           |            |
           <ospf>|        S+R5_.85        |
                 |                        |
               pimd_9                     |
                 |                        |
                 |<ospf>                  |
                 |                        |
              Cisco_10 --------------------

Reachability via OSPF is preferred.
The BGP link between Cisco_10 and Cisco_2 is preferred for all pimd_n routers.
The Cisco_1 router is the only router that prefers the direct BGP route to the S+R2_.46 network.

We have intermittent failure when the link between pimd routers is restored. i.e. If the link pimd_4 and pimd_5 has been down long enough for the unicast routing to converge when it is restored the multicast routing doesn’t always converge (fails approx. 50% of the time). Specifically the multicast receiver S+R3_.117 might not see traffic from sources S+R4_.101 and / or S+R5_.85 and / or source S+R2_.46. The combination of which sources are lost from which receiver varies in the current setup.

Tracking through debug logs it appears that the change in the unicast routing is not always picked up.

The attached logs are from pimd_4 and pimd_5.

CoolSq_10.200.55.119_20160615-160927_pimd_4.txt SqCool_10.200.55.106_20160615-161131_pimd_5.txt

Events of interest are on the pimd_4 router log (I’ve bolded the timestamps of the symptoms):

• 23:13:58 (approx) link is restored
• 23:13:59.034 : first report of multicast packet being received over the wrong iif after the link restored.
• 23:13:59.232 : PIM neigbour restored with pimd_5
• 23:13:59.471 : first join/prune received from pimd_5
• 23:14:20.423 : it looks like the unicast_routing_timer has triggered and the various group routes are being tested. Both the sources .85 and .101 return a rpf of vif 1, but you can see by comparing the previous and next summaries that .85 has changed Incoming : I.. to Incoming : I.. while .101 remains Incoming : I..
• 23:14:40 : it looks like the source .101 has the incoming interface updated correctly but there are still subsequent reports of the multcast coming in on the wrong interface 23:14:57.732 Wrong iif: src 10.200.55.101, dst 239.0.4.1, iif 1

It is as if the “change_interfaces” call on line 641 of timer.c isn’t actually causing the change to be applied to the group, maybe it needs the force change flag set?