cloudstack: Duplicate NETWORK.CREATE usage events resulting in duplicate usage records

problem

For isolated networks and VPC tiers, a usage event NETWORK.CREATE is created in the cloud.usage_event table whenever the network transitions from ‘Allocated’ to ‘Implemented’ state. A NETWORK.DELETE event is created when the network is deleted.

I suspect there is a bug in the code or a bad design causing this issue. These events result in duplicate usage records for the network for the same period.

Either the NETWORK.CREATE should be created only when it is created, or if we need to capture usage for Allocated and Implemented networks separately, there could be another event.

The network below is not deleted.

mysql> select * from cloud.usage_event where  resource_id=207 and type in ('NETWORK.CREATE','NETWORK.DELETE');
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 146 | NETWORK.CREATE |          2 | 2025-04-10 06:02:14 |       1 |         207 | myNet1        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
| 151 | NETWORK.CREATE |          2 | 2025-04-10 06:08:45 |       1 |         207 | myNet1        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+

The network below got deleted:

mysql> select * from usage_event where  resource_id=204 and type in ('NETWORK.CREATE','NETWORK.DELETE');
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
|  12 | NETWORK.CREATE |          2 | 2025-03-21 07:01:05 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  29 | NETWORK.CREATE |          2 | 2025-03-21 07:34:56 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  36 | NETWORK.CREATE |          2 | 2025-03-21 08:28:06 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  55 | NETWORK.CREATE |          2 | 2025-03-24 06:17:48 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  72 | NETWORK.CREATE |          2 | 2025-03-24 08:33:39 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
| 142 | NETWORK.DELETE |          2 | 2025-04-10 05:08:30 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+

Usage records for the same:

mysql> select account_id,usage_id,usage_type,start_date,end_date,count(*) as count from cloud_usage.cloud_usage where usage_id=204 and DATE(start_date)='2025-04-10' group by account_id,usage_id,usage_type,start_date,end_date having count >1;
+------------+----------+------------+---------------------+---------------------+-------+
| account_id | usage_id | usage_type | start_date          | end_date            | count |
+------------+----------+------------+---------------------+---------------------+-------+
|          2 |      204 |         30 | 2025-04-10 00:00:00 | 2025-04-10 00:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 00:30:00 | 2025-04-10 01:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 01:30:00 | 2025-04-10 02:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 02:30:00 | 2025-04-10 03:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 03:30:00 | 2025-04-10 04:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 04:30:00 | 2025-04-10 05:29:59 |     5 |
+------------+----------+------------+---------------------+---------------------+-------+
mysql> select * from cloud_usage.usage_networks where network_id=204;
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
| id | network_offering_id | zone_id | network_id | account_id | domain_id | state       | removed             | created             |
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
|  1 |                  10 |       1 |        204 |          2 |         1 | Destroy     | 2025-04-10 05:08:30 | 2025-03-21 07:01:05 |
|  2 |                  10 |       1 |        204 |          2 |         1 | Allocated   | NULL                | 2025-03-21 07:34:56 |
|  3 |                  10 |       1 |        204 |          2 |         1 | Implemented | NULL                | 2025-03-21 08:28:06 |
|  4 |                  10 |       1 |        204 |          2 |         1 | Allocated   | NULL                | 2025-03-24 06:17:48 |
|  5 |                  10 |       1 |        204 |          2 |         1 | Allocated   | NULL                | 2025-03-24 08:33:39 |
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+

Workaround

update configuration set value=‘-1’ where name=‘network.gc.interval’;

Note that it has a side effect that it will not release guest VLANs even when there are no running instances on the network, and the VRs will continue to run.

Existing duplicate usage events and usage records need to be fixed separately.

versions

4.19.1.2, 4.19.2

The steps to reproduce the bug

  1. Create an isolated network and deploy an instance
  2. Stop the (or all) instance and let the network GC shutdown the network.
  3. Start the instance.
  4. Observe the usage_event table entries cloud.
  5. After the usage processing, observe the tables usage_event, usage_networks and cloud_usage in cloud_usage database. List usage records using API or CMK to verify the duplicate records. …

What to do about it?

Review the issue, This looks like a bug that needs to be fixed soon.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 21 (20 by maintainers)

Most upvoted comments

  • NETWORK.CREATE is not published when the network is created. Instead, it is published only after the network is implemented, which does not match what Usage expects

This is not what I observed. I’ll have another look. I saw the NETWORK.CREATE being published on both create and implement. @rajujith, what did you see ?

I do not see any usage event when I create an isolated network, i.e when it isin the 'Allocated; state. I see the NETWORK.CREATE when the network is implemented. @winterhazel mentioned above the current behavior.

For isolated networks and VPC tiers, a usage event NETWORK.CREATE is created in the cloud.usage_event table whenever the network transitions from ‘Allocated’ to ‘Implemented’ state. A NETWORK.DELETE event is created when the network is deleted.

#9888 fixed this and some other issues on 4.20.1. I’ll backport the changes to 4.19.3, and work on a normalization script.

nice! @winterhazel , I think we can mark your PR #10755 as fixing this issue. Than ks a lot.

@DaanHoogland @sureshanaparti just making sure:

  • are you checking the table cloud.usage_event, and not cloud.event? A NETWORK.CREATE is inserted on cloud.event when the network is created (allocated), but not on cloud.usage_event

correct, NETWORK.CREATE is added on cloud.event table, and not on cloud.usage_event.

  • are you checking the table before the network is ever implemented? (while its state is Allocated)

correct

I tested this multiple times on both the 4.19.2.0 and 4.20.0.0 releases, but nothing is inserted on cloud.usage_event after creating the network.

yes, I observed the same

@winterhazel comments inline ^^^

My observations below:

Created Isolated Network (Network in Allocated state) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
Empty set (0.00 sec)

Deployed VM in Isolated Network created above (Network in Implemented state) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 365 | NETWORK.CREATE |          2 | 2025-04-24 09:22:51 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
1 row in set (0.00 sec)

Stopped the VM deployed above (Network back to Allocated state, after NetworkGarbageCollector triggered) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 365 | NETWORK.CREATE |          2 | 2025-04-24 09:22:51 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
1 row in set (0.00 sec)

Started the VM (Network in Implemented state) =>

mysql> SELECT * FROM cloud.event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| id   | uuid                                 | type           | state     | description                                              | user_id | account_id | domain_id | resource_id | resource_type | created             | level | start_id | parameters | archived | display |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
| 1129 | 4af06f76-9788-40a9-80a4-f10da4dafb0f | NETWORK.CREATE | Completed | Successfully completed creating network. Network Id: 206 |       2 |          2 |         1 |         206 | Network       | 2025-04-24 09:14:41 | INFO  |        0 | NULL       |        0 |       1 |
+------+--------------------------------------+----------------+-----------+----------------------------------------------------------+---------+------------+-----------+-------------+---------------+---------------------+-------+----------+------------+----------+---------+
1 row in set (0.00 sec)

mysql> SELECT * FROM cloud.usage_event WHERE type LIKE '%NETWORK%' AND resource_id = 206;
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 365 | NETWORK.CREATE |          2 | 2025-04-24 09:22:51 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
| 370 | NETWORK.CREATE |          2 | 2025-04-24 09:33:05 |       1 |         206 | testnw02      |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
2 rows in set (0.00 sec)

I think adding “NETWORK.ALLOCATION” and “NETWORK.DEALLOCATION” would be good (intuition) but not sure if that makes sense.

@DaanHoogland I think we can keep using NETWORK.UPDATE to indicate state change for networks, as is done currently. We just need to fix when the events are published.

@DaanHoogland ah, I think I now understood what you’re proposing. It seems good for me.

I’ll change the current code on NETWORK.CREATE event processing to not create new helper entries for a network when an active one already exists, and make the normalization keep the first created entry as not removed instead of the last one.

I’ll also take a look at the duplicated NETWORK.CREATE event creation.

@Pearl1594 today 🙂

Regarding the normalization script, though, I hope to look into it within this week.