cloudstack: Duplicate NETWORK.CREATE usage events resulting in duplicate usage records
problem
For isolated networks and VPC tiers, a usage event NETWORK.CREATE is created in the cloud.usage_event table whenever the network transitions from ‘Allocated’ to ‘Implemented’ state. A NETWORK.DELETE event is created when the network is deleted.
I suspect there is a bug in the code or a bad design causing this issue. These events result in duplicate usage records for the network for the same period.
Either the NETWORK.CREATE should be created only when it is created, or if we need to capture usage for Allocated and Implemented networks separately, there could be another event.
The network below is not deleted.
mysql> select * from cloud.usage_event where resource_id=207 and type in ('NETWORK.CREATE','NETWORK.DELETE');
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 146 | NETWORK.CREATE | 2 | 2025-04-10 06:02:14 | 1 | 207 | myNet1 | 10 | NULL | NULL | NULL | 0 | NULL |
| 151 | NETWORK.CREATE | 2 | 2025-04-10 06:08:45 | 1 | 207 | myNet1 | 10 | NULL | NULL | NULL | 0 | NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
The network below got deleted:
mysql> select * from usage_event where resource_id=204 and type in ('NETWORK.CREATE','NETWORK.DELETE');
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 12 | NETWORK.CREATE | 2 | 2025-03-21 07:01:05 | 1 | 204 | Berlin | 10 | NULL | NULL | NULL | 0 | NULL |
| 29 | NETWORK.CREATE | 2 | 2025-03-21 07:34:56 | 1 | 204 | Berlin | 10 | NULL | NULL | NULL | 0 | NULL |
| 36 | NETWORK.CREATE | 2 | 2025-03-21 08:28:06 | 1 | 204 | Berlin | 10 | NULL | NULL | NULL | 0 | NULL |
| 55 | NETWORK.CREATE | 2 | 2025-03-24 06:17:48 | 1 | 204 | Berlin | 10 | NULL | NULL | NULL | 0 | NULL |
| 72 | NETWORK.CREATE | 2 | 2025-03-24 08:33:39 | 1 | 204 | Berlin | 10 | NULL | NULL | NULL | 0 | NULL |
| 142 | NETWORK.DELETE | 2 | 2025-04-10 05:08:30 | 1 | 204 | Berlin | 10 | NULL | NULL | NULL | 0 | NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
Usage records for the same:
mysql> select account_id,usage_id,usage_type,start_date,end_date,count(*) as count from cloud_usage.cloud_usage where usage_id=204 and DATE(start_date)='2025-04-10' group by account_id,usage_id,usage_type,start_date,end_date having count >1;
+------------+----------+------------+---------------------+---------------------+-------+
| account_id | usage_id | usage_type | start_date | end_date | count |
+------------+----------+------------+---------------------+---------------------+-------+
| 2 | 204 | 30 | 2025-04-10 00:00:00 | 2025-04-10 00:29:59 | 5 |
| 2 | 204 | 30 | 2025-04-10 00:30:00 | 2025-04-10 01:29:59 | 5 |
| 2 | 204 | 30 | 2025-04-10 01:30:00 | 2025-04-10 02:29:59 | 5 |
| 2 | 204 | 30 | 2025-04-10 02:30:00 | 2025-04-10 03:29:59 | 5 |
| 2 | 204 | 30 | 2025-04-10 03:30:00 | 2025-04-10 04:29:59 | 5 |
| 2 | 204 | 30 | 2025-04-10 04:30:00 | 2025-04-10 05:29:59 | 5 |
+------------+----------+------------+---------------------+---------------------+-------+
mysql> select * from cloud_usage.usage_networks where network_id=204;
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
| id | network_offering_id | zone_id | network_id | account_id | domain_id | state | removed | created |
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
| 1 | 10 | 1 | 204 | 2 | 1 | Destroy | 2025-04-10 05:08:30 | 2025-03-21 07:01:05 |
| 2 | 10 | 1 | 204 | 2 | 1 | Allocated | NULL | 2025-03-21 07:34:56 |
| 3 | 10 | 1 | 204 | 2 | 1 | Implemented | NULL | 2025-03-21 08:28:06 |
| 4 | 10 | 1 | 204 | 2 | 1 | Allocated | NULL | 2025-03-24 06:17:48 |
| 5 | 10 | 1 | 204 | 2 | 1 | Allocated | NULL | 2025-03-24 08:33:39 |
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
Workaround
update configuration set value=‘-1’ where name=‘network.gc.interval’;
Note that it has a side effect that it will not release guest VLANs even when there are no running instances on the network, and the VRs will continue to run.
Existing duplicate usage events and usage records need to be fixed separately.
versions
4.19.1.2, 4.19.2
The steps to reproduce the bug
- Create an isolated network and deploy an instance
- Stop the (or all) instance and let the network GC shutdown the network.
- Start the instance.
- Observe the usage_event table entries cloud.
- After the usage processing, observe the tables usage_event, usage_networks and cloud_usage in cloud_usage database. List usage records using API or CMK to verify the duplicate records. …
What to do about it?
Review the issue, This looks like a bug that needs to be fixed soon.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 21 (20 by maintainers)
I do not see any usage event when I create an isolated network, i.e when it isin the 'Allocated; state. I see the
NETWORK.CREATEwhen the network is implemented. @winterhazel mentioned above the current behavior.#9888 fixed this and some other issues on 4.20.1. I’ll backport the changes to 4.19.3, and work on a normalization script.
nice! @winterhazel , I think we can mark your PR #10755 as fixing this issue. Than ks a lot.
correct, NETWORK.CREATE is added on
cloud.eventtable, and not oncloud.usage_event.correct
yes, I observed the same
@winterhazel comments inline ^^^
My observations below:
Created Isolated Network (Network in Allocated state) =>
Deployed VM in Isolated Network created above (Network in Implemented state) =>
Stopped the VM deployed above (Network back to Allocated state, after NetworkGarbageCollector triggered) =>
Started the VM (Network in Implemented state) =>
@DaanHoogland I think we can keep using
NETWORK.UPDATEto indicate state change for networks, as is done currently. We just need to fix when the events are published.@DaanHoogland ah, I think I now understood what you’re proposing. It seems good for me.
I’ll change the current code on NETWORK.CREATE event processing to not create new helper entries for a network when an active one already exists, and make the normalization keep the first created entry as not removed instead of the last one.
I’ll also take a look at the duplicated NETWORK.CREATE event creation.
@Pearl1594 today 🙂
Regarding the normalization script, though, I hope to look into it within this week.