webdataset: Unexpected Shuffling Behavior
Even with buffer size and initial are set to the size of the dataset, I am not seeing completely a completely shuffled dataset.
I have created a toy dataset with 10000 integers split into 100 shards.
A random batch looks something like:
tensor([[3251, 6459, 3253, 6470, 3256, 8078, 3257, 3255, 3259, 4836],
[ 151, 149, 153, 9749, 155, 139, 8186, 8174, 6542, 1741],
[9874, 1825, 8253, 6642, 8255, 6630, 8257, 8246, 8259, 218],
[1998, 3512, 3553, 3520, 3555, 9930, 3557, 3500, 3572, 8324]])
and many integers are close together (e.g. <= 20 apart).
Why is this? Is there any way to get around this? I am doing contrastive learning so diverse batches is very important.
The sample code I am using is
import torch
import os
import numpy as np
import webdataset as wds
import torch.distributed as dist
import torch.multiprocessing as mp
from webdataset.iterators import batched
def train(gpu, ngpus_per_node, dataloader):
for epoch in range(3):
if gpu == 0:
print(f'\n----------- Epoch = {epoch} ----------\n')
for i, x in enumerate(dataloader):
xarr = torch.from_numpy(np.array(x).astype('int')).to(gpu)
gathered_xarr = [
torch.zeros_like(xarr) for _ in range(ngpus_per_node)
]
dist.all_gather(gathered_xarr, xarr)
gathered_xarr = torch.cat(gathered_xarr)
if (i % 100) == 0:
if gpu == 0:
print(gathered_xarr.cpu())
def main_worker(gpu, ngpus_per_node):
dist.init_process_group(
backend="nccl",
init_method="tcp://127.0.0.1:6100",
world_size=ngpus_per_node,
rank=gpu,
)
input_shards = 'shard_{00..99}.tar'
size = 10000
batch_size = 10
workers = 4
num_batches = size // batch_size
dataset = wds.WebDataset(input_shards, length=num_batches, shardshuffle=True)
dataset = dataset.shuffle(size, initial=size)
dataset = (
dataset
.decode("pil")
.rename(text="txt")
.to_tuple("text")
)
dataset = dataset.batched(batch_size)
dataloader = wds.WebLoader(
dataset, batch_size=None, shuffle=False, num_workers=workers,
)
train(gpu, ngpus_per_node, dataloader)
def main():
torch.multiprocessing.set_start_method("spawn")
ngpus_per_node = 4
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node,))
if __name__ == "__main__":
main()
Any suggestions would be very appreciated.
Sorry I initially wrote this in another issue, which I did not realize was closed. https://github.com/webdataset/webdataset/issues/62
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 15 (3 by maintainers)
Why you are seeing the bunching.
You have 100 shards of 100 samples each. You also have 16 processes reading those shards. That means that you only have 6 shards per worker and that’s where all the samples for that worker come from during one epoch. Your shuffle buffer of 10000 samples will only ever contain 600 samples. Each of these six shards will sample from a range of 100 values, and you construct batches within the workers, so it’s not surprising that you should see such clusters of values. For large training sets, this clustering of values goes away. In practice, it isn’t even much of a problem for small training sets.
However, if you want to avoid it for small training sets, there are several things you can do:
Solution: Don’t split by shard and instead select by slice.
This does roughly what indexed datasets do and is about as efficient; it’s probably the most straightforward answer. I may add a couple of functions to WebDataset to make this easier.
Solution: Shuffle and rebatch after the loader.
This requires minimal changes and gives you shuffling between workers (but not between nodes).
Solution: Sample with replacement.
This is actually what I recommend. It’s simpler, works better for both small and large datasets, and has some statistical advantages as well. However, it is not common practice.
If you’re converting code that is uses an epoch-drive main loop, you can use
ResizedDatasetto impose “epochs” on this infinite data stream.In WebDataset, data is primarily shuffled at the shard level, and you get the equivalent of PyTorch’s samplers now: that’s the
shardlist=argument. Yourshardlistclass can sample the shards any way it likes. A node-splitting sampler and sampling with replacement are provided. I will be adding a weighted shard sampler in the future, since it’s needed for very large scale training.Separately, you can also perform shuffling and resampling at the per sample level, but those need to be streaming operations. Those operations are found as filters or operations on datasets; for example, dataset.rsample lets you subsample randomly. I will probably add a class frequency equalizing stream sampler at some point.
WebDataset is a massively parallel I/O library, and that necessitates that you deal with parallel (shards) and serial (samplers) aspects separately. The WebDataset library is further constrained by how PyTorch’s dataloader handles I/O processes.
(Note that if you have really complex I/O, shuffling, or augmentation needs, you might use something like nvlabs/tensorcom, which gives you complete freedom in shuffling samples between GPU nodes.)
It makes sense to add that for clarity, though the existing logic already effectively shuffles the buffer. The case doesn’t usually come up, since the shuffle buffer is usually smaller than the amount of data in the worker.
Your code appends the sample and then uses the same sample to shuffle it into the buffer, leading to two copies of the sample being present in the shuffle buffer.
The existing code appends an extra sample for each sample that is returned until the buffer is full. The code that does that is inserted by itself at the beginning of the loop and doesn’t affect the rest of the loop otherwise.