gitoxide: `panic!("encountered unsupported command code: 0")` when enumerating repository objects

What I’m seeing

Using Nosey Parker to scan a particular obscure repository from GitHub, I’m seeing a panic in the data::delta::apply function in gix-pack-0.59.1.

I’m seeing a crash when I run noseyparker scan -d test.np -j1 -vvv deqp.git, with a local clone of the https://github.com/elongbug/deqp repository. Here’s a debug stack trace from that:

The application panicked (crashed).
Message:  encountered unsupported command code: 0
Location: /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-pack-0.59.1/src/data/delta.rs:60
[..,]
Full Log
The application panicked (crashed).
Message:  encountered unsupported command code: 0
Location: /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-pack-0.59.1/src/data/delta.rs:60

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
⠒ Scanning content 1.55 GiB [00:01:14]                                                                                                                                                                                                                                                                                                                                      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                              ⋮ 9 frames hidden ⋮
10: gix_pack::data::delta::apply::h990db6d667116761
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-pack-0.59.1/src/data/delta.rs:60
      58 │                     .expect("delta copy from base: byte slices must match");
      59 │             }
      60 >             0 => panic!("encountered unsupported command code: 0"),
      61 │             size => {
      62 │                 std::io::Write::write(&mut target, &data[i..i + *size as usize])
11: gix_pack::data::file::decode::entry::<impl gix_pack::data::File>::resolve_deltas::h14b6fc9b4dbdff00
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-pack-0.59.1/src/data/file/decode/entry.rs:377
     375 │                 last_result_size = Some(result_size);
     376 │             }
     377 >             delta::apply(&source_buf[..base_size], &mut target_buf[..result_size], data);
     378 │             // use the target as source for the next delta
     379 │             std::mem::swap(&mut source_buf, &mut target_buf);
12: gix_pack::data::file::decode::entry::<impl gix_pack::data::File>::decode_entry::h485eb73abedf8206
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-pack-0.59.1/src/data/file/decode/entry.rs:186
     184 │                     })
     185 │             }
     186 >             OfsDelta { .. } | RefDelta { .. } => self.resolve_deltas(entry, resolve, inflate, out, delta_cache),
     187 │         }
     188 │     }
13: gix_odb::store_impls::dynamic::find::<impl gix_odb::store_impls::dynamic::Handle<S>>::try_find_cached_inner::h68d1ee577f15b8ae
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-odb-0.69.1/src/store_impls/dynamic/find.rs:151
     149 │                         let entry = pack.entry(pack_offset)?;
     150 │                         let header_size = entry.header_size();
     151 >                         let res = pack.decode_entry(
     152 │                             entry,
     153 │                             buffer,
14: gix_odb::store_impls::dynamic::find::<impl gix_pack::find_traits::Find for gix_odb::store_impls::dynamic::Handle<S>>::try_find_cached::h356f98a75f4393f2
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-odb-0.69.1/src/store_impls/dynamic/find.rs:356
     354 │         let mut snapshot = self.snapshot.borrow_mut();
     355 │         let mut inflate = self.inflate.borrow_mut();
     356 >         self.try_find_cached_inner(id, buffer, &mut inflate, pack_cache, &mut snapshot, None)
     357 │             .map_err(|err| Box::new(err) as _)
     358 │     }
15: gix_odb::cache::impls::<impl gix_pack::find_traits::Find for gix_odb::Cache<S>>::try_find_cached::h22b8bd39957c46de
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-odb-0.69.1/src/cache.rs:224
     222 │                 }
     223 │             }
     224 >             let possibly_obj = self.inner.try_find_cached(id.as_ref(), buffer, pack_cache)?;
     225 │             if let (Some(mut obj_cache), Some((obj, _location))) =
     226 │                 (self.object_cache.as_ref().map(RefCell::borrow_mut), &possibly_obj)
16: gix_odb::cache::impls::<impl gix_pack::find_traits::Find for gix_odb::Cache<S>>::try_find::hc3f634d96c8f0040
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-odb-0.69.1/src/cache.rs:208
     206 │         ) -> Result<Option<(Data<'a>, Option<Location>)>, gix_object::find::Error> {
     207 │             match self.pack_cache.as_ref().map(RefCell::borrow_mut) {
     208 >                 Some(mut pack_cache) => self.try_find_cached(id, buffer, pack_cache.deref_mut()),
     209 │                 None => self.try_find_cached(id, buffer, &mut gix_pack::cache::Never),
     210 │             }
17: gix_odb::cache::impls::<impl gix_object::traits::find::Find for gix_odb::Cache<S>>::try_find::h505b462e9cba2ee2
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-odb-0.69.1/src/cache.rs:163
     161 │     {
     162 │         fn try_find<'a>(&self, id: &oid, buffer: &'a mut Vec<u8>) -> Result<Option<Data<'a>>, gix_object::find::Error> {
     163 >             gix_pack::Find::try_find(self, id, buffer).map(|t| t.map(|t| t.0))
     164 │         }
     165 │     }
18: <gix_odb::memory::Proxy<T> as gix_object::traits::find::Find>::try_find::h40b6c57e98500767
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-odb-0.69.1/src/memory.rs:159
     157 │             }
     158 │         }
     159 >         self.inner.try_find(id, buffer)
     160 │     }
     161 │ }
19: gix_object::traits::find::ext::FindExt::find::h0ce666a910c19e76
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-object-0.49.1/src/traits/find.rs:234
     232 │             buffer: &'a mut Vec<u8>,
     233 │         ) -> Result<crate::Data<'a>, find::existing::Error> {
     234 >             self.try_find(id, buffer)
     235 │                 .map_err(find::existing::Error::Find)?
     236 │                 .ok_or_else(|| find::existing::Error::NotFound { oid: id.to_owned() })
20: gix::repository::object::<impl gix::types::Repository>::find_object::hb284647272300f9e
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/gix-0.72.1/src/repository/object.rs:53
      51 │         }
      52 │         let mut buf = self.free_buf();
      53 >         let kind = self.objects.find(&id, &mut buf)?.kind;
      54 │         Ok(Object::from_data(id, kind, buf, self))
      55 │     }
21: <noseyparker_cli::cmd_scan::GitRepoResultIter as rayon::iter::ParallelIterator>::drive_unindexed::{{closure}}::{{closure}}::h2e81d9987823c008
    at /home/blarsen/projects/noseyparkerplusplus/crates/noseyparker-cli/src/cmd_scan.rs:199
     197 │
     198 │                     let blob = || -> Result<Blob> {
     199 >                         let mut blob = repo.find_object(blob_id)?.try_into_blob()?;
     200 │                         let data = std::mem::take(&mut blob.data); // avoid a copy
     201 │                         Ok(Blob::new(BlobId::from(&blob_id), data))
22: <noseyparker_cli::cmd_scan::GitRepoResultIter as rayon::iter::ParallelIterator>::drive_unindexed::{{closure}}::hcfc65172c38b96e7
    at /home/blarsen/projects/noseyparkerplusplus/crates/noseyparker-cli/src/cmd_scan.rs:198
     196 │                     let blob_id = md.blob_oid;
     197 │
     198 >                     let blob = || -> Result<Blob> {
     199 │                         let mut blob = repo.find_object(blob_id)?.try_into_blob()?;
     200 │                         let data = std::mem::take(&mut blob.data); // avoid a copy
23: core::ops::function::impls::<impl core::ops::function::Fn<A> for &F>::call::h04920647f8c3f68a
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:262
     260 │     {
     261 │         extern "rust-call" fn call(&self, args: A) -> F::Output {
     262 >             (**self).call(args)
     263 │         }
     264 │     }
24: <rayon::iter::map_with::MapWithFolder<C,U,F> as rayon::iter::plumbing::Folder<T>>::consume_iter::with::{{closure}}::hce48ffd6a09956f4
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map_with.rs:317
     315 │             map_op: impl Fn(&mut U, T) -> R + 'f,
     316 │         ) -> impl FnMut(T) -> R + 'f {
     317 >             move |x| map_op(item, x)
     318 │         }
     319 │
25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once::h8cf6aabb49d00555
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:305
     303 │         type Output = F::Output;
     304 │         extern "rust-call" fn call_once(self, args: A) -> F::Output {
     305 >             (*self).call_mut(args)
     306 │         }
     307 │     }
26: core::option::Option<T>::map::he9fed2335951ae0c
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:1113
    1111 │     {
    1112 │         match self {
    1113 >             Some(x) => Some(f(x)),
    1114 │             None => None,
    1115 │         }
27: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::ha3fb456556810b5c
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:107
     105 │     #[inline]
     106 │     fn next(&mut self) -> Option<B> {
     107 >         self.iter.next().map(&mut self.f)
     108 │     }
     109 │
28: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next::ha7f60e1c0a324d21
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:107
     105 │     #[inline]
     106 │     fn next(&mut self) -> Option<B> {
     107 >         self.iter.next().map(&mut self.f)
     108 │     }
     109 │
29: rayon::iter::plumbing::Folder::consume_iter::hf8e09de90f9ea415
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:177
     175 │         I: IntoIterator<Item = Item>,
     176 │     {
     177 >         for item in iter {
     178 │             self = self.consume(item);
     179 │             if self.full() {
30: <rayon::iter::map_with::MapWithFolder<C,U,F> as rayon::iter::plumbing::Folder<T>>::consume_iter::hcb8661039dd1a0f9
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map_with.rs:322
     320 │         {
     321 │             let mapped_iter = iter.into_iter().map(with(&mut self.item, self.map_op));
     322 >             self.base = self.base.consume_iter(mapped_iter);
     323 │         }
     324 │         self
31: <rayon::iter::map_with::MapWithFolder<C,U,F> as rayon::iter::plumbing::Folder<T>>::consume_iter::hfd6b6a675578c81f
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map_with.rs:322
     320 │         {
     321 │             let mapped_iter = iter.into_iter().map(with(&mut self.item, self.map_op));
     322 >             self.base = self.base.consume_iter(mapped_iter);
     323 │         }
     324 │         self
32: rayon::iter::plumbing::Producer::fold_with::ha6e5cd69b027c08d
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:109
     107 │         F: Folder<Self::Item>,
     108 │     {
     109 >         folder.consume_iter(self.into_iter())
     110 │     }
     111 │ }
33: <rayon::iter::len::MinLenProducer<P> as rayon::iter::plumbing::Producer>::fold_with::h8428a8c1bf7a486c
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/len.rs:134
     132 │         F: Folder<Self::Item>,
     133 │     {
     134 >         self.base.fold_with(folder)
     135 │     }
     136 │ }
34: rayon::iter::plumbing::bridge_producer_consumer::helper::h372f7337d5f6da17
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:437
     435 │             reducer.reduce(left_result, right_result)
     436 │         } else {
     437 >             producer.fold_with(consumer.into_folder()).complete()
     438 │         }
     439 │     }
35: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}::h9aad568542e81dd4
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:426
     424 │                 },
     425 │                 |context| {
     426 >                     helper(
     427 │                         len - mid,
     428 │                         context.migrated(),
36: rayon_core::join::join_context::call_b::{{closure}}::h2d4a355590d1944e
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:129
     127 │     #[inline]
     128 │     fn call_b<R>(f: impl FnOnce(FnContext) -> R) -> impl FnOnce(bool) -> R {
     129 >         move |migrated| f(FnContext::new(migrated))
     130 │     }
     131 │
37: rayon_core::job::StackJob<L,F,R>::run_inline::h4de831323c4e97e7
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/job.rs:102
     100 │
     101 │     pub(super) unsafe fn run_inline(self, stolen: bool) -> R {
     102 >         self.func.into_inner().unwrap()(stolen)
     103 │     }
     104 │
38: rayon_core::join::join_context::{{closure}}::h5822005cf64c68e4
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:159
     157 │                     //
     158 │                     // Note that this could panic, but it's ok if we unwind here.
     159 >                     let result_b = job_b.run_inline(injected);
     160 │                     return (result_a, result_b);
     161 │                 } else {
39: rayon_core::registry::in_worker::hb110d42b82230fbb
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:951
     949 │             // current thread, so we know the data structure won't be
     950 │             // invalidated until we return.
     951 >             op(&*owner_thread, false)
     952 │         } else {
     953 │             global_registry().in_worker(op)
40: rayon_core::join::join_context::hc0311f36f2a71c18
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:132
     130 │     }
     131 │
     132 >     registry::in_worker(|worker_thread, injected| unsafe {
     133 │         // Create virtual wrapper for task b; this all has to be
     134 │         // done here so that the stack frame can keep it all live
41: rayon::iter::plumbing::bridge_producer_consumer::helper::h372f7337d5f6da17
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:415
     413 │             let (left_producer, right_producer) = producer.split_at(mid);
     414 │             let (left_consumer, right_consumer, reducer) = consumer.split_at(mid);
     415 >             let (left_result, right_result) = join_context(
     416 │                 |context| {
     417 │                     helper(
42: rayon::iter::plumbing::bridge_producer_consumer::h8f825ccc0043937b
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:396
     394 │ {
     395 │     let splitter = LengthSplitter::new(producer.min_len(), producer.max_len(), len);
     396 >     return helper(len, false, splitter, producer, consumer);
     397 │
     398 │     fn helper<P, C>(
43: <rayon::iter::plumbing::bridge::Callback<C> as rayon::iter::plumbing::ProducerCallback<I>>::callback::h1abc936b73e245c6
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:372
     370 │             P: Producer<Item = I>,
     371 │         {
     372 >             bridge_producer_consumer(self.len, producer, self.consumer)
     373 │         }
     374 │     }
44: <<rayon::iter::len::MinLen<I> as rayon::iter::IndexedParallelIterator>::with_producer::Callback<CB> as rayon::iter::plumbing::ProducerCallback<T>>::callback::h28a6ba45beb8316b
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/len.rs:83
      81 │                     min: self.min,
      82 │                 };
      83 >                 self.callback.callback(producer)
      84 │             }
      85 │         }
45: <rayon::vec::Drain<T> as rayon::iter::IndexedParallelIterator>::with_producer::h0cfbf39e5e74bcbf
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/vec.rs:147
     145 │
     146 │             // The producer will move or drop each item from the drained range.
     147 >             callback.callback(producer)
     148 │         }
     149 │     }
46: <rayon::vec::IntoIter<T> as rayon::iter::IndexedParallelIterator>::with_producer::hc0ffe17d25b11b5f
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/vec.rs:83
      81 │     {
      82 │         // Drain every item, and then the vector only needs to free its buffer.
      83 >         self.vec.par_drain(..).with_producer(callback)
      84 │     }
      85 │ }
47: <rayon::iter::len::MinLen<I> as rayon::iter::IndexedParallelIterator>::with_producer::h553f26dda48af215
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/len.rs:60
      58 │         CB: ProducerCallback<Self::Item>,
      59 │     {
      60 >         return self.base.with_producer(Callback {
      61 │             callback,
      62 │             min: self.min,
48: rayon::iter::plumbing::bridge::h773705a9be8800aa
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:356
     354 │ {
     355 │     let len = par_iter.len();
     356 >     return par_iter.with_producer(Callback { len, consumer });
     357 │
     358 │     struct Callback<C> {
49: <rayon::iter::len::MinLen<I> as rayon::iter::ParallelIterator>::drive_unindexed::hffeef17194c9017c
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/len.rs:36
      34 │         C: UnindexedConsumer<Self::Item>,
      35 │     {
      36 >         bridge(self, consumer)
      37 │     }
      38 │
50: <rayon::iter::map_with::MapInit<I,INIT,F> as rayon::iter::ParallelIterator>::drive_unindexed::h94a2b6f49cb5515b
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/map_with.rs:382
     380 │     {
     381 │         let consumer1 = MapInitConsumer::new(consumer, &self.init, &self.map_op);
     382 >         self.base.drive_unindexed(consumer1)
     383 │     }
     384 │
51: <noseyparker_cli::cmd_scan::GitRepoResultIter as rayon::iter::ParallelIterator>::drive_unindexed::h8390c4732055d594
    at /home/blarsen/projects/noseyparkerplusplus/crates/noseyparker-cli/src/cmd_scan.rs:176
     174 │         let repo = self.inner.repository.into_sync();
     175 │         let repo_path = Arc::new(self.inner.path.clone());
     176 >         self.inner
     177 │             .blobs
     178 │             .into_par_iter()
52: <noseyparker_cli::cmd_scan::FoundInputIter as rayon::iter::ParallelIterator>::drive_unindexed::h49dd338b93298ca7
    at /home/blarsen/projects/noseyparkerplusplus/crates/noseyparker-cli/src/cmd_scan.rs:301
     299 │         match self {
     300 │             FoundInputIter::File(i) => i.drive_unindexed(consumer),
     301 >             FoundInputIter::GitRepo(i) => i.drive_unindexed(consumer),
     302 │             FoundInputIter::EnumeratorFile(i) => i.drive_unindexed(consumer),
     303 │         }
53: <rayon::iter::flatten::FlattenFolder<C,<C as rayon::iter::plumbing::Consumer<<T as rayon::iter::IntoParallelIterator>::Item>>::Result> as rayon::iter::plumbing::Folder<T>>::consume::h3bcfa490e60c0810
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/flatten.rs:114
     112 │         let par_iter = item.into_par_iter();
     113 │         let consumer = self.base.split_off_left();
     114 >         let result = par_iter.drive_unindexed(consumer);
     115 │
     116 │         let previous = match self.previous {
54: <rayon::iter::filter_map::FilterMapFolder<C,P> as rayon::iter::plumbing::Folder<T>>::consume::hc8806136aef5923b
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/filter_map.rs:124
     122 │         let filter_op = self.filter_op;
     123 │         if let Some(mapped_item) = filter_op(item) {
     124 >             let base = self.base.consume(mapped_item);
     125 │             FilterMapFolder { base, filter_op }
     126 │         } else {
55: <&rayon::iter::par_bridge::IterParallelProducer<Iter> as rayon::iter::plumbing::UnindexedProducer>::fold_with::h6426e9e9ba9e3af3
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/par_bridge.rs:145
     143 │                 if let Some(it) = iter.next() {
     144 │                     drop(iter);
     145 >                     folder = folder.consume(it);
     146 │                     if folder.full() {
     147 │                         return folder;
56: rayon::iter::plumbing::bridge_unindexed_producer_consumer::h2b035d48d0c4f326
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:478
     476 │                 reducer.reduce(left_result, right_result)
     477 │             }
     478 >             (producer, None) => producer.fold_with(consumer.into_folder()).complete(),
     479 │         }
     480 │     } else {
57: rayon::iter::plumbing::bridge_unindexed_producer_consumer::{{closure}}::hb8fb6da870149c5b
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-1.10.0/src/iter/plumbing/mod.rs:473
     471 │                 let bridge = bridge_unindexed_producer_consumer;
     472 │                 let (left_result, right_result) = join_context(
     473 >                     |context| bridge(context.migrated(), splitter, left_producer, left_consumer),
     474 │                     |context| bridge(context.migrated(), splitter, right_producer, right_consumer),
     475 │                 );
58: rayon_core::join::join_context::call_a::{{closure}}::h66b269fbf4a554a7
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:124
     122 │     #[inline]
     123 │     fn call_a<R>(f: impl FnOnce(FnContext) -> R, injected: bool) -> impl FnOnce() -> R {
     124 >         move || f(FnContext::new(injected))
     125 │     }
     126 │
59: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hca943e5c7c818128
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272
     270 │     #[inline]
     271 │     extern "rust-call" fn call_once(self, _args: ()) -> R {
     272 >         (self.0)()
     273 │     }
     274 │ }
60: std::panicking::try::do_call::h3c325e65ad71efc0
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:557
     555 │             let data = &mut (*data);
     556 │             let f = ManuallyDrop::take(&mut data.f);
     557 >             data.r = ManuallyDrop::new(f());
     558 │         }
     559 │     }
61: __rust_try
    at <unknown source file>
62: std::panicking::try::h58b0fcdeaabdd12f
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:520
     518 │     // See their safety preconditions for more information
     519 │     unsafe {
     520 >         return if intrinsics::catch_unwind(do_call::<F, R>, data_ptr, do_catch::<F, R>) == 0 {
     521 │             Ok(ManuallyDrop::into_inner(data.r))
     522 │         } else {
63: std::panic::catch_unwind::hcd77458f398ee48c
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:358
     356 │ #[stable(feature = "catch_unwind", since = "1.9.0")]
     357 │ pub fn catch_unwind<F: FnOnce() -> R + UnwindSafe, R>(f: F) -> Result<R> {
     358 >     unsafe { panicking::r#try(f) }
     359 │ }
     360 │
64: rayon_core::unwind::halt_unwinding::h4c6f1ef13155c7d7
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/unwind.rs:17
      15 │     F: FnOnce() -> R,
      16 │ {
      17 >     panic::catch_unwind(AssertUnwindSafe(func))
      18 │ }
      19 │
65: rayon_core::join::join_context::{{closure}}::haeadd60ea8035103
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/join/mod.rs:142
     140 │
     141 │         // Execute task a; hopefully b gets stolen in the meantime.
     142 >         let status_a = unwind::halt_unwinding(call_a(oper_a, injected));
     143 │         let result_a = match status_a {
     144 │             Ok(v) => v,
66: rayon_core::registry::Registry::in_worker_cold::{{closure}}::{{closure}}::h4ffcd99f706ffada
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:522
     520 │                     let worker_thread = WorkerThread::current();
     521 │                     assert!(injected && !worker_thread.is_null());
     522 >                     op(&*worker_thread, true)
     523 │                 },
     524 │                 LatchRef::new(l),
67: rayon_core::job::JobResult<T>::call::{{closure}}::h2d8e738783f1af7f
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/job.rs:218
     216 │ impl<T> JobResult<T> {
     217 │     fn call(func: impl FnOnce(bool) -> T) -> Self {
     218 >         match unwind::halt_unwinding(|| func(true)) {
     219 │             Ok(x) => JobResult::Ok(x),
     220 │             Err(x) => JobResult::Panic(x),
68: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h62dbad7b756c450e
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272
     270 │     #[inline]
     271 │     extern "rust-call" fn call_once(self, _args: ()) -> R {
     272 >         (self.0)()
     273 │     }
     274 │ }
69: std::panicking::try::do_call::h10d264f94e4540fe
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:557
     555 │             let data = &mut (*data);
     556 │             let f = ManuallyDrop::take(&mut data.f);
     557 >             data.r = ManuallyDrop::new(f());
     558 │         }
     559 │     }
70: __rust_try
    at <unknown source file>
71: std::panicking::try::h6b62f24fc180418c
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:520
     518 │     // See their safety preconditions for more information
     519 │     unsafe {
     520 >         return if intrinsics::catch_unwind(do_call::<F, R>, data_ptr, do_catch::<F, R>) == 0 {
     521 │             Ok(ManuallyDrop::into_inner(data.r))
     522 │         } else {
72: std::panic::catch_unwind::hca78f5ba028ec6c3
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:358
     356 │ #[stable(feature = "catch_unwind", since = "1.9.0")]
     357 │ pub fn catch_unwind<F: FnOnce() -> R + UnwindSafe, R>(f: F) -> Result<R> {
     358 >     unsafe { panicking::r#try(f) }
     359 │ }
     360 │
73: rayon_core::unwind::halt_unwinding::hf2452592a011c996
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/unwind.rs:17
      15 │     F: FnOnce() -> R,
      16 │ {
      17 >     panic::catch_unwind(AssertUnwindSafe(func))
      18 │ }
      19 │
74: rayon_core::job::JobResult<T>::call::h6f3c0cd413514835
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/job.rs:218
     216 │ impl<T> JobResult<T> {
     217 │     fn call(func: impl FnOnce(bool) -> T) -> Self {
     218 >         match unwind::halt_unwinding(|| func(true)) {
     219 │             Ok(x) => JobResult::Ok(x),
     220 │             Err(x) => JobResult::Panic(x),
75: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::h684a52e81cc0409d
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/job.rs:120
     118 │         let abort = unwind::AbortIfPanic;
     119 │         let func = (*this.func.get()).take().unwrap();
     120 >         (*this.result.get()) = JobResult::call(func);
     121 │         Latch::set(&this.latch);
     122 │         mem::forget(abort);
76: rayon_core::job::JobRef::execute::ha601b4ad0234eefa
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/job.rs:64
      62 │     #[inline]
      63 │     pub(super) unsafe fn execute(self) {
      64 >         (self.execute_fn)(self.pointer)
      65 │     }
      66 │ }
77: rayon_core::registry::WorkerThread::execute::h303b24019f5d76f5
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:860
     858 │     #[inline]
     859 │     pub(super) unsafe fn execute(&self, job: JobRef) {
     860 >         job.execute();
     861 │     }
     862 │
78: rayon_core::registry::WorkerThread::wait_until_cold::h81dc9ddf37ae1fd2
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:794
     792 │                 if let Some(job) = self.find_work() {
     793 │                     self.registry.sleep.work_found();
     794 >                     self.execute(job);
     795 │                     // The job might have injected local work, so go back to the outer loop.
     796 │                     continue 'outer;
79: rayon_core::registry::WorkerThread::wait_until::h542fc7d746a67754
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:769
     767 │         let latch = latch.as_core_latch();
     768 │         if !latch.probe() {
     769 >             self.wait_until_cold(latch);
     770 │         }
     771 │     }
80: rayon_core::registry::WorkerThread::wait_until_out_of_work::h31529aeb7dda607b
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:818
     816 │         let index = self.index;
     817 │
     818 >         self.wait_until(&registry.thread_infos[index].terminate);
     819 │
     820 │         // Should not be any work left in our queue.
81: rayon_core::registry::main_loop::h71f1e250e38a88e2
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:923
     921 │     }
     922 │
     923 >     worker_thread.wait_until_out_of_work();
     924 │
     925 │     // Normal termination, do not abort.
82: rayon_core::registry::ThreadBuilder::run::h0d5c633bf4b630cc
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:53
      51 │     /// thread pool is dropped.
      52 │     pub fn run(self) {
      53 >         unsafe { main_loop(self) }
      54 │     }
      55 │ }
83: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}::h6e8a8fe4e34fd376
    at /home/blarsen/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.12.1/src/registry.rs:98
      96 │             b = b.stack_size(stack_size);
      97 │         }
      98 >         b.spawn(|| thread.run())?;
      99 │         Ok(())
     100 │     }
84: std::sys::backtrace::__rust_begin_short_backtrace::ha240e7ef2440fa50
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/backtrace.rs:154
     152 │     F: FnOnce() -> T,
     153 │ {
     154 >     let result = f();
     155 │
     156 │     // prevent this frame from being tail-call optimised away
85: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h638b8c0b1d37c2e1
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:561
     559 │             let try_result = panic::catch_unwind(panic::AssertUnwindSafe(|| {
     560 │                 crate::sys::backtrace::__rust_begin_short_backtrace(|| hooks.run());
     561 >                 crate::sys::backtrace::__rust_begin_short_backtrace(f)
     562 │             }));
     563 │             // SAFETY: `their_packet` as been built just above and moved by the
86: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h2f63e73aa378b5e9
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:272
     270 │     #[inline]
     271 │     extern "rust-call" fn call_once(self, _args: ()) -> R {
     272 >         (self.0)()
     273 │     }
     274 │ }
87: std::panicking::try::do_call::h6d6d8a3d23323aad
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:557
     555 │             let data = &mut (*data);
     556 │             let f = ManuallyDrop::take(&mut data.f);
     557 >             data.r = ManuallyDrop::new(f());
     558 │         }
     559 │     }
88: __rust_try
    at <unknown source file>
89: std::panicking::try::hec4f5105d251e24f
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:520
     518 │     // See their safety preconditions for more information
     519 │     unsafe {
     520 >         return if intrinsics::catch_unwind(do_call::<F, R>, data_ptr, do_catch::<F, R>) == 0 {
     521 │             Ok(ManuallyDrop::into_inner(data.r))
     522 │         } else {
90: std::panic::catch_unwind::h54574b824fc683a3
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:358
     356 │ #[stable(feature = "catch_unwind", since = "1.9.0")]
     357 │ pub fn catch_unwind<F: FnOnce() -> R + UnwindSafe, R>(f: F) -> Result<R> {
     358 >     unsafe { panicking::r#try(f) }
     359 │ }
     360 │
91: std::thread::Builder::spawn_unchecked_::{{closure}}::h5561b49cc0050c06
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:559
     557 │
     558 │             let f = f.into_inner();
     559 >             let try_result = panic::catch_unwind(panic::AssertUnwindSafe(|| {
     560 │                 crate::sys::backtrace::__rust_begin_short_backtrace(|| hooks.run());
     561 │                 crate::sys::backtrace::__rust_begin_short_backtrace(f)
92: core::ops::function::FnOnce::call_once{{vtable.shim}}::h6cc0d940ae51894e
    at /home/blarsen/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
     248 │     /// Performs the call operation.
     249 │     #[unstable(feature = "fn_traits", issue = "29625")]
     250 >     extern "rust-call" fn call_once(self, args: Args) -> Self::Output;
     251 │ }
     252 │
93: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h9578f6ea1d4e1c4b
    at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1972
94: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf4a2f438d8019348
    at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/alloc/src/boxed.rs:1972
95: std::sys::pal::unix::thread::Thread::new::thread_start::h14f1eb868ff90fc9
    at /rustc/9fc6b43126469e3858e2fe86cafb4f0fd5068869/library/std/src/sys/pal/unix/thread.rs:105
96: start_thread
    at ./nptl/pthread_create.c:442
97: __GI___clone3
    at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

What I expect

gitoxide code should not crash, even if provided malformed input

The crashing gitoxide code

https://github.com/GitoxideLabs/gitoxide/blob/08e7777454bdce466363321fce7b168d425bb833/gix-pack/src/data/delta.rs#L18-L70

In particular, it’s the panic!("encountered unsupported command code: 0") code that I’m seeing crash. However, there is also an expect call just above that which also could be triggered by ill-formed user-controllable input.

FYI, looking at the git blame for that code, it appears to be pretty old, much of it from 2021.

Reproducing this

The input repository to Nosey Parker (that’s triggering the gitoxide crash when enumerating inputs) is elongbug/deqp, cloned with git clone --bare https://github.com/elongbug/deqp.git. The last changes there are from 10 years ago.

The packfile contents that I see:

% shasum deqp.git/objects/pack/*
77809f4a1783bc011dd9f49c0ee18fe45d0cb792  deqp.git/objects/pack/pack-d57c91f4836f96358552c59966566d6a8f522440.idx
8325905c1a6b262deaadf6c6a1a9ede804859d00  deqp.git/objects/pack/pack-d57c91f4836f96358552c59966566d6a8f522440.pack

I see the crash deterministically, with or without multithreading in Nosey Parker, but only on an x86_64 machine, and not on Apple Silicon-based machines (i.e., ARM).

Other notes

I see the gix_pack::data::delta::apply function used only in 3 places in gitoxide, and all 3 locations return Result values already. Could gix_pack::data::delta::apply be modified to return an error instead of panicking on malformed input? Would you like a PR for this? It might not be a semver-breaking change, as that function looks like it’s only exposed pub(crate) from the gix_pack::data module.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 40 (33 by maintainers)

Most upvoted comments

(Tangent)

Some kind of image data, perhaps this?

No, probably not that format, as that PKM image format expects a 780-byte header, and we only have 24 bytes. I’m not sure what the data’s supposed to be.

I was reading along and wondering what the file format might be. After some searching I stumbled over https://stackoverflow.com/a/35881621/7847252 which appears to match the header.

Quick & Dirty parsing implementation
/*
Based on https://stackoverflow.com/a/35881621/7847252

The header states it is a PKM image format.

The format is:

4 byte magic number(504b 4d20): "PKM "
2 byte version "10"
2 byte data type: 0 (ETC1_RGB_NO_MIPMAPS)
16 bit big endian extended width
16 bit big endian extended height
16 bit big endian original width
16 bit big endian original height
data, 64 bit big endian words.
From Game Developers SE Question

PKM format for ETC1 texture (All OGLES 2.0 devices compatibles)
PKM is useful for packaging ETC1-compressed images, but like with PNG, it doesn't support actual texture features.

*/

fn main() {
    let data = "50 4b 4d 20 31 30 00 00 00 04 00 04 00 01 00 01 40 58 80 b6 00 00 00 00";
    let data = data
        .split(' ')
        .map(|b| u8::from_str_radix(b, 16))
        .collect::<Result<Vec<_>, _>>()
        .unwrap();

    if let &[
        b'P',
        b'K',
        b'M',
        b' ',
        version_h,
        version_l,
        type_h,
        type_l,
        width_ext_h,
        width_ext_l,
        height_ext_h,
        height_ext_l,
        width_orig_h,
        width_orig_l,
        height_orig_h,
        height_orig_l,
        ref data @ ..,
    ] = data.as_slice()
    {
        let version = format!("{}{}", char::from(version_h), char::from(version_l));
        let kind = u16::from_be_bytes([type_h, type_l]);
        let width_ext = u16::from_be_bytes([width_ext_h, width_ext_l]);
        let height_ext = u16::from_be_bytes([height_ext_h, height_ext_l]);
        let width_orig = u16::from_be_bytes([width_orig_h, width_orig_l]);
        let height_orig = u16::from_be_bytes([height_orig_h, height_orig_l]);

        let words = data
            .chunks(8)
            .map(|entry| u64::from_be_bytes(*<&[u8; 8]>::try_from(entry).unwrap()))
            .collect::<Vec<_>>();

        dbg!(
            version,
            kind,
            width_ext,
            height_ext,
            width_orig,
            height_orig,
            words
        );
    }
}
[src\main.rs:63:9] version = "10"
[src\main.rs:63:9] kind = 0
[src\main.rs:63:9] width_ext = 4
[src\main.rs:63:9] height_ext = 4
[src\main.rs:63:9] width_orig = 1
[src\main.rs:63:9] height_orig = 1
[src\main.rs:63:9] words = [
    4636597335550328832,
]

Furthe seaching brought me to https://learn.microsoft.com/en-us/dotnet/api/android.opengl.etc1?view=net-android-35.0 which links https://registry.khronos.org/OpenGL/extensions/OES/OES_compressed_ETC1_RGB8_texture.txt

So the 24 bytes file appear to be a valid image in this format.

Image 1x1px

Thank you!

And, I still don’t get it 😃. Let’s assume the reason we found this issue is that coincidentally SIMD methods write some extra bytes into unused portions of an output buffer if it’s big enough, and for all we can tell ZLIB isn’t at fault here. That’s great! Or not, because it means we have to figure out what else could be going on it and how Git got into the situation in the first place. We are talking about a broken delta stream, advertised as 60 bytes large, but actually it’s only 24 bytes. Somehow, these 60 bytes that Git uses to apply the delta still work and both Git and gitoxide produce a buffer with the same hash. As we know, it’s perfectly possible for that delta buffer to not apply correctly if these bytes aren’t what they currently are, so Git seems to accidentally rely on ZLIB overwriting the buffer in a very reproducible way. Obviously, it also doesn’t check if the amount of decoded bytes matches the output buffer size.

How can Git get into a situation like this? When creating an object, it takes an input buffer and hashes it. Then it puts it into a tree, creates a commit, and that’s it. So we know that at some point it must have seen exactly that buffer, otherwise it wouldn’t be referenced in the tree. The issue we are seeing is with the delta entry in a pack though. I don’t know much about the deltification algorithm, and all I know is that it works backwards, finding bases that when deltified with the target have the smallest set of instructions. Coming for that object that has our hash, we compare to the base that it ends up using, out comes a delta buffer with instructions that is 60 bytes large. All of these instructions would have to be relevant. Somehow we manage to compress that buffer such that it never decompresses in full again… ?

This… is where I think it’s too much of a conincidence. Should I believe that the delta buffer also works when some other garbage is mixed in? After all, it’s always consumed in full, the full 60 bytes, and they produce a data buffer which hashes to the correct object. Definitely, this is a strange delta, after all the object itself is just 24 bytes, it probably shouldn’t be deltified at all, but in the end it doesn’t matter as the final result is correct. So… I’d rather believe the consumed_out might be wrong somehow, maybe it’s a bug even in the original implementation? Is that the reason that Git also doesn’t check consumed_out?

Don’t get me wrong, I’d be happy to fix it by comparing these two values and bailing if they mismatch, because they really shouldn’t, but I also have a feeling that this would be very wrong and might yield many false-positives.

If it was true, and if consumed_out can’t be trusted, then maybe what we are seeing really is some very strange inconsistency on the Rust zlib implementation, maybe even to the point where it’s a compiler bug.

This object 5665283d8c34f78a615fb6bf76fdd14bdb19b882 being connected to a tree which is connected to a commit really changes everything.

What I’d find really interesting is if we’d hack the pack delta apply machinery to store each slice of compressed bytes on disk, numbered. Then the highest number should be the slice that decompresses incorrectly.

From there, it could be put into a unit test which can run on x86_64 CI to reproduce there and be made available to researchers. It’s notable that this also happened when miniz_oxide was used.

Just to sketch the path of data flow, we end with passing a decompressed slice of bytes to delta::apply():

https://github.com/GitoxideLabs/gitoxide/blob/fce70950006892f51b32af233656be6fe5de9df3/gix-pack/src/data/file/decode/entry.rs#L363-L377

I does some very clever buffer handling to avoid allocations, but ultimately here is the decompression function:

https://github.com/GitoxideLabs/gitoxide/blob/fce70950006892f51b32af233656be6fe5de9df3/gix-pack/src/data/file/decode/entry.rs#L119-L132

…which calls the once primitive that is used everywhere in the codebase.

https://github.com/GitoxideLabs/gitoxide/blob/beba7204a50a84b30e3eb81413d968920599e226/gix-features/src/zlib/mod.rs#L33-L43

It’s notable how the status is ignored, so it would be possible for the decoding to not be complete leaving zeroes in the output buffer. Thus we assume that ZLIB will always decompress the whole input if the output is big enough.

I applied this patch to see what happens when accessing the reproduction object.

diff --git a/gix-pack/src/data/file/decode/entry.rs b/gix-pack/src/data/file/decode/entry.rs
index bc8d5a2ca..1ee236079 100644
--- a/gix-pack/src/data/file/decode/entry.rs
+++ b/gix-pack/src/data/file/decode/entry.rs
@@ -128,7 +128,10 @@ impl File {
         inflate.reset();
         inflate
             .once(&self.data[offset..], out)
-            .map(|(_status, consumed_in, _consumed_out)| consumed_in)
+            .map(|(_status, consumed_in, _consumed_out)| {
+                dbg!(_status);
+                consumed_in
+            })
     }
 
     /// Like `decompress_entry_from_data_offset`, but returns consumed input and output.

On an ARM64 machine, this looks like this:

❯ /Users/byron/dev/github.com/GitoxideLabs/gitoxide/target/debug/gix cat 5665283d8c34f78a615fb6bf76fdd14bdb19b882
[gix-pack/src/data/file/decode/entry.rs:132:17] _status = StreamEnd
[gix-pack/src/data/file/decode/entry.rs:132:17] _status = StreamEnd
PKM 10@X%                     

I am pretty sure hope that if you would apply this patch and run it on an x86 machine, the last status would not be StreamEnd.

Could you try that for me?

If that was the case, then a fix could be devised, ideally with a dumped version of the compressed slice input so we can bake this into a test, one that then is testing once() which either is made to not do things just once anymore, or there is another function that is called all() so there can be a little more machinery to keep going as long as the output buffer isn’t fully filled.

Exciting!

Discarded

A wrong path I plotted to see who calls apply().

https://github.com/GitoxideLabs/gitoxide/blob/fce70950006892f51b32af233656be6fe5de9df3/gix-pack/src/cache/delta/traverse/resolve.rs#L352-L370

These bytes are decompressed here, with the decompressed length coming from the pack slice header.

https://github.com/GitoxideLabs/gitoxide/blob/fce70950006892f51b32af233656be6fe5de9df3/gix-pack/src/cache/delta/traverse/resolve.rs#L303-L312

We re-use a Zlib context to decompress all at once…

https://github.com/GitoxideLabs/gitoxide/blob/fce70950006892f51b32af233656be6fe5de9df3/gix-pack/src/cache/delta/traverse/resolve.rs#L455-L468

Could you try to compile with an older version, one before 96164c5 (96164c5~1 == f3684a4f67f98c0e3884e2d348cb092f48ca443a) to see if this is still reproducible without zlib-rs?

I tried this on my x86_64 machine, pinning the gix package to revision f3684a4f67f98c0e3884e2d348cb092f48ca443a with a line like this in a few Nosey Parker Cargo.toml files:

gix = { git = "https://github.com/GitoxideLabs/gitoxide", rev = "f3684a4f67f98c0e3884e2d348cb092f48ca443a", features = ["max-performance", "serde"] }

With this change, it looks like libz-ng-sys is used instead of zlib-rs. I do not see the crash with this setup.

HOWEVER, if I change the line instead to disable the max-performance feature, I do still see the crash.

In either case, debug vs release build makes no difference in whether the crash occurs.

I’ll open a PR for the error handling changes.

I will also try reproducing with the older version before 96164c5 to see if I still have trouble there. But that’s going to be next week.