opendal: Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6)
Webhdfs with hadoop 2.6 will return error like the following for multi write:
Finished dev [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/opendal_test`
thread 'main' panicked at src/main.rs:21:35:
called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: Unexpected (permanent) at Writer::close => deserialize json
Context:
service: webhdfs
path: webhdfs_test1/sink_file.txt
Source:
EOF while parsing a value at line 1 column 0
Backtrace:
0: std::backtrace_rs::backtrace::libunwind::trace
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: std::backtrace::Backtrace::create
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/backtrace.rs:331:13
3: opendal::types::error::Error::new
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/types/error.rs:338:24
4: opendal::raw::serde_util::new_json_deserialize_error
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/serde_util.rs:43:5
5: core::ops::function::FnOnce::call_once
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
6: core::result::Result<T,E>::map_err
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:829:27
7: opendal::services::webhdfs::backend::WebhdfsBackend::webhdfs_create_object_request::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/services/webhdfs/backend.rs:273:13
8: <opendal::services::webhdfs::writer::WebhdfsWriter as opendal::raw::oio::write::block_write::BlockWrite>::write_block::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/services/webhdfs/writer.rs:78:14
9: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
10: opendal::raw::oio::write::block_write::WriteBlockFuture::new::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/oio/write/block_write.rs:123:14
11: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/future/future.rs:125:9
12: futures_util::future::future::FutureExt::poll_unpin
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/future/mod.rs:558:9
13: <opendal::raw::oio::write::block_write::WriteBlockFuture as core::future::future::Future>::poll
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/oio/write/block_write.rs:111:9
14: futures_util::future::future::FutureExt::poll_unpin
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/future/future/mod.rs:558:9
15: <opendal::raw::futures_util::ConcurrentFutures<F> as futures_core::stream::Stream>::poll_next
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/futures_util.rs:179:28
16: futures_util::stream::stream::StreamExt::poll_next_unpin
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.29/src/stream/stream/mod.rs:1638:9
17: <opendal::raw::oio::write::block_write::BlockWriter<W> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/oio/write/block_write.rs:262:57
18: <opendal::raw::enum_utils::TwoWays<ONE,TWO> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/enum_utils.rs:116:29
19: <opendal::layers::error_context::ErrorContextWrapper<T> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/layers/error_context.rs:420:9
20: <opendal::layers::complete::CompleteWriter<W> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/layers/complete.rs:728:16
21: <opendal::raw::enum_utils::TwoWays<ONE,TWO> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/enum_utils.rs:116:29
22: <alloc::boxed::Box<T> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/oio/write/api.rs:125:9
23: <opendal::raw::oio::write::api::CloseFuture<W> as core::future::future::Future>::poll
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/raw/oio/write/api.rs:197:9
24: opendal::types::writer::Writer::close::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-036c748dbc56dae9/00c6129/core/src/types/writer.rs:208:28
25: opendal_test::multi_write_example::{{closure}}
at ./src/main.rs:38:15
26: opendal_test::main::{{closure}}
at ./src/main.rs:21:29
27: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/park.rs:282:63
28: tokio::runtime::coop::with_budget
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/coop.rs:107:5
29: tokio::runtime::coop::budget
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/coop.rs:73:5
30: tokio::runtime::park::CachedParkThread::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/park.rs:282:31
31: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/context/blocking.rs:66:9
32: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
33: tokio::runtime::context::runtime::enter_runtime
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/context/runtime.rs:65:16
34: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
35: tokio::runtime::runtime::Runtime::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/runtime.rs:350:45
36: opendal_test::main
at ./src/main.rs:22:5
37: core::ops::function::FnOnce::call_once
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
38: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/sys_common/backtrace.rs:154:18
39: std::rt::lang_start::{{closure}}
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/rt.rs:166:18
40: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:284:13
41: std::panicking::try::do_call
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504:40
42: std::panicking::try
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468:19
43: std::panic::catch_unwind
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142:14
44: std::rt::lang_start_internal::{{closure}}
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/rt.rs:148:48
45: std::panicking::try::do_call
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:504:40
46: std::panicking::try
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:468:19
47: std::panic::catch_unwind
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panic.rs:142:14
48: std::rt::lang_start_internal
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/rt.rs:148:20
49: std::rt::lang_start
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/rt.rs:165:17
50: _main
}
stack backtrace:
0: rust_begin_unwind
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
1: core::panicking::panic_fmt
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
2: core::result::unwrap_failed
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:1652:5
3: core::result::Result<T,E>::unwrap
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/result.rs:1077:23
4: opendal_test::main::{{closure}}
at ./src/main.rs:21:5
5: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/park.rs:282:63
6: tokio::runtime::coop::with_budget
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/coop.rs:107:5
7: tokio::runtime::coop::budget
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/coop.rs:73:5
8: tokio::runtime::park::CachedParkThread::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/park.rs:282:31
9: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/context/blocking.rs:66:9
10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
11: tokio::runtime::context::runtime::enter_runtime
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/context/runtime.rs:65:16
12: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
13: tokio::runtime::runtime::Runtime::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/runtime.rs:350:45
14: opendal_test::main
at ./src/main.rs:22:5
15: core::ops::function::FnOnce::call_once
at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Process finished with exit code 101
Seems we didn’t handle the case that body is empty.
About this issue
- Original URL
- State: closed
- Created 5 months ago
- Comments: 21 (21 by maintainers)
I have reached out to Brahma Reddy Battula, the reporter for https://issues.apache.org/jira/browse/HDFS-6641 to find the exact HDFS version in which concat was fixed. He is an Hadoop PMC. This could help us document exactly the least version OpenDAL will be able to support.
Sure, I will raise in some time. Thank you!
It seems hdfs 2.6 cannot be supported with this implementation. It would be good to have a mention in doc for the least version of hdfs being supported. Thank you!