theseus: RuntimeError: sym_strides() called on an undefined Tensor

❓ Questions and Help

Hi, do you have any idea about this issue?

Traceback (most recent call last):
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/huangkun/.virtualenvs/theseus/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/huangkun/Git/lora_slam/test/slam_test.py", line 32, in main
    bundle_adjustment(global_map, True, cfg, timestamp, 2)
  File "/home/huangkun/Git/lora_slam/slam/bundle_adjustment.py", line 75, in bundle_adjustment
    theseus_outputs, info = theseus_layer.forward(optimizer_kwargs={
  File "/home/huangkun/Git/theseus/theseus/theseus_layer.py", line 93, in forward
    vars, info = _forward(
  File "/home/huangkun/Git/theseus/theseus/theseus_layer.py", line 169, in _forward
    objective.update(input_tensors)
  File "/home/huangkun/Git/theseus/theseus/core/objective.py", line 808, in update
    self.update_vectorization_if_needed()
  File "/home/huangkun/Git/theseus/theseus/core/objective.py", line 826, in update_vectorization_if_needed
    self._vectorization_run()
  File "/home/huangkun/Git/theseus/theseus/core/vectorizer.py", line 401, in _vectorize
    self._handle_singleton_wrapper(schema, cost_fn_wrappers, mode)
  File "/home/huangkun/Git/theseus/theseus/core/vectorizer.py", line 353, in _handle_singleton_wrapper
    ) = Vectorize._get_fn_results_for_mode(mode, wrapper.cost_fn)
  File "/home/huangkun/Git/theseus/theseus/core/vectorizer.py", line 307, in _get_fn_results_for_mode
    return cost_fn.weighted_jacobians_error()
  File "/home/huangkun/Git/theseus/theseus/core/cost_function.py", line 121, in weighted_jacobians_error
    jacobian, err = self.jacobians()
  File "/home/huangkun/Git/theseus/theseus/core/cost_function.py", line 355, in jacobians
    jacobians_full = self._compute_autograd_jacobian_vmap(
  File "/home/huangkun/Git/theseus/theseus/core/cost_function.py", line 341, in _compute_autograd_jacobian_vmap
    return vmap(jacrev(jac_fn, argnums=0))(optim_tensors, aux_tensors)
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 434, in wrapped
    return _flat_vmap(
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 39, in fn
    return f(*args, **kwargs)
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 619, in _flat_vmap
    batched_outputs = func(*batched_inputs, **kwargs)
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/eager_transforms.py", line 598, in wrapper_fn
    flat_jacobians_per_input = compute_jacobian_stacked()
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/eager_transforms.py", line 529, in compute_jacobian_stacked
    chunked_result = vmap(vjp_fn)(basis)
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 434, in wrapped
    return _flat_vmap(
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 39, in fn
    return f(*args, **kwargs)
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/vmap.py", line 619, in _flat_vmap
    batched_outputs = func(*batched_inputs, **kwargs)
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/eager_transforms.py", line 325, in wrapper
    result = _autograd_grad(flat_primals_out, flat_diff_primals, flat_cotangents,
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/_functorch/eager_transforms.py", line 113, in _autograd_grad
    grad_inputs = torch.autograd.grad(diff_outputs, inputs, grad_outputs,
  File "/home/huangkun/.local/lib/python3.8/site-packages/torch/autograd/__init__.py", line 303, in grad
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: sym_strides() called on an undefined Tensor

The code has two kinds of loss to do pose-graph bundle adjustment:

class DepthConstraint(CostFunction):
    def __init__(
            self,
            vector: Vector,
            cost_weight: CostWeight,
            name: Optional[str] = None,
    ):
        super().__init__(cost_weight, name=name)
        self.vector = vector
        self.register_optim_var("vector")

    def dim(self):
        return 1

    def _compute_error(self) -> Tuple[torch.Tensor, torch.Tensor]:
        vector = self.vector.tensor
        below_idx = vector < 0.01
        error = vector.new_zeros(vector.shape)
        error = torch.where(below_idx, 0.01 - vector, error)
        return torch.sum(error, dim=1).unsqueeze(1), below_idx

    def error(self) -> torch.Tensor:
        return self._compute_error()[0]

    def jacobians(self) -> Tuple[List[torch.Tensor], torch.Tensor]:
        error, below_idx = self._compute_error()
        J = self.vector.tensor.new_zeros(
            self.vector.shape[0], 1, self.vector.dof()
        )
        J[below_idx.unsqueeze(1)] = -1.0
        return [J], error

    def _copy_impl(self, new_name: Optional[str] = None) -> "DepthConstraint":
        return DepthConstraint(
            self.vector.copy(),
            self.weight.copy(),
            name=new_name,
        )

def photometric_error(optim_vars, aux_vars) -> torch.Tensor:
    d1: th.Vector = optim_vars[0]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)
    d2: th.Vector = optim_vars[1]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)
    T_wc1: th.SE3 = optim_vars[2]
    T_wc2: th.SE3 = optim_vars[3]

    image1: th.Variable = aux_vars[0]  # (batch_size, 1, h, w) {normalized_intensity}
    image2: th.Variable = aux_vars[1]  # (batch_size, 1, h, w) {normalized_intensity}
    p_unit_sphere_or_plane: th.Variable = aux_vars[2]  # (batch_size=1, h, w, 3)
    k: th.Variable = aux_vars[3]  # (batch_size=1, 4) {fx, fy, cx, cy}

    return _photometric_error(T_wc1, T_wc2, d1, d2, image1, image2, p_unit_sphere_or_plane, k)


def photometric_error_fix_pose(optim_vars, aux_vars) -> torch.Tensor:
    d1: th.Vector = optim_vars[0]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)
    d2: th.Vector = optim_vars[1]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)

    image1: th.Variable = aux_vars[0]  # (batch_size, 1, h, w) {normalized_intensity}
    image2: th.Variable = aux_vars[1]  # (batch_size, 1, h, w) {normalized_intensity}
    p_unit_sphere_or_plane: th.Variable = aux_vars[2]  # (batch_size=1, h, w, 3)
    k: th.Variable = aux_vars[3]  # (batch_size=1, 4) {fx, fy, cx, cy}
    T_wc1: th.SE3 = aux_vars[4]
    T_wc2: th.SE3 = aux_vars[5]

    return _photometric_error(T_wc1, T_wc2, d1, d2, image1, image2, p_unit_sphere_or_plane, k)


def photometric_error_fix_ref(optim_vars, aux_vars) -> torch.Tensor:
    d2: th.Vector = optim_vars[0]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)
    T_wc2: th.SE3 = optim_vars[1]

    image1: th.Variable = aux_vars[0]  # (batch_size, 1, h, w) {normalized_intensity}
    image2: th.Variable = aux_vars[1]  # (batch_size, 1, h, w) {normalized_intensity}
    p_unit_sphere_or_plane: th.Variable = aux_vars[2]  # (batch_size=1, h, w, 3)
    k: th.Variable = aux_vars[3]  # (batch_size=1, 4) {fx, fy, cx, cy}
    d1: th.Vector = aux_vars[4]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)
    T_wc1: th.SE3 = aux_vars[5]

    return _photometric_error(T_wc1, T_wc2, d1, d2, image1, image2, p_unit_sphere_or_plane, k)


def photometric_error_fix_pose_and_ref(optim_vars, aux_vars) -> torch.Tensor:
    d2: th.Vector = optim_vars[0]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)

    image1: th.Variable = aux_vars[0]  # (batch_size, 1, h, w) {normalized_intensity}
    image2: th.Variable = aux_vars[1]  # (batch_size, 1, h, w) {normalized_intensity}
    p_unit_sphere_or_plane: th.Variable = aux_vars[2]  # (batch_size=1, h, w, 3)
    k: th.Variable = aux_vars[3]  # (batch_size=1, 4) {fx, fy, cx, cy}
    d1: th.Vector = aux_vars[4]  # unknown scale, > 0, better rescale ahead, (batch_size, h x w x 1)
    T_wc1: th.SE3 = aux_vars[5]
    T_wc2: th.SE3 = aux_vars[6]

    return _photometric_error(T_wc1, T_wc2, d1, d2, image1, image2, p_unit_sphere_or_plane, k)

    def add_edge_cost_function(self, kf1: Keyframe, kf2: Keyframe, fix_ref: bool):
        if self.fix_pose:
            if fix_ref:
                optim_vars = kf2.depth
                aux_vars = kf1.image, kf2.image, self.p_unit_sphere_or_plane, self.k, kf1.depth, kf1.T_wc, kf2.T_wc
                cost_function = th.AutoDiffCostFunction(optim_vars, photometric_error_fix_pose_and_ref, 1,
                                                        aux_vars=aux_vars)
            else:
                optim_vars = kf1.depth, kf2.depth
                aux_vars = kf1.image, kf2.image, self.p_unit_sphere_or_plane, self.k, kf1.T_wc, kf2.T_wc
                cost_function = th.AutoDiffCostFunction(optim_vars, photometric_error_fix_pose, 1,
                                                        aux_vars=aux_vars)
        else:
            if fix_ref:
                optim_vars = kf2.depth, kf2.T_wc
                aux_vars = kf1.image, kf2.image, self.p_unit_sphere_or_plane, self.k, kf1.depth, kf1.T_wc
                cost_function = th.AutoDiffCostFunction(optim_vars, photometric_error_fix_ref, 1,
                                                        aux_vars=aux_vars)
            else:
                optim_vars = kf1.depth, kf2.depth, kf1.T_wc, kf2.T_wc
                aux_vars = kf1.image, kf2.image, self.p_unit_sphere_or_plane, self.k
                cost_function = th.AutoDiffCostFunction(optim_vars, photometric_error, 1,
                                                        aux_vars=aux_vars)
        self.objective.add(cost_function)

    def add_node_cost_function(self, kf: Keyframe):
        kf.data_updated = True

        # constraint: depth > 0
        cost_function = DepthConstraint(kf.depth, th.ScaleCostWeight(
            torch.tensor(1000., dtype=kf.depth.dtype, device=kf.depth.device)))
        self.objective.add(cost_function)

    optimizer = th.LevenbergMarquardt(
        problem.objective,
        linear_solver_cls=th.BaspachoSparseSolver,
        linearization_cls=th.SparseLinearization,
    )
    theseus_layer = th.TheseusLayer(optimizer)
    with global_map.keyframes_mutex:
        theseus_outputs, info = theseus_layer.forward(optimizer_kwargs={
            "verbose": cfg.inner_optim.verbose,
            "adaptive_damping": True})

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 24 (11 by maintainers)

Most upvoted comments

For problems with many cost functions and sparse connectivity, Baspacho is usually the best choice. On the other hand, if you have a single cost function, then there is no advantage from using a sparse solver, because the linear systems will be dense; in that case CholeskyDenseSolver is probably better. When there are only a few cost functions, it’s hard to say, might require some experimentation.

That being said, for your current formulation the main issue is the extremely large number of optimization variables. Are you able to reformulate this problem in some way so that the dimension of the optimization variables is lower?

luisenp on Sep 10, 2023

Thanks!

I played a bit with this and it seems that torch.vmap doesn’t like this operation inside _photometric_error --> p_unit_sphere_or_plane.tensor * d2.tensor.view(batch_size, h, w, 1) (particularly, the view function). You’ll need to rewrite this line in some way without using view if you want to use vmap. Note that this autograd mode is only useful if your batch size is > 1.

I also ran with autograd mode dense, with a smaller image size (48 * 64), and the code also works. However for your full image size I’m not sure you’ll be able to run our optimizers. You have two optimization variables of size 480 * 640 = 307200, so you’ll end having to solve a dense linear system with matrix size of 614000 * 614000, which is something like 1.5TB of memory.

Maybe you need to revisit how you are formulating this problem?

luisenp on Sep 10, 2023