operator: Permission denied on relation-get

Today I saw the following error on the juju logs, and when running the command with juju run I get the same error:

$ juju run --unit slurmctld/leader "relation-get -r 71 - slurmdbd --app"
ERROR permission denied

Here are some logs:

machine-32: 17:16:27 DEBUG jujuc running hook tool "relation-list" for slurmctld/11-slurmd-relation-created-2533241742911058760
machine-32: 17:16:27 DEBUG jujuc running hook tool "relation-get" for slurmctld/11-slurmd-relation-created-2533241742911058760
machine-32: 17:16:27 DEBUG jujuc running hook tool "juju-log" for slurmctld/11-slurmd-relation-created-2533241742911058760
machine-32: 17:16:27 ERROR unit.slurmctld/11.juju-log slurmd:72: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/model.py", line 1284, in _run
    result = run(args, **kwargs)
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-slurmctld-11/relation-get', '-r', '71', '-', 'slurmdbd', '--app', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./src/charm.py", line 415, in <module>
    main(SlurmctldCharm)
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/main.py", line 404, in main
    framework.reemit()
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/framework.py", line 732, in reemit
    self._reemit()
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/framework.py", line 767, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/src/interface_slurmd.py", line 63, in _on_relation_created
    if not self._charm.slurmdbd_info:
  File "./src/charm.py", line 85, in slurmdbd_info
    return self._slurmdbd.get_slurmdbd_info()
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/src/interface_slurmdbd.py", line 111, in get_slurmdbd_info
    if app_data:
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/model.py", line 394, in __len__
    return len(self._data)
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/model.py", line 384, in _data
    data = self._lazy_data = self._load()
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/model.py", line 748, in _load
    return self._backend.relation_get(self.relation.id, self._entity.name, self._is_app)
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/model.py", line 1351, in relation_get
    return self._run(*args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-slurmctld-11/charm/venv/ops/model.py", line 1286, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: b'ERROR permission denied\n'
unit-slurmctld-11: 17:16:27 ERROR juju.worker.uniter.operation hook "slurmd-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-slurmctld-11: 17:16:27 DEBUG juju.machinelock machine lock released for slurmctld/11 uniter (run relation-created (72; app: slurmd) hook)

The charm code that triggered this error is here

Juju controller 2.9.4, Juju client 2.9.4.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (8 by maintainers)

Most upvoted comments

Closing this issue for now, the discussion continues at: https://bugs.launchpad.net/juju/+bug/1911010. I think this is not likely to be fixed by OF, but rather in Juju. Thanks for the feedback! 😃

jnsgruk on Sep 28, 2021

Let’s continue this discussion on Launchpad. Strictly speaking, permissions on data bags are “below” the operator framework, and all we can do here is inherit Juju’s permissions/behavior.

rbarry82 on Jul 21, 2021

All units are not able to read their own application data. Eg:

foo <=> bar

foo/0 is the leader of foo bar/1 is the leader of bar

app data for foo can be read by any unit of bar, and if any changes are made by foo/0 then all units of bar will have a relation-data-changed event triggered for the associated application data app data for foo can only be read and set by foo/0 (or whoever the current leader is) if units of foo need to coordinate data, they should not do so by setting app data that is being shared with bar, but by using a peer relation with foo. In a peer relation, all units of foo can read the app data (but only the leader can set app data).

This is because app data in a relation to another app is not meant as a communication channel with your own units. If it was, then any change to application relation data would need to trigger relation-data-changed both for all units of bar, but also for all units of foo (because they need to be informed if something they read has been changed). But we already have a way to have that time of alerting (a peer relation).

There are a lot of bits about making sure the right units have the right data and ordering, etc that this solves. It also ensures that you won’t read stale data without being told when it is updated.

On Wed, Jun 16, 2021 at 3:17 PM jamesbeedy @.***> wrote:

Geh … I am getting some strange behavior around relation-get now.

$ juju status Model Controller Cloud/Region Version SLA Timestamp default localhost-localhost localhost/localhost 2.9.4 unsupported 19:13:37Z

App Version Status Scale Charm Store Channel Rev OS Message percona-cluster 5.7.20 active 1 percona-cluster charmstore stable 293 ubuntu Unit is ready slurmctld 0.6.4-2-g11b… active 1 slurmctld local 0 ubuntu slurmctld available slurmd 0.6.4-2-g11b… error 2 slurmd local 1 ubuntu hook failed: “slurmd-relation-created” slurmdbd 0.6.4-2-g11b… active 1 slurmdbd local 0 ubuntu slurmdbd ready slurmrestd 0.6.4-2-g11b… active 1 slurmrestd local 0 ubuntu slurmrestd available

Unit Workload Agent Machine Public address Ports Message percona-cluster/0* active idle 0 10.19.68.195 3306/tcp Unit is ready slurmctld/0* active idle 1 10.19.68.36 slurmctld available slurmd/0* active idle 2 10.19.68.192 slurmd available slurmd/1 error idle 5 10.19.68.240 hook failed: “slurmd-relation-created” slurmdbd/0* active idle 3 10.19.68.117 slurmdbd ready slurmrestd/0* active idle 4 10.19.68.144 slurmrestd available

Machine State DNS Inst id Series AZ Message 0 started 10.19.68.195 juju-a3736f-0 bionic Running 1 started 10.19.68.36 juju-a3736f-1 focal Running 2 started 10.19.68.192 juju-a3736f-2 focal Running 3 started 10.19.68.117 juju-a3736f-3 focal Running 4 started 10.19.68.144 juju-a3736f-4 focal Running 5 started 10.19.68.240 juju-a3736f-5 focal Running

(venv) @.:~/allcode/github/omnivector/slurm-bundles$ juju run --unit slurmd/1 “/var/lib/juju/tools/unit-slurmd-1/relation-get -r 5 - slurmd --app --format=json” ERROR permission denied (venv) @.:~/allcode/github/omnivector/slurm-bundles$ # now from the leader (venv) @.***:~/allcode/github/omnivector/slurm-bundles$ juju run --unit slurmd/0 “/var/lib/juju/tools/unit-slurmd-0/relation-get -r 5 - slurmd --app --format=json” {“partition-name”:“juju-compute-JxBr”,“partition_info”:“{"partition_name": "juju-compute-JxBr", "partition_state": "UP", "partition_config": ""}”,“partition_name”:“juju-compute-JxBr”}

I was under the impression that all units of an application were able to read their own application level data on a relation. If this is correct possibly I am hitting a bug, otherwise I think I had the wrong understanding.

Any ideas here?

Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/canonical/operator/issues/551#issuecomment-862646974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABRQ7ORVPIWLWKXIREUUSLTTD2CZANCNFSM46PJOTLA .

jameinel on Jun 16, 2021