cue: cue: optional fields cannot be referenced in for loops of optional fields

What version of CUE are you using (cue version)?

$ cue version
0.4.3

Does this issue reproduce with the latest release?

yes

What did you do?

parameter: {
	optioanlArr?: [...{
		name:  string
		data?: string
	}]
}

outputs: {
	for v in parameter.optionalArr if v.data != _|_  {
		name: v.name
	}
}

What did you expect to see?

This is also the result in cue 0.2.2:

outputs: {
    for v in parameter.optionalArr if false {
        name: v.name
    }
}

What did you see instead?

outputs: {
    for v in parameter.optionalArr if v.data != _|_ // explicit error (_|_ literal) in source
    {
        name: v.name
    }
}

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 35 (21 by maintainers)

Most upvoted comments

I’ve been looking into this a bit more. Here’s a smaller version of the example which I think represents the problem concisely:

p?: []
a: [for k in p {k}] | null

A cue export on this succeeds OK on v0.2.2 but fails with “cannot reference optional field: p” on v0.4.3.

Note that without the disjunction (just using a: [for k in p {k}]) it fails on both versions, so the key here seems to be the disjunction.

By contrast, this code works fine on both versions of CUE:

p?: {}
a: p | null
b: p.foo | null

so it’s clear to me that a reference to an optional value evaluates to bottom, as do expressions containing such a reference, and according to the spec:

The disjunction of a value a with bottom is always a.

AFAICS there should not be a qualitative difference between a comprehension involving a reference to an optional value and any other expression involving an optional value, so at the moment I see this as a straightforward bug in the latest version of CUE.

+3
rogpeppe on Aug 4, 2022

@FogDong I believe Roger is just pointing out that the bug appears in one case but not the other.

+1
mvdan on Aug 5, 2022

So, to be a bit clearer for folks following along, fixing the bug in v0.4.3 will fix some but not all of the issues that are being seen. As an example, this code has several list comprehensions that will be fixed, but some struct comprehensions (e.g. this) which won’t and will need the CUE to be fixed.

+1
rogpeppe on Aug 4, 2022

Yes, I think we need to write the fixed thing back, and the Kubernetes mutate webhook can actually do that

Great.

But we really need an API/Function Call to help us do that on our own server.

https://pkg.go.dev/cuelang.org/go/tools/fix#Instances will likely be that; but will wait for @rogpeppe’s analysis to confirm.

+1
myitcv on Aug 3, 2022

Thanks, @FogDong. For anyone following along, here is a standalone reproducer:

# v0.2.2
go get cuelang.org/go@v0.2.2
go mod tidy
go run common.go v0.2.2.go

# current tip
go get cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353
go mod tidy
go run common.go tip.go

-- common.go --
package main

import (
	"fmt"

	"cuelang.org/go/cue/ast"
	"cuelang.org/go/cue/token"
)

var storage = `
pvcVolumesList: *[
	for v in parameter.pvc if v.mountPath != _|_{
	{
		name: "pvc-" + v.name
		persistentVolumeClaim: claimName: v.name
	}
},
] | []

configMapVolumesList: *[
		for v in parameter.configMap if v.mountPath != _|_ {
	{
		name: "configmap-" + v.name
		configMap: {
			name:        v.name
		}
	}
},
] | []


volumesList: pvcVolumesList + configMapVolumesList
deDupVolumesArray: [
for val in [
	for i, vi in volumesList {
		for j, vj in volumesList if j < i && vi.name == vj.name {
			ignore: true
		}
		vi
	},
] if val.ignore == _|_ {
	val
},
]

patch: spec: template: spec: {
// +patchKey=name
volumes: deDupVolumesArray
}

parameter: {
// +usage=Declare pvc type storage
pvc?: [...{
	name:       string
	mountPath?: string
}]

// +usage=Declare config map type storage
configMap?: [...{
	name:      string
	mountPath?:  string
}]
}
`

// copy from cuelang.org/go/internal/internal.go
func ToFile(n ast.Node) *ast.File {
	switch x := n.(type) {
	case nil:
		return nil
	case *ast.StructLit:
		return &ast.File{Decls: x.Elts}
	case ast.Expr:
		ast.SetRelPos(x, token.NoSpace)
		return &ast.File{Decls: []ast.Decl{&ast.EmbedDecl{Expr: x}}}
	case *ast.File:
		return x
	default:
		panic(fmt.Sprintf("Unsupported node type %T", x))
	}
}
-- go.mod --
module mod.com

go 1.18

require cuelang.org/go v0.2.2

-- tip.go --
package main

import (
	"fmt"

	"cuelang.org/go/cue"
	"cuelang.org/go/cue/build"
	"cuelang.org/go/cue/cuecontext"
	"cuelang.org/go/cue/format"
	"cuelang.org/go/cue/parser"
)

func main() {
	ctx := cuecontext.New()
	file, err := parser.ParseFile("-", storage)
	if err != nil {
		panic(err)
	}
	builder := &build.Instance{}
	err = builder.AddSyntax(file)
	if err != nil {
		panic(err)
	}
	v := ctx.BuildInstance(builder)

	subv := v.LookupPath(cue.ParsePath("patch"))
	node := subv.Syntax()
	// We'll do some modification to the node here so convert file to node
	f := ToFile(node)
	// panic here
	_ = ctx.BuildFile(f)

	formatting, err := format.Node(node)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(formatting))
}
-- v0.2.2.go --
package main

import (
	"fmt"

	"cuelang.org/go/cue"
	"cuelang.org/go/cue/format"
)

func main() {
	var r cue.Runtime
	cueInst, err := r.Compile("-", storage)
	if err != nil {
		panic(err)
	}
	subv := cueInst.Value().Lookup("patch")
	node := subv.Syntax()
	f := ToFile(node)
	// no panic
	_, _ = r.CompileFile(f)
	formatting, err := format.Node(node)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(formatting))
}

@FogDong’s expectation is that this test should pass, but it fails with:

# v0.2.2 (0.684s)
> go get cuelang.org/go@v0.2.2
> go mod tidy
[stderr]
go: finding module for package cuelang.org/go/cue/cuecontext
go: found cuelang.org/go/cue/cuecontext in cuelang.org/go v0.4.3

> go run common.go v0.2.2.go
[stdout]
{
        spec: {
                template: {
                        spec: {
                                volumes: deDupVolumesArray
                        }
                }
        }
}

# current tip (0.418s)
> go get cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353
[stderr]
go: upgraded cuelang.org/go v0.4.3 => v0.4.4-0.20220729051708-0a46a1624353
go: upgraded golang.org/x/net v0.0.0-20200226121028-0de0cce0169b => v0.0.0-20220425223048-2871e0cb64e4
go: upgraded gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b => v3.0.1

> go mod tidy
> go run common.go tip.go
[stderr]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x1eeec8]

goroutine 1 [running]:
cuelang.org/go/internal/core/adt.(*Environment).evalCached(0x0, 0x4000310240, {0x5d4fd0, 0x4000335830})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/composite.go:143 +0x58
cuelang.org/go/internal/core/adt.(*LetReference).evaluate(0x0?, 0x5d5190?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/expr.go:898 +0x48
cuelang.org/go/internal/core/adt.(*OpContext).evalState(0x4000310240, {0x5d5190, 0x4000332820}, 0x2)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:653 +0x234
cuelang.org/go/internal/core/adt.(*OpContext).Evaluate(0x4000310240, 0x40003379a0, {0x5d5190, 0x4000332820})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:557 +0x124
cuelang.org/go/internal/core/adt.(*OpContext).Concrete(0x4000310240, 0x0?, {0x5d5190?, 0x4000332820?}, {0x492320?, 0x888dd0})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:495 +0x38
cuelang.org/go/internal/core/adt.(*BinaryExpr).evaluate(0x4000335cb0, 0x4000310240)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/expr.go:1214 +0x22c
cuelang.org/go/internal/core/adt.(*OpContext).evalState(0x4000310240, {0x5d4d10, 0x4000335cb0}, 0x2)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:653 +0x234
cuelang.org/go/internal/core/adt.(*Environment).evalCached(0x40003379a0, 0x4000310240, {0x5d4d10, 0x4000335cb0})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/composite.go:150 +0x178
cuelang.org/go/internal/core/adt.(*LetReference).evaluate(0x0?, 0x5d5190?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/expr.go:898 +0x48
cuelang.org/go/internal/core/adt.(*OpContext).unifyNode(0x4000310240, {0x5d5190, 0x4000332a80}, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:738 +0x2a4
cuelang.org/go/internal/core/adt.(*OpContext).node(0x4000310240, {0x5d27f0?, 0x4000335ce0}, {0x5d5190?, 0x4000332a80?}, 0x1, 0x52?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:919 +0x50
cuelang.org/go/internal/core/adt.(*ForClause).yield(0x4000335ce0, 0x4000310240, 0xffff82e30108?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/expr.go:1702 +0x58
cuelang.org/go/internal/core/adt.(*OpContext).yield(0x4000310240, 0x0, 0x4000337b30, 0x1d358?, 0x0?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:486 +0x1bc
cuelang.org/go/internal/core/adt.(*nodeContext).addLists(0x4000347500)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:2176 +0x1504
cuelang.org/go/internal/core/adt.(*nodeContext).postDisjunct(0x4000347500, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:488 +0x100
cuelang.org/go/internal/core/adt.(*nodeContext).expandDisjuncts(0x4000347500, 0x0?, 0x4000347500, 0x0, 0x0, 0x1)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/disjunct.go:151 +0x320
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x4000342fc0, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:323 +0x878
cuelang.org/go/internal/core/adt.(*ListLit).evaluate(0x4000335800, 0x4000310240)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/expr.go:307 +0x124
cuelang.org/go/internal/core/adt.(*OpContext).unifyNode(0x4000310240, {0x5d51d0, 0x4000335800}, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:738 +0x2a4
cuelang.org/go/internal/core/adt.(*OpContext).node(0x4000310240, {0x5d27f0?, 0x4000335e60}, {0x5d51d0?, 0x4000335800?}, 0x1, 0x0?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:919 +0x50
cuelang.org/go/internal/core/adt.(*ForClause).yield(0x4000335e60, 0x4000310240, 0x4000321500?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/expr.go:1702 +0x58
cuelang.org/go/internal/core/adt.(*OpContext).yield(0x4000310240, 0x0, 0x4000337ae0, 0x23b770?, 0x40000b0630?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/context.go:486 +0x1bc
cuelang.org/go/internal/core/adt.(*nodeContext).addLists(0x4000347180)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:2176 +0x1504
cuelang.org/go/internal/core/adt.(*nodeContext).postDisjunct(0x4000347180, 0x2)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:488 +0x100
cuelang.org/go/internal/core/adt.(*nodeContext).expandDisjuncts(0x4000347180, 0x4?, 0x4000347180, 0x0, 0x0, 0x1)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/disjunct.go:151 +0x320
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x4000342f30, 0x2)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:323 +0x878
cuelang.org/go/internal/core/adt.(*nodeContext).addVertexConjuncts(0x4000346e00, {0x4000337a90, {0x5d27a0, 0x4000333000}, {0x0, 0x0, 0x0, 0x0}}, 0x4000342f30, 0x0)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:1594 +0x454
cuelang.org/go/internal/core/adt.(*nodeContext).evalExpr(0x4000346e00, {0x4000337a90, {0x5d27a0, 0x4000333000}, {0x0, 0x0, 0x0, 0x0}})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:1450 +0x4bc
cuelang.org/go/internal/core/adt.(*nodeContext).addExprConjunct(0x4000346e00, {0x4000337a90, {0x5d27a0, 0x4000333000}, {0x0, 0x0, 0x0, 0x0}})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:1410 +0x614
cuelang.org/go/internal/core/adt.(*nodeContext).insertConjuncts(0x4000346e00, 0x60?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:421 +0xe8
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x4000342ea0, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:255 +0xdac
cuelang.org/go/internal/core/adt.(*nodeContext).completeArcs(0x4000346a80, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:732 +0x264
cuelang.org/go/internal/core/adt.(*nodeContext).postDisjunct(0x4000346a80, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:611 +0x694
cuelang.org/go/internal/core/adt.(*nodeContext).expandDisjuncts(0x4000346a80, 0x14?, 0x4000346a80, 0x0, 0x0, 0x1)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/disjunct.go:151 +0x320
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x4000342e10, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:323 +0x878
cuelang.org/go/internal/core/adt.(*nodeContext).completeArcs(0x4000346700, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:732 +0x264
cuelang.org/go/internal/core/adt.(*nodeContext).postDisjunct(0x4000346700, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:611 +0x694
cuelang.org/go/internal/core/adt.(*nodeContext).expandDisjuncts(0x4000346700, 0xf0?, 0x4000346700, 0x0, 0x0, 0x1)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/disjunct.go:151 +0x320
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x4000342d80, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:323 +0x878
cuelang.org/go/internal/core/adt.(*nodeContext).completeArcs(0x4000346380, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:732 +0x264
cuelang.org/go/internal/core/adt.(*nodeContext).postDisjunct(0x4000346380, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:611 +0x694
cuelang.org/go/internal/core/adt.(*nodeContext).expandDisjuncts(0x4000346380, 0x0?, 0x4000346380, 0x0, 0x0, 0x1)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/disjunct.go:151 +0x320
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x4000342cf0, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:323 +0x878
cuelang.org/go/internal/core/adt.(*nodeContext).completeArcs(0x4000346000, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:732 +0x264
cuelang.org/go/internal/core/adt.(*nodeContext).postDisjunct(0x4000346000, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:611 +0x694
cuelang.org/go/internal/core/adt.(*nodeContext).expandDisjuncts(0x4000346000, 0x0?, 0x4000346000, 0x0, 0x0, 0x1)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/disjunct.go:151 +0x320
cuelang.org/go/internal/core/adt.(*OpContext).Unify(0x4000310240, 0x400031fcb0, 0x5)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/eval.go:323 +0x878
cuelang.org/go/internal/core/adt.(*Vertex).Finalize(...)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/internal/core/adt/composite.go:487
cuelang.org/go/cue.newVertexRoot(0x4908e0?, 0x4000310240?, 0x4000011bc0?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/cue/types.go:623 +0x7c
cuelang.org/go/cue.newValueRoot(0x5d5590?, 0x4000021e80?, {0x5d5510?, 0x400031fcb0?})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/cue/types.go:632 +0x44
cuelang.org/go/cue.(*Context).make(0x4000021e80?, 0x0?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/cue/context.go:248 +0x64
cuelang.org/go/cue.(*Context).compile(0x4000091eb8?, 0x400031de68?, 0x400007e840?)
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/cue/context.go:167 +0x44
cuelang.org/go/cue.(*Context).BuildFile(0x4000021e80, 0x4000336ff0?, {0x0, 0x0, 0x0?})
        /home/myitcv/gostuff/pkg/mod/cuelang.org/go@v0.4.4-0.20220729051708-0a46a1624353/cue/context.go:160 +0xd8
main.main()
        $WORK/tip.go:31 +0x2a4
exit status 2

[exit status 1]
FAIL: /tmp/testscript476803494/repro.txt/script.txt:9: unexpected go command failure
error running repro.txt in /tmp/testscript476803494/repro.txt

Looking into this now.

+1
myitcv on Aug 1, 2022

We are close to committing a first version of both the reworked cycle detection algorithm and comprehension rewrite.

I’ll keep this on the list of things to validate in rolling this out.

+1
mpvl on Jul 29, 2022
Read more comments on GitHub
← cue: cmd/cue: unclear how to validate selector and qualified identifier errors
cue: cue: ResolveReferences option not work in method "Syntax"  →