hypothesmith: Generated programs are missing spaces

I must be missing something simple. I run this program:

from hypothesis import given
import hypothesmith as hs

@given(hs.from_grammar("eval_input"))
@settings(max_examples=50)
def test(expr):
    print(repr(expr))

test()

This prints examples like these:

'A'
'A\n'
'A,A,'
'A,A'
'A,'
'A,A,\n'
'A\n'
'AorA,A'
'AorA,'
'A,\n'
'lambda:A,\n'

I am pretty sure it meant A or A, not AorA. (I saw more similar example in other runs and variations of the program.)

It also occasionally prints a traceback and this error:

hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check fo
und 50 filtered examples but only 6 good ones. This will make your tests much slower, and also will probably dis
tort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by 
a low max_leaves parameter in recursive() calls

This is Python 3.9.2 on Windows.

hypothesis          6.13.0
hypothesmith        0.1.8

I figure Iโ€™m doing something wrong or not understanding something?

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Comments: 17 (11 by maintainers)

Most upvoted comments

Just to save it:

List of nodes by category (wip)
    # bases
    [libcst.BaseAssignTargetExpression,],
    [libcst.BaseAugOp,],
    [libcst.BaseBinaryOp,],
    [libcst.BaseBooleanOp,],
    [libcst.BaseComp,],
    [libcst.BaseCompOp,],
    [libcst.BaseCompoundStatement,],
    [libcst.BaseDelTargetExpression,],
    [libcst.BaseDict,],
    [libcst.BaseDictElement,],
    [libcst.BaseElement,],
    [libcst.BaseExpression,],
    [libcst.BaseFormattedStringContent,],
    [libcst.BaseList,],
    [libcst.BaseNumber,],
    [libcst.BaseParenthesizableWhitespace,],
    [libcst.BaseSet,],
    [libcst.BaseSimpleComp,],
    [libcst.BaseSlice,],
    [libcst.BaseSmallStatement,],
    [libcst.BaseStatement,],
    [libcst.BaseString,],
    [libcst.BaseSuite,],
    [libcst.BaseUnaryOp,],

    # binary ops
    [libcst.BinaryOperation,],
    [libcst.Add,],
    [libcst.AddAssign,],
    [libcst.Subtract,],
    [libcst.SubtractAssign,],
    [libcst.Multiply,],
    [libcst.MultiplyAssign,],
    [libcst.Divide,],
    [libcst.DivideAssign,],
    [libcst.FloorDivide,],
    [libcst.FloorDivideAssign,],
    [libcst.Power,],
    [libcst.PowerAssign,],
    [libcst.Modulo,],
    [libcst.ModuloAssign,],
    [libcst.MatrixMultiply,],
    [libcst.MatrixMultiplyAssign,],
    [libcst.BitAnd,],
    [libcst.BitAndAssign,],
    [libcst.BitOr,],
    [libcst.BitOrAssign,],
    [libcst.BitXor,],
    [libcst.BitXorAssign,],
    [libcst.LeftShift,],
    [libcst.LeftShiftAssign,],
    [libcst.RightShift,],
    [libcst.RightShiftAssign,],

    # unary ops
    [libcst.UnaryOperation,],
    [libcst.BitInvert,],
    [libcst.Minus,],
    [libcst.Plus,],

    # comparisons
    [libcst.Equal,],
    [libcst.NotEqual, st.just("!=")],
    [libcst.GreaterThan,],
    [libcst.GreaterThanEqual,],
    [libcst.LessThan,],
    [libcst.LessThanEqual,],

    # boolean ops
    [libcst.BooleanOperation,],
    [libcst.And,],
    [libcst.Or,],
    [libcst.Not,],

    # identity
    [libcst.Is,],
    [libcst.IsNot, infer, nonempty_whitespace, infer],

    # membership
    [libcst.In,],
    [libcst.NotIn, infer, nonempty_whitespace, infer],

    # built-in types
    [libcst.Float,],
    [libcst.Integer,],
    [libcst.Imaginary,],
    [libcst.SimpleString,],

    # strings
    [libcst.ConcatenatedString,],
    [libcst.FormattedString,],
    [libcst.FormattedStringExpression,],
    [libcst.FormattedStringText,],

    # built-in structures
    [libcst.Dict,],
    [libcst.List,],
    [libcst.Set, nonempty_seq(libcst.Element, libcst.StarredElement)],
    [libcst.Tuple,],

    # non-whitespace tokens
    [libcst.Comma,],
    [libcst.Colon,],
    [libcst.Dot,],
    [libcst.LeftCurlyBrace,],
    [libcst.LeftParen,],
    [libcst.LeftSquareBracket,],
    [libcst.RightCurlyBrace,],
    [libcst.RightParen,],
    [libcst.RightSquareBracket,],
    [libcst.ParenthesizedWhitespace,],
    [libcst.Semicolon,],

    # whitespace tokens
    [libcst.EmptyLine, infer, infer, infer],
    [libcst.Newline,],
    [libcst.SimpleWhitespace,],
    [libcst.TrailingWhitespace, infer, infer],

    # expressions
    [libcst.Call,],
    [libcst.Lambda,],
    [libcst.Expr,],
    [libcst.Slice,],

    # keywords
    [libcst.Del,],
    [libcst.Assert,],
    [libcst.Break,],
    [libcst.Continue,],
    [libcst.Ellipsis,],
    [libcst.Pass,],
    [libcst.Raise,],
    [libcst.Return,],
    [libcst.Yield,],

    # conditions
    [libcst.If,],
    [libcst.Else,],
    [libcst.IfExp,],

    # loops
    [libcst.For,],
    [libcst.While,],

    # match statements
    [libcst.Match,],
    [libcst.MatchAs,],
    [libcst.MatchCase,],
    [libcst.MatchClass,],
    [libcst.MatchKeywordElement,],
    [libcst.MatchList,],
    [libcst.MatchMapping,],
    [libcst.MatchMappingElement,],
    [libcst.MatchOr,],
    [libcst.MatchOrElement,],
    [libcst.MatchPattern,],
    [libcst.MatchSequence,],
    [libcst.MatchSequenceElement,],
    [libcst.MatchSingleton,],
    [libcst.MatchStar,],
    [libcst.MatchTuple,],
    [libcst.MatchValue,],

    # generators/comprehensions
    [libcst.GeneratorExp,],
    [libcst.CompFor,],
    [libcst.CompIf,],
    [libcst.DictComp,],
    [libcst.ListComp,],
    [libcst.SetComp,],

    # exceptions
    [libcst.Try,],
    [libcst.ExceptStarHandler,],
    [libcst.Finally,],

    # imports
    [libcst.From,],
    [libcst.ImportAlias,],
    [libcst.ImportStar,],

    # functions/parameters
    [libcst.FunctionDef,],
    [libcst.Param,],
    [libcst.Parameters,],
    [libcst.ParamSlash,],
    [libcst.ParamStar,],

    # elements (?)
    [libcst.Element,],
    [libcst.DictElement,],
    [libcst.StarredElement,],
    [libcst.SubscriptElement,],

    # comments
    [libcst.Comment,],

    # ??
    [libcst.Annotation,],
    [libcst.Arg,],
    [libcst.AssignEqual,],
    [libcst.AssignTarget,],
    [libcst.AugAssign,],
    [libcst.ClassDef,],
    [libcst.ComparisonTarget,],
    [libcst.Index,],
    [libcst.Module],
    [libcst.Name,],
    [libcst.NameItem,],
    [libcst.SimpleStatementLine,],
    [libcst.SimpleStatementSuite,],
    [libcst.StarredDictElement,],
    [libcst.TryStar,],
    [libcst.WithItem,],

I believe I can start by adding the built-in types nodes (like integer)! edit: Ah, no, integers and floats are already registered ๐Ÿ˜Š Well, lets go with a tuple then, it should be just the same as sets.

Nevermind, Iโ€™ve taken the full list of nodes and am ordering them by categories (binary ops, unary ops, boolean ops, etc.). This is quite straightforward, I should be able to answer these questions by myself ๐Ÿ™‚

After reading your comment a second time and carefully looking at the REGISTERED list and how itโ€™s use in the loop below, it all makes sense ๐Ÿ™‚

Since I already played with each type of node ast provides, I believe Iโ€™ll be able to add strategies for other libcst nodes, and, as you suggested, start with just one in a first PR ๐Ÿ˜„

An idea for following the true grammar: maybe we can augment the PEG parser generator we use for Python 3.9 and newer to spit out the grammar definition in a way thatโ€™s consumable by hypothesmith (or maybe lark?).

  • ๐ŸŽ‰
  • base nodes are resolved as a union of their concrete subclasses, so itโ€™s fine to infer from them but you should avoid registering a strategy directly for them
  • there may in fact be trees representing invalid code, which we can still construct without validation errors (see e.g. https://github.com/Instagram/LibCST/issues/531). At runtime these are discarded by the .filter(compilable) check, but it might be nice for LibCST if we ran some tests without this in order to report bugs upstream.
  • Remember, PR for a single node to start! ๐Ÿ˜‰