yard: Question: checking common documentation errors for CI

Hey.

What I’m trying to achieve currently (on large codebase constantly changed by hundreds of people) is adding continuous integration check for YARD docs. What should be checked, ideally:

level 0: No dumb errors (@param vs @params, documentation for parameter that is not there and so on);
level 1: “You have changed file X, but it has undocumented methods/classes/modules now”;
…more to come…

Let’s skip levels 1+ for now completely (just a mention for future discussions) and focus on level 0. yard command currently has two levels of error reporting (amirite?): [error] (can’t continue parsing this file) and [warn] (file is parsed but not everything is allright). Most of warn are indeed useful, so, it almost looks like “if there are any warnings, YARD CI step is failed”, but there is one case, looking like this:

[warn]: in YARD::Handlers::Ruby::MixinHandler: Undocumentable mixin: YARD::Parser::UndocumentableError for class CodilityService
[warn]:     in file '<some_file_name>.rb':3:

    3: include Rails.application.routes.url_helpers

As far as I understand, “official” solution for this is “just ignore all warnings”, which also gets rid of things like "Unknown tag @params", "Cannot resolve link to", "@param tag has unknown parameter name" which are pretty useful, to say the less.

So, to be honest, my “ideal” feature request would be:

distinguishable warnings (and ability to turn them off by special comment tags?..);
machine-ready output format (JSON?..), turned on by yardoc option;
special option for “just check, no docs generation” (or is it yardoc --no-save?).

…but I’m open to any advice and suggestion 😃

(Also, I’m not afraid and even willing to provide PRs and other contributions, if it could be appropriate.)

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 33 (14 by maintainers)

Commits related to this issue

Update multi-line log messages to single log call References #1007 — committed to lsegal/yard by lsegal 8 years ago
Add ability to modify log messages in callbacks Enables ability to transition YARD log calls away from inline log messages and log only the data required for callbacks to build the correct message. E... — committed to lsegal/yard by lsegal 8 years ago
Tag release v0.9.6 References: #1002, #1004, #1005, #1006, #1007, #1008, #1009, #1011, #1012, #1016, #1024, #1028, #1033, #1042, #1045, #1047, #1049, #1050, #1051, #1052, #799, #948, #995, #997... — committed to lsegal/yard by lsegal 7 years ago

Most upvoted comments

Great read. Thanks for the efforts @zverok

0x1eef on Aug 29, 2017

@lsegal Any plan to merge the structured-logging branch in the mainline? I am developing a small doc validation tool for our CI at work (in fact, borrowing a good deal of @zverok’s code – thanks!) and we’d love to gemify and open source the whole thing

lordofthelake on Feb 16, 2017

@lsegal Looks good! My attempts to use the new functionality also seem successful. The class looks like this. Sample output:

Running YARD documentation validator....

Files:        2327
Modules:       525 (  467 undocumented)
Classes:      2192 ( 1857 undocumented)
Constants:     968 (  758 undocumented)
Attributes:    259 (   12 undocumented)
Methods:      9536 ( 4349 undocumented)
 44.78% documented


Warnings
--------
(mistyped tags or other typos in documentation)

app/models/<some_nda_here>.rb:15: @param tag has unknown parameter name: foo
...
app/controllers/<some_nda_here>.rb:216: Unknown tag @internal
...
apq/<some_nda_here>.rb:240: Unknown tag @retun

Undocumentability notices
-------------------------
(related to metaprogramming: not to be fixed, but should be documented with directives)

app/services/<some_nda_here>.rb:3: Undocumentable mixin
...
lib/<some_nda_here>.rb:13: Undocumentable ATTRIBUTES

0 errors, 8 warnings, 8 notices (123.96 seconds to validate)

(And it exits with 2 on errors, 1 on warnings and 0 on what I’m calling “notices” here)

The only thing I’m disliking currently is a need for pretty cumbersome code from here and below, to re-format YARDS “free form” messages into more regular ones.

zverok on Aug 16, 2016

OK, I’ve investigated the matters for some time and have more focused question, about this (and similar) code line: https://github.com/lsegal/yard/blob/master/lib/yard/docstring_parser.rb#L210

if library.has_tag?(tag_name)
  @tags += [library.tag_create(tag_name, tag_buf)].flatten
else
  log.warn "Unknown tag @#{tag_name}" +                                 # <== HERE!
    (object ? " in file `#{object.file}` near line #{object.line}" : "")
end

What is my concern about it? It is that log.warn is a) unstructured and b) unconditional (except for global log level). So, nobody can build a tool that uses YARD’s parser which then gathers all the problems and return them in orderly manner; or, for example, nobody can do things like yardoc --ignore-warning UndocumentableMixin.

So, to my proposal. What do you think about introducing structured warnings mechanizm in YARD, so that aforementioned line will look like:

Warnings.add 'UnknownTag', tag_name, object

Then, default behavior would be the same: just print warnings the moment they appear; but that approach allows several good things to emerge:

option like --warning-format json which reports them in machine-readable format;
option like --ignore-warning UndocumentableMixin;
using YARD’s native parser for building documentation quality analysis tools (better than yardstick, which only can recognize whether known tag like @return is absent, but just silently ignores unknown or wrongly formatted ones).

Would you accept PR like that, being it properly done and tested?..

zverok on Aug 9, 2016