Lambda the Ultimate

If you haven't dealt with heavyweight industrial parsing before (of which C++ is the standard candle), reread the above comment. I'm going to embrace and extend the thoughts there, but you should get the main point in Sean's concision before moving on to my verbosity.

The underlying theme is that "industrial parsing is hard (expensive), so know what you need". If your output is destined solely for machines, you don't care about comment handling. If your output is for humans (such as javadoc or a translation), losing comments is intolerable. As mentioned, if you're paying then anything is possible, but overapproximation of feature needs is a dangerous financial approach. This answers your first question about standard practice: it's whatever is most effective getting the job at hand done without too much extra effort. Comment retention is rare in the literature and off-the-shelf machinery because those areas are dominated by parsers aimed for compilers.

Sean's other vastly important point is that when shopping for such technology, extensibility will be as important as any particular feature. Comment preservation will mean little if your source doesn't parse right in the first place. As mentioned, parsing is a seriously non-trivial problem for C++, especially in the context of the C preprocessor. Chances are you'll then be in the market for name resolution, and things get plenty more interesting from there on. To again wantonly abstract the premise, one has to think about fundamental limitations.* You can wall yourself off from practical extensibility by "baking in" systemic assumptions. Comment handling might be layered on to an existing parse system without undue pain, but you also have to consider, for example, whether the parser will scale to your codebase, whether it can handle multiple input/output languages simultaneously, etc. Some of these features are hard to come by if they weren't there from the get-go.

Per having a comment field in the parse data structure, this is exactly the solution I've used practice. Shap beat me to the punch on that, so I'll post details there.

* proper credit for these abstractions belongs to Ira Baxter, architect of DMS and my boss at Semantic Designs. We've apparently only been bitten by bad assumptions a couple times in fifteen years (10 million lines of code is big enough for anyone, right?)

CPP emits "#" directives to report filenames and line numbers properly. C compilers already deal with that and already decorate the AST with that information. They typically keep track of columns, as well.

To fully handle preserving comments and other pre-processor information, all that a typical parser needs to do is build an index that maps file/line/column positions to a number of datums: the extents of the previous, following, and enclosing comments (if any); the "include stack", and the "macro expansion stack". A "positition" as a compiler might emit it in an error message is then "file X, line Y, column Z, as expanded by the macro definition at file A, line B, column C, as expanded by [....], as included from [....]".

In other words, ASTs *already* contain (as node attributes) file / line / column information -- and that is completely sufficient to index a record of "what CPP did". It shouldn't "hair up" parsers or lexers, much at all.

A separate question is the idea of defining new languages in which comments are constrained as to their location and are explicitly attached to a specific AST node (e.g., a comment attached to a function definition or attached to the declaration of one of the parameters to that function).

-t

Yes, there are ways to preserve line-number information for error reporting. But I was talking about an AST that represents all the information in the original source code (as opposed to just a single preprocessing of it).

For example:

int f
#ifdef TEST
  = 2;
#else
  (int param) { return param * 2; }
#endif

What would the AST for this code look like?

I think that in-AST comments, such (comment "...") or Python docstrings, are the way to go. If you want to take comments into account when working on the program structure, they need to be part of the structure. Existing heuristics to attach free-floating comments to some nodes are fragile and we could design something more solid.

Comments are related to other construct that are not directly part of the core language : annotations and pragmas. Annotations have traditionnaly been attached to specific AST nodes, and we may handle comments as a special case of "meaningless" annotations.

I think there is a continuum between those features, which may or may not impact the execution, static semantics, or tooling of the program. Annotations that drive code generation (such as Python decorators) have strong semantics. Some compiler pragma may change the compilation optimizations, but should not change the observable semantics of the program. Other (@deprecated, @nowarn("foo")) should change the compiler behavior at compile time, and finally some are only useful for non-semantics tools such as documentation generators. In-code comments are the less structured of all.

Are those different instances of a generic design? Should they be handled uniformly? If not, where should the frontier stand?

The question would be how close can we get. In your example, the '+' operator would usually be parsed as the root node of the expression tree. The parser could incarnate this as something like

op_add:
  child[0] = parse('x')
  child[1] = parse('/* intervening comment  */')
  child[2] = parse('y')

That would still be ambiguous, since you would lose the information whether the comment was before or after the '+' in infix. But you retain the information that the comment was between operators 'x' and 'y'. Further comments could come in between, others might be added before 'x' and after 'y', always trying to keep them as far down the parse tree as possible. The child list would retain their relative order. That appears quite neutral to me and might be close enough, even for rewriting purposes, because comments are kept 'near by' and further decisions are delayed as much as possible.

In fact, if you install the CDT package for eclipse, and hand it input involving missing macro definitions, you will find that it produces a quasi-AST using very much this sort of idea.

My personal opinion is that it is enough to attach the comment to some plausible AST node (hopefully in a well-defined way) along with some information about the relative position between the comment and the AST node. Most of the reason to preserve comments is for the sake of transformation, and this would probably be enough. Perhaps more to the point, I'm not clear how to do better in principle.

Just as one possible example, every serious compiler defines a notion of "location" for every AST node. If one simply adds a linked list of comments to the AST node, each of which has a location, I think that might do fairly well.

This is indeed the industrial solution that we* use; comments are attributes that decorate abstract syntax tree nodes. There are several other considerations to make in the design and implementation, though:

• Does the comment attach to the token before or after? (we call these pre- and post-comments)
• Where do you attach file-global (header, footer) comments?
• Handling sequences of comments
• Where do they go under tree compaction? (concrete to abstract tree, should be automatic)
• Where do they go under tree manipulation? (transformation/rewrite, needs manual mangling)

We have answers to all these, though sometimes only as far as our customers' needs have pushed us.

* Semantic Designs

Adding preprocessor directives and comments to syntax trees is also worth a look.

A final pointer (just for the records) Douglas Crockford's Pratt parser for Javascript keeps track of comments within the AST, as properties of nodes (and not nodes themselves).