Automate parser generation

[devnote-dev]

Everything in the src/ directory is generated by tree-sitter from grammar.js, so really this should be the only file we care about when it comes to grammar changes. I'm not sure if we can outright remove the directory from the repository as I haven't been able to find any examples that do so, however, the scala tree-sitter automates the updating of src/ files using a sync workflow. It looks like a straightforward process but I'm not entirely sure how that will affect contributions...

[keidax]

src/scanner.c is written by hand 😄

There are several benefits to keeping the generated files in source control:

• A user can download the sources and compile the parser without requiring the tree-sitter CLI as a dependency.
• The generated files can change when tree-sitter is updated. We can track the historical state of those files, without guessing about which tree-sitter version was used at the time.

The sync job is interesting. I guess the benefit would be automatically alerting us when a new version of tree-sitter changes how the files are generated?

[nobodywasishere]

I do agree that keeping the generated parser is beneficial. The major downside I've been experiencing when working with this repository though is that it tends to slow down the git vscode integration to a crawl, probably due to the massive diffs that are generated by the generated parser. I don't know if this is just an isolated issue with our tree sitter parser or a wider problem, or if I have something configured incorrectly locally.

[keidax]

Yeah, I also avoid opening parser.c because it freezes my editor...

I just checked out several other tree-sitter projects, and crystal's parser is the largest:

~/src λ ls -lh */src/parser.c
-rw-r--r-- 1 gabe gabe 9.7M Dec 15 17:33 tree-sitter-bash/src/parser.c
-rw-r--r-- 1 gabe gabe  17M Dec 15 17:33 tree-sitter-cpp/src/parser.c
-rw-r--r-- 1 gabe gabe  25M Dec 13 19:34 tree-sitter-crystal/src/parser.c
-rw-r--r-- 1 gabe gabe 1.5M Dec 15 17:32 tree-sitter-go/src/parser.c
-rw-r--r-- 1 gabe gabe 2.4M Dec 15 17:29 tree-sitter-javascript/src/parser.c
-rw-r--r-- 1 gabe gabe 2.5M Dec 15 17:27 tree-sitter-java/src/parser.c
-rw-r--r-- 1 gabe gabe 3.3M Dec 15 17:29 tree-sitter-python/src/parser.c
-rw-r--r-- 1 gabe gabe  15M Dec 13 21:16 tree-sitter-ruby/src/parser.c
-rw-r--r-- 1 gabe gabe 5.9M Dec 15 17:29 tree-sitter-rust/src/parser.c
-rw-r--r-- 1 gabe gabe  23M Dec 15 17:27 tree-sitter-scala/src/parser.c

And our grammar isn't complete, so it's only going to grow.

I wonder if using git's skip-worktree feature would help?

[keidax]

A lot of relevant info is summarized here: https://github.com/sogaiu/ts-questions/tree/master/questions/should-parser-source-be-committed