MEC Development Notes

如何构建ANTLR

大多数开发者只需要下载jar或者通过maven（使用ANTLR的antlr4-maven-plugin）使用ANTLR即可，不需要了解本页上的信息。如果你想要fork这个项目或者调整运行时代码生成，那么你需要自己构建ANTLR。这里有两个组件：

一个负责将文法翻译成目标语言表述的parser和lexer的工具
生成的parser和lexer所需要的运行时

从4.4版本开始，我们使用主目录下的一个名为bild.py的Python脚本来构建ANTLR，它从自己的库pull内容实现自举。你不需要了解内部的细节，不过代码其实也比较好懂。

在下面的例子中，我会假设根目录是/tmp。

获取源代码

第一步是从github上的ANTLR 4仓库获取Java源代码。你可以直接从github下载打包好的代码压缩包，但是最简单的方式还是直接用git克隆一份到本地硬盘：

$ cd /tmp
/tmp $ git clone https://github.com/antlr/antlr4.git
Cloning into 'antlr4'...
remote: Counting objects: 43273, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 43273 (delta 26), reused 0 (delta 0)
Receiving objects: 100% (43273/43273), 18.76 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (22419/22419), done.
Checking connectivity... done.

不过其实如果你需要支持所有的目标语言，那么得下载所有的项目：

cd /tmp
git clone https://github.com/antlr/antlr4.git
git clone https://github.com/antlr/antlr4-csharp.git
git clone https://github.com/antlr/antlr4-javascript.git
git clone https://github.com/antlr/antlr4-python2.git
git clone https://github.com/antlr/antlr4-python3.git

编译

首先，确保我们的环境是干净的：

/tmp $ cd antlr4
/tmp/antlr4 $ ./bild.py clean
bootstrapping; downloading bilder.py
target clean
bild succeeded

编译源代码很简单（但是要注意，你必须要先下载好目标repo才能保证编译通过，见下面描述）：

/tmp/antlr4 $ ./bild.py compile
target compile
require parsers
build compile
bild succeeded

如果你对编译过程中都执行了什么命令、这些命令的输出是什么感兴趣的话，可以看一眼bild.log文件：

/tmp/antlr4 $ head bild.log
[01/13/15 18:56:16 <module> ./bild.py:39 bilder.py:107] platform=darwin
[01/13/15 18:56:16 <module> ./bild.py:39 bilder.py:108] jdk={'1.6': '/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home', '1.7': '/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home', '1.8': '/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home'}
[01/13/15 18:56:16] require parsers
[01/13/15 18:56:16 require ./bild.py:74 bilder.py:370] require parsers
[01/13/15 18:56:16 antlr3 ./bild.py:67 bilder.py:402] java -cp /Users/parrt/.bild/jars/antlr-3.5.1-complete.jar org.antlr.Tool -o /private/tmp/antlr4/gen3/org/antlr/v4/parse /private/tmp/antlr4/tool/src/org/antlr/v4/parse/GrammarTreeVisitor.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/BlockSetTransformer.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/LeftRecursiveRuleWalker.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ANTLRLexer.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ANTLRParser.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ATNBuilder.g /private/tmp/antlr4/tool/src/org/antlr/v4/parse/ActionSplitter.g
[01/13/15 18:56:19 antlr3 ./bild.py:69 bilder.py:402] java -cp /Users/parrt/.bild/jars/antlr-3.5.1-complete.jar org.antlr.Tool -o /private/tmp/antlr4/gen3/org/antlr/v4/codegen -lib /private/tmp/antlr4/gen3/org/antlr/v4/parse /private/tmp/antlr4/tool/src/org/antlr/v4/codegen/SourceGenTriggers.g
[01/13/15 18:56:20 antlr4 ./bild.py:71 bilder.py:423] java -cp /Users/parrt/.bild/jars/antlr-4.4-complete.jar org.antlr.v4.Tool -o /private/tmp/antlr4/gen4/org/antlr/v4/runtime/tree/xpath -package org.antlr.v4.runtime.tree.xpath /private/tmp/antlr4/runtime/Java/src/org/antlr/v4/runtime/tree/xpath/XPathLexer.g4
[01/13/15 18:56:20] build compile
[01/13/15 18:56:20 require ./bild.py:74 bilder.py:378] build compile
[01/13/15 18:56:20 javac ./bild.py:81 bilder.py:452] javac -version
...

你也可以发现，编译脚本已经从网上下载了必要的jar文件，这个行为有点儿像maven：

$ ls ~/.bild/jars/
ST-4.0.8.jar            antlr-4.5-SNAPSHOT-complete.jar    bild-junit.jar
antlr-3.5.1-complete.jar    antlr-4.5-complete.jar        hamcrest-core-1.3.jar
antlr-3.5.1-runtime.jar        antlr-4.5-rc-1-complete.jar    junit-4.11.jar
antlr-4.4-complete.jar        antlr4-4.5-complete.jar

编译工具和运行时jar

由于编译脚本需要所有目标语言的代码（通常情况下是这样），所以你必须从网上下载好所有这些代码。我们将目标语言的代码分割为独立的代码仓库，这只是出于管理的目的，不意味着它们属于不同的项目。

下面列出了各个目标语言代码对应的github代码仓库（Java语言的代码直接包含在antlr4工具中）：

你可以运行如下的命令：

cd /tmp
git clone https://github.com/antlr/antlr4.git
git clone [email protected]:antlr/antlr4-csharp.git
git clone [email protected]:antlr/antlr4-javascript.git
git clone [email protected]:antlr/antlr4-python2.git
git clone [email protected]:antlr/antlr4-python3.git

然后回到antlr4目录，编译一个jar看看是不是一切正常。

/tmp $ cd antlr4
/tmp/antlr4 $ grep VERSION bild.py | head -2
BOOTSTRAP_VERSION = "4.4"
VERSION = "4.5-SNAPSHOT"
/tmp/antlr4 $ ./bild.py mkjar
target mkjar
require compile
require parsers
build compile
build mkjar_complete
Generated dist/antlr4-4.5-SNAPSHOT-complete.jar
require compile
Generated dist/antlr4-4.5-SNAPSHOT-complete.jar
Generated dist/antlr4-4.5-SNAPSHOT.jar
bild succeeded
/tmp/antlr4 $ ls dist
antlr4-4.5-SNAPSHOT-complete.jar    antlr4-4.5-SNAPSHOT.jar

你会看到这个命令会编译两次完整的jar。ANTLR 4.5代码里面包含有一个ANTLR 4 grammar，我们必须用之前的版本来编译它，因为我们不能假定当前版本可用。一旦我们用之前的版本（例如，4.4）编译出了有效的ANTLR 4.5，那么我们再用4.5重新编译一遍自己。

测试工具和目标语言

在执行这些测试之前，确保你已经下载了前面提到的所有目标语言代码。然后，使用如下命令测试所有的目标语言代码：

./bild.py tests

这条命令应该会占满你的CPU时间，因为它会并发测试所有内容。注意，我们已经发现有些测试会在并行执行C#目标测试时失败。它们在串行时总是成功的（例如，你的开发环境里就是这样执行的）。

你应该会见到这样的输出：

...
Testing Python3 ...
org.antlr.v4.test.rt.py3.TestCompositeLexers: 2 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSemPredEvalLexer: 7 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParseTrees: 8 tests, 0 failures
org.antlr.v4.test.rt.py3.TestListeners: 7 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLexerErrors: 12 tests, 0 failures
org.antlr.v4.test.rt.py3.TestCompositeParsers: 15 tests, 0 failures
org.antlr.v4.test.rt.py3.TestFullContextParsing: 15 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSets: 23 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParserErrors: 26 tests, 0 failures
org.antlr.v4.test.rt.py3.TestParserExec: 32 tests, 0 failures
org.antlr.v4.test.rt.py3.TestSemPredEvalParser: 26 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLeftRecursion: 95 tests, 0 failures
org.antlr.v4.test.rt.py3.TestLexerExec: 38 tests, 0 failures
Python3 tests complete
Testing JavaScript ...
org.antlr.v4.test.rt.js.node.TestBitSetWordSize: 1 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestCompositeLexers: 2 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSemPredEvalLexer: 7 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestListeners: 7 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParseTrees: 8 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLexerErrors: 12 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestCompositeParsers: 15 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestFullContextParsing: 15 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSets: 23 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParserErrors: 26 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestSemPredEvalParser: 26 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestParserExec: 32 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLexerExec: 38 tests, 0 failures
org.antlr.v4.test.rt.js.node.TestLeftRecursion: 95 tests, 0 failures
JavaScript tests complete
bild succeeded

使用Intellij构建ANTLR

我在开发时会直接用Intellij来构建ANTLR。必要的项目文件都在.idea目录和下述文件中：

$ cd /tmp/antlr4
$ find . -name '*.iml'
./antlr4.iml
./runtime/Java/runtime.iml
./tool/tool.iml

使ANTLR运行时jar符合OSGi

如果不符合OSGi，ANTLR运行时是不能在eclipse里使用的。这就很讨厌了。为了解决这个问题，我们在一个pull request中解释了OSGi涉及到的问题并介绍了如何修改jar的manifest文件来解决这个问题。

首先，需要下载bnd这个工具，为了使用方便，我们再给它起一个别名：

alias bnd='java -jar /usr/local/lib/biz.aQute.bnd-3.0.0.jar'

我们的常规mkjar bild target会生成如下的manifest：

Manifest-Version: 1.0
Implementation-Vendor: ANTLR
Implementation-Title: ANTLR 4 Runtime
Implementation-Version: 4.5
Implementation-Vendor-Id: org.antlr
Build-Jdk: 1.6
Built-By: parrt
Created-By: http://www.bildtool.org

Interesting. This prints exactly the existing jar manifest looks like:

$ bnd antlr4-4.5.jar 
[MANIFEST antlr4-4.5]
Build-Jdk                                1.6                                     
Built-By                                 parrt                                   
Created-By                               http://www.bildtool.org                 
Implementation-Title                     ANTLR 4 Runtime                         
Implementation-Vendor                    ANTLR                                   
Implementation-Vendor-Id                 org.antlr                               
Implementation-Version                   4.5                                     
Manifest-Version                         1.0

我运行如下的命令来生成符合OGSi的jar。

cd dist
bnd wrap --output antlr4-4.5-osgi.jar antlr4-4.5.jar

它生成的manifest文件是这样：

Manifest-Version: 1.0
Bnd-LastModified: 1421371533578
Build-Jdk: 1.6
Built-By: parrt
Bundle-ManifestVersion: 2
Bundle-Name: dist
Bundle-SymbolicName: dist
Bundle-Version: 0
Created-By: 1.7.0_25 (Oracle Corporation)
Export-Package: org.abego.treelayout,org.abego.treelayout.internal.util,
 org.abego.treelayout.internal.util.java.lang,org.abego.treelayout.inter
 nal.util.java.lang.string,org.abego.treelayout.internal.util.java.util,
 org.abego.treelayout.util;uses:="org.abego.treelayout",org.antlr.v4.run
 time;uses:="javax.print,javax.swing,org.antlr.v4.runtime.atn,org.antlr.
 v4.runtime.dfa,org.antlr.v4.runtime.misc,org.antlr.v4.runtime.tree,org.
 antlr.v4.runtime.tree.pattern",org.antlr.v4.runtime.atn;uses:="org.antl
 r.v4.runtime,org.antlr.v4.runtime.dfa,org.antlr.v4.runtime.misc",org.an
 tlr.v4.runtime.dfa;uses:="org.antlr.v4.runtime,org.antlr.v4.runtime.atn
 ",org.antlr.v4.runtime.misc;uses:="javax.annotation.processing,javax.la
 ng.model,javax.lang.model.element,javax.print,javax.swing,org.antlr.v4.
 runtime",org.antlr.v4.runtime.tree;uses:="org.antlr.v4.runtime,org.antl
 r.v4.runtime.misc",org.antlr.v4.runtime.tree.gui;uses:="javax.print,jav
 ax.swing,org.abego.treelayout,org.antlr.v4.runtime.tree",org.antlr.v4.r
 untime.tree.pattern;uses:="org.antlr.v4.runtime,org.antlr.v4.runtime.mi
 sc,org.antlr.v4.runtime.tree",org.antlr.v4.runtime.tree.xpath;uses:="or
 g.antlr.v4.runtime,org.antlr.v4.runtime.atn,org.antlr.v4.runtime.dfa,or
 g.antlr.v4.runtime.tree"
Implementation-Title: ANTLR 4 Runtime
Implementation-Vendor: ANTLR
Implementation-Vendor-Id: org.antlr
Implementation-Version: 4.5
Import-Package: javax.annotation.processing;resolution:=optional,javax.i
 mageio;resolution:=optional,javax.lang.model;resolution:=optional,javax
 .lang.model.element;resolution:=optional,javax.lang.model.type;resoluti
 on:=optional,javax.lang.model.util;resolution:=optional,javax.print;res
 olution:=optional,javax.print.attribute;resolution:=optional,javax.swin
 g;resolution:=optional,javax.swing.border;resolution:=optional,javax.sw
 ing.event;resolution:=optional,javax.swing.filechooser;resolution:=opti
 onal,javax.swing.tree;resolution:=optional,javax.tools;resolution:=opti
 onal
Originally-Created-By: http://www.bildtool.org
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.6))"
Tool: Bnd-3.0.0.201501151451

使用如下命令来测试生成的jar：

bnd print --verify antlr4-4.5-osgi.jar

Ok, added to mkjar target so happens automatically now. It downloads 2.4 bnd (not 3.0) and wraps.

编辑ANTLR github wiki

If you are a kind soul and would like to fix some of the documentation associated with building and managing handler, such as this page, you can fork antlr/antlr4 at github and then clone from YOUR_ID:

$ cd /tmp
$ git clone [email protected]:YOUR_ID/antlr4.wiki.git