add test suite documentation

README.md

@@ -140,12 +140,17 @@ front end if there is significant interest.
 
 ## Testing
 
-Once the [test dependencies](#dependencies) are installed, tests can be run with
-`make test`. GitHub Actions is used to automatically test commits.
+Once the [test dependencies] are installed, tests can be run with `make test`.
+GitHub Actions is used to [automatically test] commits. Please review the [test
+documentation] for guidance on adding, debugging, and interpreting tests.
 
-There is also a [SystemVerilog compliance suite] being created to test
-open-source tools' SystemVerilog support. Although not every test in the suite
-is applicable, it has been a valuable asset in finding edge cases.
+[test dependencies]: #dependencies
+[test documentation]: test/README.md
+[automatically test]: https://github.com/zachjs/sv2v/actions
+
+There is also a [SystemVerilog compliance suite] that tests open-source tools'
+SystemVerilog support. Although not every test in the suite is applicable, it
+has been a valuable asset in finding edge cases.
 
 [SystemVerilog compliance suite]: https://github.com/chipsalliance/sv-tests
 

test/README.md

+# sv2v tests
+
+Tests are run via `make test` or by running `run-all.sh` directly. Each
+sub-folder here (except `lib`) is a suite of tests using shUnit2 as a test
+framework. Every suite contains a `run.sh` that performs its tests when run from
+within the suite's folder. The tests are run against the compiled `bin/sv2v` in
+the root of the repository.
+
+
+## Regular test suites
+
+These primary test suites collectively contain hundreds of test cases, each of
+which is converted using sv2v and simulated using iverilog. The test cases in
+these suites are automatically discovered. Each test case `foo` consists of
+three components:
+
+1. **`foo.sv`** is the SystemVerilog file to be converted by sv2v.
+2. **`foo.v`** is a manually-produced Verilog file that should behave
+   equivalently to `foo.sv`. For Verilog-compatible test cases, this file may be
+   omitted, in which case the runner uses the original `foo.sv` instead.
+3. **`foo_tb.v`** is a Verilog testbench provided to iverilog to exercise both
+   `foo.v` and the converted `foo.sv`. If this file is not present, the runner
+   assumes both `foo.sv` and `foo.v` contain a `module top`.
+
+For each test, the simulation results are compared by log output (e.g., from
+`$display` or `$monitor`) and by top-level VCD (excluding parameters). The test
+is repeated with the conversion performed in `--verbose` mode.
+
+All three source files are run through sv2v repeatedly to perform the following
+additional consistency checks:
+
+* Each file should convert with no errors or warnings.
+* When the converted output is converted again, it should not change.
+* `sv2v file.sv` should exactly match `sv2v --pass-through file.sv | sv2v -`.
+* The converted output should not contain certain SystemVerilog-only constructs
+  that iverilog happens to support in Verilog-2005 mode, such as `$bits`.
+
+Each regular test suite has a particular focus:
+
+* `basic` contains Verilog-compatible tests without a `.v`
+* `core` contains standard tests with at least a `.sv` and a `.v`
+* `lex` contains tests for lexing and preprocessing (e.g., macros)
+* `relong` contains tests provided by Reid Long in 2019
+
+Rarely, a file may have an adjacent `.pat` file (e.g., [unneeded_scope.sv.pat])
+used to assert the presence or omission of specific substrings in the converted
+output. This can be used to check aspects of sv2v's conversion behavior that are
+not apparent in simulation.
+
+[unneeded_scope.sv.pat]: core/unneeded_scope.sv.pat
+
+
+## Specialized test suites
+
+The `error` suite tests inputs that should cause sv2v to output an error. The
+suite runs each `.sv` file in the folder and ensures that sv2v returns a
+non-zero exit code and outputs something to `stderr`. Each test may contain a
+`// pattern: ...` and a `// location: ...` flag at the top. When present, the
+test runner checks if the `stderr` contains the given error and location
+information.
+
+The `nosim` suite simply checks that each of the `.sv` files can be converted by
+sv2v, without evaluating the converted output. This is used to provide limited
+coverage of language constructs that sv2v can output but that are not supported
+by `iverilog`.
+
+The remaining test suites have a custom `run.sh` that defines a list of test
+procedures that may not correspond directly to the other files in the folder.
+Many of these suites test a particular feature of the sv2v CLI.
+
+* `define` tests `-D`/`--define`
+* `dump` tests `--dump-prefix`
+* `help` ensures the `--help` output in the README is up to date
+* `keyword` tests `begin_keywords` version specifiers
+* `number` generates and tests short number literals
+* `siloed` tests `--siloed` and default compilation unit behavior
+* `truncate` tests number literal truncation and `--oversized-numbers`
+* `warning` tests conversion warnings
+* `write` tests `-w`/`--write`
+
+
+## Adding new tests
+
+Most tests for bug fixes or new language features are added to the `core` suite,
+with a `.sv` and a hand-converted `.v`. New CLI features typically get a new
+specialized test suite. All test files use 4 spaces for indentation. Macros or
+`.vh` files may be used to reduce duplicated code where appropriate.
+
+
+## Debugging test failures
+
+* There may be incompatibilities with a specific version of iverilog. Please see
+  the [main GitHub Actions workflow] to determine what version of iverilog is
+  currently used in CI.
+* It is often helpful to run a subset of the tests in the large suites (e.g.,
+  `core` and `error`) by passing a list of file or base names to `run.sh`. For
+  example, `./run.sh missing_join` or `./run.sh interface_*.sv`.
+* Review [functions.sh] to disambiguate test failure messages.
+* Many test cases can be run outside of the test harness, e.g.,
+  ```
+  bin/sv2v test/core/inc.sv > foo.v && iverilog -g2005 foo.v && ./a.out
+  iverilog -g2005 test/core/inc.v && ./a.out
+  rm foo.v a.out # when done
+  ```
+* Consider diffing the output against a prior build of sv2v, e.g., `diff <(sv2v
+  test/core/inc.sv) <(bin/sv2v test/core/inc.sv)`.
+
+[functions.sh]: lib/functions.sh
+[main GitHub Actions workflow]: ../.github/workflows/main.yaml

test/relong/README.md

-# relong Tests
+# Reid Long's tests
 
-These tests are borrowed from Reid Long's [HDL Examples
-repository](https://bitbucket.org/ReidLong/hdl-examples). That repository was
-intended to provide examples for how the conversions in this project could be
-done. sv2v does not necessarily convert code as demonstrated in the examples.
-Notably, sv2v does not create `generate` blocks when converted vectors with
-multiple packed dimensions, uses `localparam`s rather than macros for `enum`
-conversion, and converts `struct` literals to concatenations, rather than
-multiple statements.
+These tests are borrowed from Reid Long's [HDL Examples repository]. Early in
+sv2v's development, Reid produced these examples to demonstrate how sv2v might
+perform its conversions. sv2v does not necessarily convert code in the same way,
+but its output should behave equivalently. Notably, sv2v does not create
+`generate` blocks when flattening vectors with multiple packed dimensions, uses
+`localparam`s rather than macros to convert `enum`s, and converts `struct`
+pattern literals as concatenations, rather than multiple statements.
 
-Each test case (say, "foo") is comprised of the following files:
-
-1. `foo.sv`: original SystemVerilog
-2. `foo.v`: hand-converted Verilog
-3. `foo_tb.v`: basic testbench exercising the given modules
-
-The SystemVerilog source file is converted to Verilog using sv2v, and then both
-the converted file and the reference Verilog are simulated using Icarus Verilog.
-This produces VCD files for each which are expected to match exactly, except for
-the timestamp.
+[HDL Examples repository]: https://bitbucket.org/ReidLong/hdl-examples
 
 ## Modifications
 
@@ -29,7 +19,7 @@ corresponding versions in the source repository.
    Though some tools allow for stray semicolons, `iverilog` does not.
 2. `array.v` previously had a custom implementation of `$clog2`, which was
    removed.
-3. `cache_request.sv` was modified to include a plain decimal literal to ensure
+3. `cache_request.sv` was modified to include a plain decimal literal to provide
    coverage beyond the unbased-unsized literals.
 4. The `cache_request2` test is omitted. It was only an example for debugging a
    VCS-specific issue encountered with `cache_request`.