Skip to content

Commit a8f5771

Browse files
authored
tools: improved newgtlds.go, removed replace-between. (publicsuffix#1204)
== tools: improved newgtlds.go, removed replace-between. - publicsuffix@d996dad Previously the `tools/newgtlds.go` utility was difficult to use with the new Github action based pull-request workflow because it _always_ updated the timestamp in the header comment in the gTLD section of `public_suffix_list.dat` when it was run, even if no gTLD data changed. This meant there was a diff in the workdir after the tool ran that would cause a PR to be opened by the action. See publicsuffix#1166 for more discussion/background. In the old travis version of the automation I used a crude shell pipeline to exclude the header comment when deciding if there was a diff or not. With the new action it's not possible to change how the diff status is determined (without re-inventing the pull request action in-repo). So instead we need to abandon the timestamp in the gTLD section header comment (that would be sad) or, we need the script to be idempotent when there's no true data change. To make the tooling smarter about when it makes changes I reworked `newgtlds.go` to be able to consume the existing `public_suffix_list.dat` and to split apart the data templating and the header comment templating. Now the tool can exclude the header comment and compare against the existing data to determine if there is a real data change or not. If there's no change in the data then the file is left untouched. If there is a change then the file is updated and only then is the header comment with the date stamp updated along with the new data. I've included unit tests that get 80% statement coverage of this new tooling and also tested it locally with success. Conveniently this rework also lets us remove the `tools/replace-between` Perl script reducing the number of languages in play for this process. == CI: re-enable github tld-update workflow - publicsuffix@7856839 Now that the tooling won't produce PRs without meaningful data updates we can re-enable the `tld-update.yml` workflow. == CI: run go unit tests in tld-update workflow - publicsuffix@f6749a5 Before using the `tools/newgtlds.go` tooling to open a PR we should run the unit tests to make sure the code works the way we expect. This will also help catch any bitrot.
1 parent eee9fb9 commit a8f5771

File tree

7 files changed

+825
-101
lines changed

7 files changed

+825
-101
lines changed

.github/workflows/tld-update.yml.hold renamed to .github/workflows/tld-update.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,9 @@ jobs:
1919
with:
2020
go-version: ^1.15
2121

22+
- name: Run unit tests
23+
run: go test tools/*.go
24+
2225
- name: Set current date
2326
id: get-date
2427
run: echo "::set-output name=now::$(date +'%Y-%m-%dT%H:%M:%S %Z')"

.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
linter/log
22
libpsl
3-
3+
coverage.out

.travis.yml

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,12 @@
11
language: c
22
compiler: gcc
33

4-
script: make
4+
script:
5+
- make
6+
- go test -v -coverprofile=coverage.out tools/*.go
7+
8+
go:
9+
- "1.15.x"
510

611
addons:
712
apt:
@@ -18,4 +23,3 @@ addons:
1823
- libicu-dev
1924
- libunistring0
2025
- libunistring-dev
21-

0 commit comments

Comments
 (0)