-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
3672 lines (3672 loc) · 275 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
7ab98be (HEAD -> master, origin/master, origin/HEAD) Small fixes noticed when compiling on windows.
a76e3c9 Ignore hfst-eliminate-flags
1c8e521 pmatch: Make INPUT_MARK not special_symbols, so we can refer to it in pmscripts
432203d When verbose, print what goes into Like() operations
e80aade as_string() for symbols
93f3280 Allow negative weights
f1bc247 Merge branch 'master' of https://github.com/hfst/hfst
da7bcee Remove "." from internal punctuation & adjust url rule
2ae7765 Option to print weights in location mode
35557c0 Allow both escaped and unescaped zeros in multicharacters in all lexicon entries. Fixes issue #274.
640b91d Use c strings with printf.
34cbfc3 Allow both escaped and unescaped zeros in multicharacters in one-level lexicon entries. Partially fixes issue #274.
cc258f5 Make link to hfst-apertium-proc only if proc is enabled.
14309ef Add tool hfst-eliminate-flags.
d69404d Warn user if transducer file contains more than one transducer in hfst-optimized-lookup.
37bdede Expand @include"filename" directives as a preprocessing stage
732d645 Also reweight state that is both initial and final in hfst-reweight. Fixes issue #368.
6d9c495 Don't delete the implementation's FdTable
3048e03 Print warnings and error messages also in earlier stages of twol compilation.
a7f2714 Do not let distributed files depend on configuration.
cbf1e94 Update man files and include all of them in dist.
444c886 Remove nonexistent files from dist.
59e314d Add option --execute-and-quit (-E) to hfst-xfst.
b0a2be5 Merge pull request #392 from frankier/fix-python-lookup-leak
6a7ac65 Fix memory leak in Python .lookup(...) method
99dc9ff Add returns in non-void functions
22448e0 Option --no-readline should be enough.
a6c923b Document datatype Location and function locate and make them visible in the API.
245ae4d Add missing file pmatch_tokenize.cc.
d6258bc Fix incorrect use of delete; Fix missing virtual dtor; Remove some (void)
a1eeead Fix typo
630fd7b Merge branch 'pmatch-implode-explode-fixes'
1afaca8 Make Implode() and Explode() allow symbol expansion
ae00e6b Preserve empty lines; Flush
f1ff773 (origin/pmatch-implode-explode-fixes) Merge branch 'master' into pmatch-implode-explode-fixes
295b126 Don't try to collect initial symbols if initial state is -1
87e74a6 Merge branch 'master' of https://github.com/hfst/hfst
01cfc9f Fix internal punctuation case
9073e21 Name transducers for langname instead of langcode in these scripts too
8df796e Correct handling of markup markers
a6e96d3 No need for an allocation there; Fix final read
bb8e4d4 Be more verbose in READMEs.
4de9548 Add urns to metashare articles.
856902d Make implode and explode accept more than just bare symbolsw
3184e2a Return a list of strings by default in tokenize.
18f88cc Use langname instead of langcode in transducer names
5f69423 Merge branch 'master' of https://github.com/hfst/hfst
61713be Fix indentation
b4f726d Replace containment implementations with correct ones
f7985c7 Add keyword arguments to PmatchContainer.tokenize.
75a69b3 Quote array expansions, otherwise they break on spaces
1d0d062 Use * to concatenate array to string
f3b5acb Quote grep pattern so shell won't interpret
859de38 Convert dos2unix line endings
af328c4 Force type
8b297fd Protect line-based XML/HTML tags (WIP issue #379)
fe4c2f6 Modify tokenization implemented via PmatchContainer.
ceb5d2b Merge branch 'master' of https://github.com/hfst/hfst
d44f394 Don't free data pointer until after we've evaluated everything
80a2bcd Add package data, i.e. readme and licensing information for each language package.
1c463a0 Add metadata for each language package for download directory.
683a5f4 (tag: v3.14.0) Ready for release 3.14.0.
e824094 Update morphology tests so that they pass after small changes in pmatching and tokenization.
48e3c36 Move hacky foreign conjugation guessing to unknown token part
85ef061 Don't tag numbers after all
bcde7b9 One more weight typo
c815db6 Fix same typo again and merge changes from skeleton
ddac854 Foreign word conjugations
eb8efa4 Annotate numerical expressions
e980a90 Adjust indentation
1231580 Fix typo
9b8e3bb Merge branch 'master' of https://github.com/hfst/hfst
4b215fc Use language-specific patches when provided
9365b6c Bugfixes related to internal punctuation
ce5a05a Some language-specific customisations to the tokenizer
c685c41 Implement Ins() arcs inside right contexts
de9cc37 Fix to xerox output mode with plain, unanalyzed tokens
c4056f8 Normalize unicode apostrophies
4f2a4ce Rename morphology directory accordingly.
fcc122b Get rid of *new in new morphology tests.
d5e87db Rename old morphology tests as *old.
904647e Tentatively add tests for upcoming new morphology scripts.
e019c23 Merge branch 'master' of https://github.com/hfst/hfst
86f98c8 When uniqifying without regard for weights, make sure to sort by weights
857624e Return a skip value if tool is not found.
47ba60b Update morphology tests: omit deprecated .sh extensions and omorfi-analyze/generate. Do not include python tests in tool test.
3ab9303 Add missing file.
ffd1797 Add missing files to dist.
1032bf7 Explicitly list all functionality tests in dist.
90e161a Implement a cautious form of first-input-symbol scanning
9730a4e Correct variable name
a450d58 When not printing weights and in --unique xerox mode, suppress weight differences
4b05205 --uniq does not require an arg!
85f305e Check for tokenizer, not analyzer
27ec41d Remove duplicate analyses
4295c84 Improve punct-handling
95f49df Add paragraph-tokenizing newline behaviour to --xerox too
82165de Migrate to using hfst-tokenize --xerox instead of hfst-lookup
0bef94d Not generating analyzers anymore
00a9cf9 Reflect new --xerox functionality
d0945df Don't use the fancy Omorfi tokenizer
393407e Nuke old contents of installable dir before proceeding
9863b09 Redirect stdout from command, not stderr..
7b4042d Check that root path was set
251108f Add & parallelization to Finnish
b470ec4 Fix tests for existing programs
746c069 Wait for everything to finish before making replacements
6f7ca8e Make test expect result of previous commits
ff232a4 Utility script for building everything at once
c02ab2d Check for programs and give info if missing
0298661 Unify blankline behaviour of --finnpos and --tokenize
9c45fc9 Add function HfstTransducer::get_initial_input_symbols.
633fee2 Rewrite morphology tests.
3fc0c43 Add stubs for collecting initial symbols
d1191f9 Handle default arcs in pmatch & fix spelling of initial symbols variable
cb6e3c3 Fix definition of unknown_token
51b9c41 Simplify main morpho-tokenization
6d08f5e Remove messy first-possible-symbol detection code, read it from hopefully precompiled info instead
0538232 Add clarifying comment
13a1424 Get a bit more speed by pre-reserving space for the vectors
efeb703 Implement reading vector data from .bin binary format (much faster)
98e6697 Ins() the morphology check
1fa8784 When we have a dictionary word, don't override it
f6c33a5 When returning from a RTN, don't overwrite the original local variables!
6413d71 Add a script for testing the new morphology packages.
38af4ee Handle weight cutoffs inside pmatch
f35c5e2 Fix help message
28d8ce1 Optimized-lookup does work with hfst3 headers, so let's support it
9b53cb3 Add universal tokenizer
253d3e6 Disjunct empty string so transducers "know" about sentence boundaries
9d517a6 Improve usage notes
8b56ed6 Merge branch 'master' of https://github.com/hfst/hfst
132ff95 Link to github wiki instead of kitwiki.
f3858d0 Note about weights in existing morphologies
381bad5 Change default to weighted
13a43f4 Add notes about existing morphologies
241153b Correct name of temp morphology for building tokenizer
3d2ef76 Add option to build installable dir in-place in specified dir
548a137 Add options for capital casing and punctuation including
0e162d8 More elaborate tokenizer based on omorfi-tokenize
1949af2 General README
526cf4e Punctuation list
0e9d26d Merge branch 'master' of https://github.com/hfst/hfst
59df066 Add default tokenizer
0a0ebe2 Tentatively add match_and_print to python api as 'pmatch_tokenize'.
10e6565 Add some unicode quotes and relax the "don't know what to do" case
fc23a21 Try to guess whether we're at a sentence boundary or ordinal period.
334551e Handle colon separated conjugations differently to avoid size explosion
338b725 Cut out numerical parts of OmorFi to fix some bugs in it.
cbda671 Improve dash, hyphen & range handling
07e6f8d Assume that omorfi handles uppercasing correctly; add xml tags
54bd3c8 Merge branch 'master' of https://github.com/hfst/hfst
8232010 Add introspecion mechanism to PmatchObjects; use for optimization
e019322 travis: include macos
74da1a4 on macos, when bash is called as /bin/sh, echo prints '\x22' as "
c9eb7cb Use input symbols in lookup. Add a test for cases such as looking up string 'foo' in transducer [f:0 o:0 o:foo].
c71da22 Use only input symbols for tokenization in apply down. Add tests for cases where this matters, e.g. [f:foo o:0 o:0].
5da96ba Add one more test for compile-replace.
b1010cc Align regexp start and end tags with epsilons in compile-replace for better alignment between oridinary symbols.
220f031 Allow arbitrary symbols (except whitespace) in strings given as input to apply up and appy down commands.
8e154ae Implement variable 'retokenize'.
2db4ead Add precompiled tokenizers
ca6b412 Fix typo
54be190 Renamed to analyze-words and generate-words, for clarity
1a4d52c Add README
65994b0 Add missing files.
75fd90c Add function HfstInputStream::read_all and support an iterable object of several transducers in HfstOutputStream::write. Also fix some indentations.
7865f32 handle escaped @ as part of form, keep escape in output
693fe06 Add an extra parameter int max_number to HfstBasicTransducer::lookup, the default being -1 (infinite). Also fix a template issue with function compile_first of python API.
ba5ae81 Merge branch 'master' of https://github.com/hfst/hfst
d476a8e Update documentation link.
715eee4 More loop protection
c8d197c Endless loop protection
384ecbd hfst-proc: more fixes and tests for serial unicode ranges
eff4322 tests: use $TOOLDIR, things are not installed yet :)
3720c0c travis: make verboser check so we can see what failed
a3445f4 hfst-proc: seems like the test for serial unicode ranges was flipped
0dbbcdb Add preserving "plain" omorfi-based Finnish tokenizer.
339ccc7 If compile_sfst_file is given an empty string as filename, read from standard input.
ae77b0a Add option -H. Add symbol-harmonization to python tests.
6bdcea8 Specify exceptions thrown. Add functions to the interface. Replace quoted values 'True' and 'False' with True and False.
917801d Improvements to python command line tools.
040d2e9 Enable python tools.
9f3d164 Add hfst-strings2fst.py to tests.
43eab05 Add a skeleton for hfst-fst2strings.py.
d54e180 Add hfst-substitute.py.
4286e36 Add implementation for lexc compiler. It still fails with one test case.
bb15ded Implement hfst-fst2txt.py.
89a0ee2 Add compose intersect tool.
26d67f0 Add pmatch tools.
549e225 Implement python regexp parser.
29aabb6 Add tool for pushing weights.
132ba1c Add new python tools.
1a4ae93 Add hfst-subtract.py tool.
eb52063 Fix parameter handling and return values.
377d2eb Add a script for testing python command line tools. Add option --prolog to hfst-txt2fst.py tool.
c5e6cd5 Add hfst-tail python command line tool.
7b584ae Add new tools.
2050ec6 Add new python command line tools. Fix a typo in hfst-compose.py.
ec3d25a Return a skip value for python tests.
8223888 Tools hfst-fst2strings and hfst-head.py implemented.
5ec5cc7 Fix tests.
c5ca695 Fix scripts.
34e42a8 Fix rest of variables in tests.
e8de5ef Fix variables in scripts.
445d9e6 Modify scripts. Add a new python command line tool.
aae549f Add python versions of the tests.
bcd208d Add a python command line tool hfst-format.py.
8251357 Use string instead of char pointers in VarMaps and SVarMaps.
c14bf89 Check if foldername is the empty string and do not add a '/' in that case.
b58f63e Print an error message if input is in optimized lookup format.
12b5505 update man page about --giella-cg tag assumptions
701b032 Yet more python tools.
442d639 Add new (minimal) python versions of command line tools.
95dc1f2 Add option --python to rest of functionality tests.
f77e97f Add more --python options to functionality tests.
6c9aa62 Tentatively add option --python to functionality tests. If the option is used, the tests are run with python API, if available.
b17c286 Add a simple python implementation for hfst-calculate. Enable it with option --python in calculate-functionality.sh.
fd8dfea Keep separate RTN stacks for each depth level
29e9155 New morphology scripts
0f76826 Fix some typos, including 'kvargs' -> 'kwargs'.
c668609 correct home page url of HfstPushWeights
1e2c797 Merge pull request #365 from frankier/fix-print-dot-labels
b0d3218 don't tokenize_multichar as default in --giella-cg yet
c633117 put tokenize_multichar in TokenizeSettings; default true for --giella-cg
a189473 Update finnish morphology script.
c187d5a (tag: v3.13.0) Ready for release 3.13.0.
f6bc1cd -W/--no-weights overrides earlier -w (or -w implied by -g) options
cfa49ab Ensure print_dot labels are valid utf-8
cb9916d Add missing files.
c92bcaf Add missing files to dist.
d98ffa7 Initial commit of finnish-tagtools scripts Includes a few ones from FinnPos modified for our purposes
ee4662b Merge branch 'master' of https://github.com/hfst/hfst
d6a247e Added a variable "xerox-composition", default to "on"
355aaf9 allow creating HfstInputStream's from std::istream's
8ff49aa Use libreadline in hfst-pmatch when available
b5b7995 [].with(X = Y) feature in pmatch This provides support in the pmatch2fst compiler to define "global flags" with the syntax [your expression here].with(X = Y). This will set up a flag diacritic readable from anywhere within the brackets, even if Ins()ed.
0858e63 Support getting a list of operations involved with a particular feature. This also involves making define_diacritic() redefine the meaning of symbols if called multiple times on the same symbol number.
4d4471d tokenize tests should pass again
f81dcb5 tokenize --giella-cg: assume FST handles space around tags
b07c47b Merge branch 'master' of https://github.com/hfst/hfst
60e5c28 Check for success of the now-working test suite.
acf3c78 Cleanup of remainder of the pmatch test suite Many cases of adding set "need-separators off"; many cases of fixing mistaken test assumptions; one case of fixing code; one case of removing test that was already in pmatch2fst-functionality.sh; one case of removing entire test because it was trying to do something impossible; one case of relaxing test assumptions that should be possibly revisited. Multiple FIXMEs left but all the tests now pass.
2669274 Make both-sided OptCap() be actually both-sided. It still doesn't downcase the continuation of words, which is possibly a bug.
64a1874 tokenize --giella-cg: remove as_cg_tag, just need is_cg_tag now
9decc11 Modify check for list symbols. It was conflicting with left markers for Xerox rules (@LM@), causing a test to very surprisingly break (it didn't even seem to be using optimized-lookup format at all..)
4e4ba7c Add tests for Lst() and Exc()
7a0176a Avoid keeping an extra copy of the tokenizing dictionary in memory (fixes https://github.com/hfst/hfst/issues/362)
8c10e1d Consider list symbols (@L..." and "@X...") to be special
e4dff09 hfst-tokenize --giella-cg, is_tag: check if symbol Multichar_symbol
3ec59da tokenize --giella-cg: non-verbose as default
862ea0e Add C and C++ dlls to dist.
d653123 Fix misspelling of nrc for nlc
c005a4a Merge branch 'master' of https://github.com/hfst/hfst
86dcec6 Fix runtime handling of contexts and compilation of negative contexts. This was broken when the processing state stack was changed to accommodate "RTN safety". The local state is now pushed into a stack when entering a context and pushed again when exiting it, and popped when backtracking out of those situations.
10967c4 set need-delimiters no for more tests that were broken by it
06cbb20 Made Xerox output of unknowns follow Xerox
97ecf44 Merge pull request #359 from kartikm/patch-1
23dcc54 Fixed typo
134d5a5 Works around issue #358
21b0e82 Actually support multiple different Capture() names
be9e214 Instead of names, use pointers to the calling transducers for rtn calls
34bf445 Don't track returns from Ins() arcs in the call stack Pushing returns from Ins() calls to the call stack were clobbering nested calls. This is unnecessary, because we have the necessary information about the correct return location from the stack depth variable anyway.
1358576 Use hfst-pmatch --newline in the tests so we don't see irrelevant blank lines
e6f7067 Decrement recursion counter when returning from main loop due to no input
528952f A somewhat hacky fix to a bug introduced by changes to RTN processing pmatch is lacking recursion depth with each RTN entered. This is mostly a problem due to eventually running out of depth and truncating results. For now we reset depth at the start of each input, which makes sense anyway, but the leak should really be fixed.
c45f620 In locate mode, omit blank lines even if in blankline mode if we didn't print anything
74f3a6f Remember captures for the duration of the entire input. Also avoid copying the captured vectors around, instead passing iterators to input
01c2fd7 Fix short form of end tagging also disable automatic disabling of need-separators when encountering a context
f2b6c8c Merge branch 'master' of https://github.com/hfst/hfst
64528f7 In blankline-separated mode, keep blanklines in output too
b0a17a2 Merge branch 'master' of https://github.com/hfst/hfst
31f0767 Include pre-swig-generated wrappers to source distribution. Update README.
de22111 Remove bashisms that was breaking tests that apparently are not run with bash
368acef Fix tokenize backtrack regression, need to handle @PMATCH_BACKTRACK@ explicitly
da0ab47 Fix test which was probably written with a misunderstanding of replace rules
ed5ac3b Reinstate and fix some tests by moving them from pmatch-tests.sh (where escaping done by the testing script breaks some tests) to pmatch2fst-functionality.sh
882ee74 Allow term complement of named transducers
2686525 Merge branch 'master' of https://github.com/hfst/hfst
215a482 Support backreferences (Capture())
eea79b8 Update README.
e1d519c Link to exact place in page.
27d3e1c Update README.
2e93516 For installation instructions, link to PyPI pages instead of KitWiki.
b68f146 Update README.
7a3618e Make Like() and Unlike() much faster by not sorting the whole vocabulary (by doing an insertion sort of the n best words) and only calculating each comparison key once
85f8a8c Fixed binding of freely insert operator in hfst-twolc.
83eea8b Remove unnecessary (debug?) prints.
e10be33 Actually use other than ascii characters in att file.
5102d2d Fix some typos in test.sh. Replace epsilon with empty string in fst2strings.py. Read input in utf-8 format in prolog2fst.py, if possible.
394284b Fix a typo.
b52f348 Test if hfst-xfst exists after all options are processed.
cdaa748 Implement an equivalent python script for all command line tools needed in tests.
c6dae7a When using python for tests, also replace part of command line tools with equivalent python scripts.
88a70d3 Fix a couple of typos.
ef65667 Update python package version number. Use .cpp extension.
7bc8e18 Update README.
778ebca Get rid of unnecessary CommandLine in python api. Fix some typos.
1c345ef Error functions are void, not int.
b8edb3a Import sys.version_info before using it...
6a2107c Use extension cpp for all c++ files in pypi package. Also use the c++ implementation of foma backend by default.
4319046 A couple more fixes.
3f63a4e Fix some issues noticed by clang.
27c30ee Add a c++ version of foma backend for testing.
31c91c3 Add arguments to function declaration.
0b46845 Fix a couple more typos.
a117e05 Fix some typos.
7604850 Improve test script and add expected results for cases where xfst solution does not exist but the result from hfst script is checked.
c459207 Add two more python implementations for fsmbook tests.
c75cbb8 Merge branch 'master' of https://github.com/hfst/hfst
61be454 Add Unlike() operation, which could be better named also add option for whether information about cosine distances should be included in the result when compiling word vector operations
87cb7d6 Two more python tests...
42ba171 Add two more python tests.
221f8dc Add more tests.
e25e2d4 Add python test for FinnishNumerals
7d917a4 Allow empty string as input for hfst.fst and hfst.fsa and interpret it as epsilon.
3842501 Use xfst instead of python for slow tests.
5962b3a Add one more python test.
7b3dccb Add python test.
dd5ed1f Fix broken test assumption
a45a97d Prohibit EndTagging printable-empty lengths of input
77d5c7b Fix single-character transducers not getting automatic context separators
432d16c Verbose message about automatic separators
7bf07dd Make using Ins() -arcs safe Ins() arcs are no longer locally greedy, so there is no semantic difference between network inclusion with Ins() or by explicitly embedding. eg. define greedy "a+"; now the following work the same way: Ins(greedy) {abc} greedy {abc}
adc3b0b Use temp weights rather than subtractions like in pmatch
1378513 Fix test by having need-separators off
849d297 Minimize automatic delimiter contexts
97a7b05 Merge pull request #352 from unhammer/tokenise-lib
2fcc665 Merge branch 'master' into tokenise-lib
ef026f1 Add option --silent to tests.
c0996a8 Add .0000000000 to tests; ToDo: Trim trailing 0 after the dot, then trim the dot; ToDo: Investigate where std::fixed gets disabled
1fe2af9 Wrap compile_twolc_file inside a class as TwolcCompiler::compile.
dbd0d14 Revert "Removed rounding"
813f2d3 Add keyword arguments to compile_twolc_file.
786af19 Removed rounding
702425e Add twolc files to pypi setup.
30e3c4f Fix a typo.
e8acc1a Flush and close the stream.
2b26a3b Add tests for 'compile_twolc_file'.
b27f993 Remember to reset also the second parser.
7fb07f5 Add a possibility to reset twolc parsers between reading several twolc files.
251cef5 Add option --local-hfst to setup.py.
87a77e7 Add a tentative implementation of twolc compilation.
4567609 Make code clearer and add documentation.
8738272 Keep track of weights along context checking paths and unify weight handling
539592b Merge branch 'master' of https://github.com/hfst/hfst
31950de Add beam mode to result filtering
532925d Make sure TOP has the right name if it's eg. loaded from a @bin""
dc9bf4e Make sure the HfstTransducer name and the container names match
5289eb3 Fixes issue 353. Unbelievably, the return value for a weight-returning get_weight() was bool, which after casting mostly worked until it ran into negative weights.
2bfb55c Fix problem with rtns that are referred but not really present
2dd7767 Improve help messages of scripts.
1548480 Add hfst-specific options --with-c++11 and --without-c++11 to setup.py.
f285400 Use HFST_THROW macro instead of plain throw for HfstExceptions.
4be886d If USE_FOMA_EPSILON_REMOVAL is defined, use foma for epsilon removal in minimization for unweighted TropicalWeightTransducers to make it faster.
7b03fd0 Add an option --verbose to test.sh.
5463306 Fix the way unordered containers are used on windows with python2.7.
48de6fd Give pythonpath as first optional argument for each test, except for stream tests that take it as a second optional argument. Also specify coding for each test.
dbb055f round weight to zero decimals, non-sci, in cg/giellacg vislcg3 doesn't (yet) accept floating point weights
8d38925 Use by default c++11 unordered_map and unordered_set, unless otherwise specified via definitions (INCLUDE_TR1_UNORDERED_MAP_AND_SET and USE_TR1_UNORDERED_MAP_AND_SET).
31cd16c Merge branch 'master' of https://github.com/hfst/hfst
8f17d4e Add boolean CPP_STD_11=True to control c++11 support.
baec687 Move tokenize to correct alphabetical place.
194b9cc Couple of fixes to vc 2008 compilation without c++11 support.
38ad594 Fix issues noticed when compiling without c++11 support with VC 2008 on Windows.
89ddca2 Merge branch 'master' of https://github.com/hfst/hfst
8121be6 Fixed bug 341 (priority union)
318fdb5 Add an option --without-c++11 (defaults to 'no') to compile hfst without c++11 support.
d3a5c5d c++ standard is set in configure, do not set it here
ee84e29 make hfst-tokenise usable as a lib; include simple string→string fn
fbf8d49 Update documentation and setup.
9617edb Add a quick fix to missing 'strtof' in msvc 2012.
b22626e Add a function cross_product that takes an iterable object of transducers. Document other similar functions.
b1fbf8b Add simple help messages.
195ef73 Add a function 'compose' that takes a list of transducers. Comment out a debug print.
8cb5ee5 Skip calculate_funtionality.py test until it is fixed.
b9aa618 Add a script that copies files needed for creating a distribution for windows.
72b1463 Add a script for generating cc and hh files from yacc/flex sources for windows. Update copy script for windows. Change flex token ECHO to ECHO_ to prevent collision with flex/yacc macro with the same name.
0b61a94 Make script faster.
06a6c9d More fsmbook python scripts.
88cc5bf Add a simple --help message to test.sh. Add python versions of two tests.
3649018 Add new python scripts.
3ec8d35 If --python is requested in test.sh, use python API also for hfst scripts that have an equivalent python script.
0ebc87e Add options --python and --pythonpath to test.sh which enable compiling xfst files also via the python API.
7266f93 Add an example for HfstBasicTransducer.remove_transition.
f2a547f clang requires libc++ and osx version >= 10.7.
9a60444 Add function HfstBasicTransducer.remove_final_weight and document it. Also improve documentation of HfstBasicTransducer.remove_transition and HfstBasicTransducer.add_transition.
a2dbe8e Add function HfstBasicTransducer::remove_final_weight.
9950d90 Add option --restricted-mode (-R) to hfst-xfst. If it is requested, Write and read operations are allowed only in current directory (i.e. pathnames cannot contain '/' or '\') and system calls are disabled.
059b5b0 Add tests for hfst-xfst's 'negate' command.
cdab3f7 Add a function HfstTransducer::negate() which is equivalent to [?* - A] (where A is the transducer) with the exception that flag diacritics are treated as ordinary symbols. Use the function to implement 'negate' command of hfst-xfst.
4001b45 Allow complement/negation only for automata.
a741bdc (tag: v3.12.2) Ready for release 3.12.2.
56534b3 Add missing file.
cc2dea5 Merge branch 'master' of https://github.com/hfst/hfst
22383de Check more carefully which backends are actually enabled.
4cebb75 Actually remove commented code instead of making it visible.
c811b35 Remove commented code.
357ab0f Remove commented code.
f9796b3 Update README.
60742c4 Add extra compile option -std=c++0x.
25ce84e Reapply changes in commit de59747. Require at least automake 1.12, unless building from pre-yacc-generated sources.
750a3ca Revert changes in commit de59747. Allow automake older than 1.12.
0ae7b1e Use std::unordered_map instead of hash_map.
0b3a248 Use unordered_map instead of hash_map.
8b4b1d4 Skip lexc wrapper test.
d11b00e Add a simple exception handling mechanism.
02e16cb Fix a typo.
de59747 Stop supporting automake < 1.12. Warn that build will fail unless building with pre-flex-generated sources.
e776ecc Remove commented code. Remove unnecessary variables.
8b30936 Skip hfst-train-tagger and hfst-twolc-loc tools in version and help message testing.
ba28c39 Add missing file to dist.
b29e140 Remove lexc wrapper tests.
4033c44 Disable lexc and foma wrappers unless explicitly requested. Fix a typo in --enable-expand-equivalences.
5035132 Actually enable hfst-calculate and hfst-xfst by default as promised in commit 244b9b5...
31fc7c7 Add missing condition.
5191c88 Add option --with-openfst-log=lean to configure which supports only reading, writing and converting log openfst transducers.
30d3792 Actually check if --with-sfst=lean is configured when compiling sfst functions.
94eda73 Add configure option --with-sfst=lean which supports a limited set of sfst functions (reading, writing and converting between formats).
5df548e Add missing const to definition, remove second argument of HFST_THROW.
214ab27 Add function 'has_weights'. Make function 'get_profile_seconds' public.
a27f845 Add function 'get_profile_seconds' for profiling foma back-end.
1b64f33 Add function 'has_weights' and make function 'get_profile_seconds' static.
244b9b5 Remove duplicate of AC_ARG_ENABLE([xfst]). Enable hfst-calculate and hfst-xfst by default.
e47f010 Remove commented code. Make sure that code compiles when openfst backend is disabled. Add comments to some functions.
d0a1e91 Use iosfwd header instead of iostream when possible. Try to include only header files that are actually needed.
389679d Clean temporary files generated by tests. Add some files to be ignored.
610572f Merge branch 'master' of https://github.com/hfst/hfst
f66f396 Check for nested context conditions, ignoring the inner ones
d8db552 Update man pages. Remove man pages of tools that are no longer supported. Update copyright year.
62da656 Disable hfst-twolc script and hfst-train-tagger tool unless explicitly enabled with --enable-twolc-script and --enable-train-tagger.
e369608 Add files to be ignored.
142bfca By default, implement hfst-twolc as a single program and disable the script (can be controlled via --enable-twolc-script).
98e08f6 Get rid of hfst-twolc-system tool. The script hfst-twolc will soon be replaced by a single program.
e30202c Update windows scripts according to recent changes in twolc processing.
d676043 Update files ignored by git.
ae820a5 Revert to earlier version of htwolc-main.cc which calls twolc parsers directly instead of TwolcCompiler class. The latter approach sometimes causes a segmentation fault.
d96b441 Define warning and error streams in TwolcCompiler constructor.
9c0cca4 Throw an error instead of calling exit(1) when parsing input.
b820f11 Handle twolc error and warning streams.
e116a74 Add TwolcCompiler class and use it in htwolc program. Move CommandLine under libhfst/src/parsers.
9a19088 Move most twolc code from tools/src to libhfst/src/parsers.
0705bec if no --superblanks, newlines won't be in [], so need to print on unblanked newlines
1351257 --giella-cg: only treat superblanks if given --superblanks
efa0964 Update list of files ignored by git.
19fd1d1 Add an option to use htwolc in twolc tests. Disable it by default.
59cfdce Rename twolc files.
a6029ef Rename twolc parser and lexer files. Move functions under namespace hfst::twolcpre[1|2|3].
5844621 Actually exclude htwolc from installation.
7661b38 Add an alternative implementation of hfst-twolc which does all processing in a single program. Exclude it from installation until it is properly tested.
52d5257 only warn for backtrack-on-substrings-without-analyses if --verbose
3936fdb Control output of htwolcpre1.
0cc7582 Change names of extern twolc variables and add functions to access some of them.
f9a69b2 Add missing new file hfst_sfst_extensions.cc.
d5baf5a Separate twolc parsing and command line tools to their own files.
035b6ab Use separate prefixes in twolc lexers and parsers.
d3c4de9 Handle '^<' and '^>' operators in lexc regular expressions correctly. Should fix issue #346.
6165347 Space-separated mode
144a3f3 Ignore generated sfst source files.
dc90602 Merge branch 'master' of https://github.com/hfst/hfst
9f7208b Improve pmatch compilation error handling, still leaks memory at exit
3767b51 Add new function 'compile_sfst_file'.
512245b Merge branch 'master' of https://github.com/hfst/hfst
4649b83 Reorder and add norm caching
460cded clearer error message for when there is backtracking into substrings that have no analyses
0705c3c Apply recent changes in filenames and locations.
3560d0b Move sfst compilation from tools/src to libhfst/src/parsers.
bc49e55 Fix namespace and class names.
9f356e7 Change file and namespace names.
3f8d457 Small fixes to sfst compilation.
c28d2c5 Further separate sfst compilation and command line program.
0ccd6fe Reorganize sfst compilation into separate compilation units.
b9b8585 Fix xfst and twolc tests. Use rm -f when removing files that are conditionally created.
52d3cbf Remove all exe extensions from hfst-twolc and hfst-twolc-loc because they are scripts.
99d37e6 Remove exe extensions from hfst-twolc and hfst-twolc-loc because they are scripts.
b9b678e Start separating sfst parser and command line program to their own files. Add a prefix 'sfst' to flex/bison functions and variables.
9733812 Rename HfstCompiler, hfst-compiler and hfst-scanner to SfstCompiler, sfst-compiler and sfst-scanner.
ea9fb6b Add a test for hfst.regex using 'definitions'.
2a7ad84 Add a keyword argument 'definitions' to regex that takes a dictionary mapping transducer names into transducers.
6c76128 Add regexp operator documentation also to web pages.
2322ecc Modify regex documentation.
3155955 Tentatively add documentation about regexp operators.
77b6444 use ccache if possible, slightly faster travises
1ab384d Update fst_to_fsa and fsa_to_fst. Support both HfstTransducer and HfstBasicTransducer formats.
663224d Catch exceptions when compiling pmatch expressions. Add new tests for pmatch functions.
43e0644 Do not exit on pmatch compiling errors, throw 1 and catch it in command line program.
0556161 Merge remote-tracking branch 'refs/remotes/origin/master'
51b811e Small math fix + revert to float, which wasn't causing problems after all
e1c9af0 Modify fst_to_fsa and fsa_to_fst.
aaf2f8b Support special symbols and single-character symbols without separator in fsa_to_fst. Update also documentation.
ebea326 Add simple tests for fst_to_fsa and fsa_to_fst. Also test modifying transitions of an HfstBasicTransducer.
ecb78d0 Rewrite HfstBasicTransducer's transitions() and state_and_transitions(). Tentatively add an implementation for functions fst_to_fsa and fsa_to_fst.
a65ca94 Remove HfstBasicTransitions and HfstStates from HfstBasicTransducer and use hfst::implementations::HfstBasicTransitions and hfst::implementations::HfstBasicStates instead.
4579f7d Give the pythonpath as command line argument for tests that use 'from __future__ import' instead of catenating it to the beginning of file. The import must happen in the beginning of file.
3dda726 Rename HfstTransitions to HfstBasicTransitions.
68155b3 Add docstrings and dummy implementations for functions fst_to_fsa and fsa_to_fst.
4a2814d Document the changes to api.
3d1a1ce Use the version of HfstBasicTransducer.transitions() returning a reference. Add HfstBasicTransition.set_weight to python api. Add tests for modifying transition weights.
17e0027 Fix a typo.
04accec Add help message to copy-for-windows.sh and comments to foma wrappers.
3688695 Typedef and templatize vector math, reformulate as per article description
95c022c Add help message.
1cd8cb8 Add help message.
dd2f230 Fix vector reading for case where lines don't end with a separator
5bcb93d Support hfst.compile_xfst_file only for python3.
4995953 Fix a typo.
20f39cd Fix copying files on mac as well as pip packaging issues.
7f60005 Update file instead of deleting it...
49b0ef9 Tentatively also support python version 2.
52016b2 Actually copy all files.
01744fa Omit sdk directory on windows if compiling for python3.5 or higher. Do not redefine _MSC_VER. Do not call subprocess on mac until it works correctly.
b44f91d Update README.
f677972 Check if we are compiling with visual studio 14.
b17eb3d Update PyPI documentation.
a6ca66b Add missing file.
410d6ef Get rid of unnecessary libraries.
f1572e1 Merge branch 'master' of https://github.com/hfst/hfst
75fcdd9 Fix chaining of functions in example.
d655229 Minor version bump to combat repo inconsistencies
91539f0 oops, reset variable
5566426 add transliterate output mode
416373d Update documentation.
b5736dc Add missing file.
43db07a Move ReplaceType under hfst.xerox_rules. Reorganize some internal functions.
bc8983e Move python code from swig interface file under module hfst.
3aff142 Check if readline package is available.
391f73a Do not keep input to interactive commands in readline history.
68e1b87 Update description.
0ff9821 Support apply up and apply down in hfst.start_xfst().
c687c09 Document return values of compile_xfst_file and compile_lexc_file.
12da4ae Warn user about missing readline support if --with-readline is not requested. Issue an error if --with-unicode-handler is called as such or with a value of 'yes' or 'no'. Make it possible to disable loading entries from shared object (openfst back-end feature).
9731e75 Update version number to 3.12.1. Handle links to future and earlier releases so that they are easier to update. Add a README.
764399e Remove doc/libhfst.py as documentation has been moved under directory doc/hfst/.
f3df0b9 Add more documentation for XreCompiler. Fix links to installation instructions.
d2666d6 pre-increment (thanks, cppcheck)
5712054 Add the script for creating python package. Convert README to dos format.
4abf93e Update package scripts.
4888fc6 Add a script for creating package which contains 64-bit python bindings for python3 for mac osx.
18df468 Add a script fpr creating 64-bit python bindings for python 3.3 and 3.4 for windows.
4790986 Add README files for python packages for mac and windows.
3519e4c Add a script for creating the lgpl package.
0cb9fb5 Add alternative files for making LGPL release of HFST.
fea6723 (tag: v3.12.1) Add missing files to dist. Do not print termination message when performing tokenize tests.
65962bf Ready for release 3.12.1.
77c845d Fix flag elimination bug (reported in issue #342).
c827bb6 Do not call fsm_destroy until issue with 'double free or corruption' has been fixed.
3b0e58a Provide a get_current_dir_name() for platforms without one
b0fb7a6 Merge branch 'master' of https://github.com/hfst/hfst
65a4d79 Search for included files under scriptdir, not working dir. If an included filename is non-absolute, prefix it with the directory the script was in when trying to read it. This only happens in non-Windows environments for now.
f5e94d7 Free allocated memory.
6144e75 tokenise -g: check for eof
cf46e69 Apply more changes from latest release of foma.
a9f8397 Apply some changes from newest release of foma.
cc8c96b Make a separate directory python/pypi for creating pypi packages.
479789e Rename the pypi package to 'hfst' and use a four-number versioning scheme. Also define '__version__'. Add an empty config file as hfst c++ headers include it.
eb304f0 tests: trying sed instead of awk for exit on <STREAMCMD:FLUSH>
798383d tokenize: handle Apertium-style superblanks in --giella-cg format
7922399 Provide a way to scale the vector similarity calculations via variable
166d418 Merge branch 'master' of https://github.com/hfst/hfst
9bc5315 parse word2vec -style text format and assume first line is info
5d3f6b8 Do not allow unescaped dots in regular expressions.
ce5e2a5 Merge branch 'master' of https://github.com/hfst/hfst
a8f21f0 Observe HAVE_SFST and HAVE_FOMA when including back-end header files.
8a9c0de nwords catenate-style argument for Like() function
1433e45 Merge branch 'master' of https://github.com/hfst/hfst
1ec8e03 Add experimental two-vector model for word sense
72d4f72 (tag: v3.12.0) Ready for release 3.12.0.
da86647 Read vector models and do simple ranking
42df833 Small fixes to memory leaks and efficiency.
e2abd89 Fix some memory leaks.
da54acc Fix leaks reported by valgrind.
92cf4dc Fix more memory leaks.
9bd9749 Fix some lost memory issues reported by valgrind.
6e129a4 Remove unnecessary variable. Delete transducer definitions and latest regexp when destructor is called.
d700309 flush given outstream, not necessarily cout
8d4f824 Add a destructor to XfstCompiler.
6b10aab Update README.
863e94a Add missing files to dist and remove generated ones.
e4c1304 Rename hfst.rules into hfst.sfst_rules. Fix hfst.types into hfst.ImplementationType in tests.
80f18a8 Use single quotes instead of double quotes in docstrings.
52a9d75 Add test for hfst.xerox_rules.
00b020c Rename xerox_replace_left and xerox_restriction back to replace_left and restriction in package hfst.xerox_rules.
d1556dd Fix documentation of parameter 'output' of function 'extract_paths'.
3df4a48 Sentence separation for finnpos-mode
d3058c7 Issue a warning if automake is too old, which can make build fail if pre-generated yacc files are modified.
9166cdc Move ImplementationTypes and ReplaceTypes to their own classes and get rid of package hfst.types.
6d6c631 Fix datatypes used in for loops.
c6aa5e1 Fix some typos in documentation.
8073675 Use renamed versions of xerox rule functions.
ae70056 Tentatively add HfstXeroxRules to python API in package hfst.xerox_rules.
dee0176 Fix some more warnings.
37edbbb Fix minor warnings generated by clang.
faa7647 Fix typos in ifdef guards.
41c5a32 Add a simple script for fetching statistics about hfst downloads.
d9d8077 Update pypi setup.
822f4bb doh, forgot to add test expected output
cccba22 hfst-tokenise giella-cg: fix indentation issue with backtracking
5a26269 Fix still some warnings.
32c82b3 giella-cg backtracking: ensure ignored spaces are still output
03d6a23 Do not warn about pragma push/pop issues or unsafe cstring functions on windows
9c12094 Use hfst::hfst_fopen instead of fopen. Fix some warnings related to datatype conversions.
d64cb49 Add function hfst::hfst_fopen that uses fopen_s when compiling with msvc.
5ed27ea Fix some warnings caused by conversions (possible loss of data).
b93a144 Do not use extern templates when compiling with msvc.
73f7e18 Remove files generated by the tests.
61ac4f8 Add a batch file version of test.sh for testing the python bindings on windows.
b53a128 Fix warnings.
7cb8740 Update pypi setup script.
cc9cca4 Remove unnecessary back-end include directories as well as commented code from Makefiles.
fd5f007 Fix some minor warnings.
ec114ea --giella-cg support for @PMATCH_BACKTRACK@ symbol (#337)
15fbb1f Update windows script.
097826c Remove extra comma from initialization.
d6b001d Add missing header file.
9b3c122 Fix warnings generated when compiling with -Wall.
a0dee64 Fix most of the warnings generated when compiling with -Wall.
89b2061 rename --gtd to --giella-cg, and more descriptive help text
335df14 Add support for compiling on windows.
85585c4 Fix a typo.
e628f55 Add a manifest file for pypi packaging.
48c34ef Tentatively add a setup file for creating pypi packages.
6abde49 update to proc to make it possible to weight surface forms
5e379fc Update windows compilation script.
1f55a51 Define int64 for msvc.
21f52d0 Implement snprintf for msvc compilation.
c05469e Replace definition _Bool with Boolean, as _Bool conflicts with typedefs in msvc header files.
00bbb1d Get rid of unnecessary typenames and typos noticed during compiling HFST on windows.
b160c07 Forward declare back-end implementation datatypes and wrap their destructors.
067c3c9 Add missing <iterator> header.
2f0c8df Add missing std:: to vector.
34e99ad Issue an error message if glib unicode handler is requested but not found.
d28075c Declare all functions that are needed in start_xfst().
ec34dea Add math.h
4a3be6b Make preparations for supporting OpenFst version 1.5.4.
c9940f6 Include HfstBasicTransducer.h instead of HfstTransitionGraph.h.
8560638 Add extern templates.
dc96953 Remove unnecessary OpenFst headers and use forward declarations instead.
699c098 Wrap StdVectorFst and LogFst destructors as both classes are forward declared.
7f81f41 Forward declare as much as possible when using OpenFst library. Also remove unnecessary include directives and add missing std namespaces.
a0b2fb3 Prepend missing 'std::'.
c8137fa Make HfstBasicTransducer and HfstBasicTransition fully implemented classes instead of generated template specializations.
eb5b877 Improve the way error messages are handled.
b8278ec Change 'import libhfst' to 'import hfst'.
2b9e44b Move implementations of HfstTropicalTransducerTransitionData away from header file.
ce05c67 Merge branch 'master' of https://github.com/hfst/hfst
3ec0d09 Alter expected result to conform with newly expected behaviour
759605c Do not allow newline characters inside quoted literals.
f296b6b Merge branch 'master' of https://github.com/hfst/hfst
63d7544 Allocate enough memory for XRE error message.
f968ebf Trying out facility for communicating sentence boundaries to plain tokenization mode
6a78c4c Merge branch 'master' of https://github.com/hfst/hfst
bf2ded3 Don't print size info when unverbose
5a33841 Don't separate tokens with empty lines in plain tokenization mode
d0a1b55 Warn about undefined multichar symbols when compiling lexc in verbose mode.
5d2c67d Fix a typo in option handling.
e2fee81 Tentatively add an option --do-not-minimize to hfst-regexp2fst which performs determinization instead of minimization. Also make hfst-xfst use determinization if variable 'minimal' is set to 'OFF'. Implement this by adding function hfst::set_minimization(bool) to HFST API and a function HfstTransducer::optimize() which performs minimization, or determinization if hfst::set_minimization(false) has been called. Replace calls to minimize() with optimize() in parts of HFST code relevant to the tools mentioned.
82aa26e Get rid of CR-LF newlines.
44f2e21 Remove commented code.
94b7607 Print an error message if output type is defined several times.
966a895 Fix mismatched free/delete reported by valgrind.
6dc1d8f Do not use strdup with -std=c99.
407a517 Use delete_foma instead of delete/free when deallocating memory used by a foma transducer.
07508b5 Limit to C++11 for now
d15b4be Use free with malloc, not delete.
b693b3a Add a function 'void LexcCompiler::reset()' and use it in XfstCompiler when reading lexc input again.
bc5ac27 Remove revision info from output of hfst-info as it doesn't work correctly in git.
ace4801 Pass cascade argument by reference in lookup_cascading functions.
e2ea4cb Fix a typo in function 'strip_percents' which resulted in incorrectly escaped zeros.
92a7152 delete more lines
9f940a3 remove osx lines from yml
f7bd085 Unobservable change - nothing happened here
d1bf79b close #336: off-by-one segfaults in OlLetterTrie
2574e27 Avoid using projections in casing functions (which cause minimization issues with large weighted transducers).
73a3ff0 Parametrize optionality in casing functions
0adc1f9 Make sure there's no case fallthrough
58eb35d Fix other casing functions too
c366c4c Another fix to Cap()
704c1c7 Merge branch 'master' of https://github.com/hfst/hfst
5a5df2d Change operation of Cap() and make timing per-object
44f0ae7 Fix missing empty lines in output when cascading composition is used.
284f9d0 Add an option --max-number=N for OL transducers which limits the number of results. If not defined, defaults to 5 for infinitely ambiguous input and to infinity for finitely ambiguous input.
5686864 There are some problems in using the bindings with python version 2, so refer only to python3 in README.
5a17fb0 Add a missing 'break' to switch block, making variable infinite_cutoff work correctly.
4c2f26f If no results are found, print it only once.
d703776 Tentatively add support for --xfst=print-pairs for OL transducers.
b7fcab9 Remember to flush the streams also when no results are found.
7ba32cf Merge branch 'master' of https://github.com/hfst/hfst
bc7a80a Implement a pair-lookup for optimized-lookup transducers.
ce3133f Add a fourth argument of type hfst::ImplementationType to the constructor and a function get_type() to ImplementationTypeNotAvailableException.
4ffe105 Print to console on windows if print-pairs is specified.
587f3c0 Do not print extra newlines in cascading composition.
4353894 Fix a bug/typo in string comparison.
72de782 Improve documentation in HfstTransducer.lookup.
0c888e9 Rename HfstBasicTransducer.lookup_fd to HfstBasicTransducer.lookup and add an argument 'obey_flags'.
d5493ac Rename HfstTransitionGraph's lookup_fd to lookup and change some parameter names to be more descriptive.
56a8d2f Add instructions for linking hfst c++ library statically in comments.
67e25ff Add missing file to windows scripts.
8db14ce (tag: v3.11.0) Ready for release 3.11.0.
fe37aa0 Update morphology checks.
8d3f162 Update EXTRA_DIST.
6dac246 Do not use binary transducers in tests.
ee62c44 Add missing doc file.
a367645 Mention that python tests are no longer available.
da2678d Skip Python tests as they can now be performed in the folder python/test.
ea47f0b One more stray memory leaking cache
95e3480 Memory conservation: only cache named objects and always minimize them.
6b113b2 Merge branch 'master' of https://github.com/hfst/hfst
974d240 Turn off automatic delimiter-contexts ("set need-separators off") in the presence of user-defined contexts
01fccfd Use docstrings.i instead of swig's autodoc. Document the contents of hfst and hfst.exceptions modules.
4322ca7 Add a docstrings file created semiautomatically from doxygen documentation.
816ce6c Add at least a simple docstring for all functions.
82c8f6a Merge branch 'master' of https://github.com/hfst/hfst
1b11fbf Obey -i option for giving the pmatch archive filename
4620b1b std=c++ works on OS X but not Linux, so disable Linux+Clang
d245a74 Test if gnu++ appeases Travis' clang
3d9c66b Include needed m4
5e73db3 Copy hfst-ospell's C++11+ detection
9af66d0 short int -> short; long int -> long
daa457e Yet more documentation..
e1b01ff More docstrings.
711a089 Add docstrings.
acf5696 tokenise: don't skip output-empties
b5b4434 tokenise: don't let empty analyses affect weight classes
18f5a2b travis: fix indentation
62d216a getopt_long() returns an int, not char; this is important because char is unsigned on many platforms which means char(-1) is 255
39d28d1 Do not check if lookup is infinitely ambiguous if --time-cutoff is defined.
c42af37 Remove redundant flag diacritic check. It was once removed, but got pushed there again.
b7b5882 Add support for --cascade also for optimized-lookup transducers.
9169e38 Support combination --cascade=composition --xfst=print-pairs.
51a0b61 Merge branch 'master' of https://github.com/hfst/hfst
ef6ed81 Fix the way cycle limit argument is passed.
9ad0950 Merge branch 'master' of https://github.com/hfst/hfst
6d94c90 Add a preliminary omorfi-friendly conll-u mode
45a0bf2 Remove redundant filtering of paths containing flag diacritics. Flags are handled already in function HfstBasicTransducer::lookup_fd.
5c7135a Implement option --cascade.
18d5e66 Tentatively add option --cascade to hfst-lookup.
1dcad09 Print error message if --xfst=obey-flags is used with optimized-lookup transducers. They always obey flag diacritics.
670ee39 Merge branch 'master' of https://github.com/hfst/hfst
f903d9c When doing apply up or down in hfst-xfst's, check if 'obey-flags' is ON or OFF.
8e9c6e6 travis: --enable-silent-rules
a253b8d Add an argument obey_flags=false to is_lookup_infinitely_ambiguous functions.
14bdd9c Make is_lookup_infinitely_ambiguous aware of flags. Use typedef hfst::StringVector in HfstLookupFlagDiacritics to prevent conflicting definitions in hfst-twolc which defines a class StringVector.
349fb73 Add option obey_flags=false to lookup_fd.
fdf912a Add tentative script for converting from doxygen documentation to docstrings.
7618907 Modify function visibility.
bc01c0b Control visibility of functions in HfstTransducer.
b535aa6 Follow foma and xfst convention for operator ignore '/', i.e. expand any identities and unknowns before inserting the strings to be ignored.
c1b4341 Add an argument 'force' to HfstTransducer::harmonize which makes the function harmonize also FOMA_TYPE transducers.
724b4e9 Obey variable 'print-sigma' when print_net is called.
6bec507 Add brief descriptions for hfst packages.
ca892d1 Document the implementation types.
98764da Fix bugs in options --xfst=show-flags and --xfst=obey-flags when non-optimized lookup is performed.
93b9fe2 Close the outstream when done. Not doing this was causing a bug where some buffered data never got written to the output file.
7d97106 Obey variable 'print-pairs' when performing apply down in non-optimized format.
ac2fc3a Add parentheses in conditionals of function 'flag_build' to ensure correct interpretation. Also remove redundant code.
28867c1 Allow punctuation as context in the "need-separators on" mode
0cd05ee Add "need-separators no" to test script to reflect changed default in pmatch
2597799 exclude gcc from osx builds
7dfa8ec Track used symbol names
81128c2 Allow lowercase define()
52a4ca4 Merge branch 'master' of https://github.com/hfst/hfst
af71662 Add side arguments to casing functions
7c1b388 Revert back to the original extra_link_args in python bindings.
2d5c670 Rename examples.py to test_examples.py. Perform tests in a more logical order.
4c8a1e3 Update documentation about linking to the HFST C++ library.
9f8ca0f If --enable-no-tools is used, skip tests in test/tools/.
41dc9a2 Mention module hfst.types.
a8c3e56 Remember to apply modifications to weight pushing to tests and hfst package.
835f0ef Get rid of TO_FINAL_STATE and TO_INITIAL_STATE and replace function 'push_weights' with 'push_weights_to_start' and 'push_weights_to_end'.
d05b9db Wrap HfstTransducer implementation types in package 'hfst.types'.
5e1a245 Add missing files to dist.
58d4602 Update README. Handle linking to extension library better.
de888b6 Update README. Handle cases where an implementation type is not available. Add missing files to dist.
1a68126 Fix a typo in URL.
7a2296b Update link in README.
861baf6 tokeniser: --weight-classes=N opt, as in hfst-proc
151f1fa Add functions lookup_optimize, remove_optimization and copy to HfstTransducer.
4efc9af Make HfstTransducer() create the empty transducer.
0f237c1 Comment out hfst_ol::set_extract_tags_mode(bool). Improve error messages in att and prolog readers.
57426ca Remove unnecessary commented file.
b7139cd Fix HfstException documentation.
3b13bf5 Merge branch 'master' of https://github.com/hfst/hfst
7e6d27c Fix a typo.
9f4ceb7 Merge branch 'master' of https://github.com/hfst/hfst
829c9e2 Implement remaining variables in the library and support them in hfst-pmatch, with the exception of need-separators which has to be specified during rule compilation, at least for now.
457c5de Document AttReader and PrologReader.
8695c47 Add HfstException::what() to python api.
2cffffa Add an alias what() for operator() to HfstException.
5fdf61a Add class PrologReader and tests for it.
77eda55 Add a new class AttReader and tests for it.
18edac5 Add package hfst to setup. The package is essentially a wrapper around libhfst module which is still available as before.
e41f131 travis: re-encrypt channel to #hfst
6329bb8 Use all implementation formats in tests.
5a5018c Do not use default_fst_type as pmatch transducers are always in ol format.
b38f87b Add stream tests.
ccaea0e Represent empty transducers as empty lines so that read_att_transducer will work correctly.
fe66b18 Remove unnecessary files.
bf5511d Forgot to do git add before committing...
3710f91 Update test examples.py and add it to test.sh. Remove extra files.
7d5064d Add directory tree for package 'hfst' and its subpackages 'exceptions' and 'rules'. This package encapsulates module libhfst.
7818c1e Delete unnecessary file.
91b86a0 Rewrite part of tests. Document the interface file.
657013d Add new tests.
3b4c186 Merge pull request #323 from unhammer/tokenise--gtd-tags-are-multichars
67125ab rm unneeded import
d927d79 pmatch: store input/output_symbol_strings, _parts refer to them
c665f8a Add missing endverbatim tag.
884d1dc Add missing file.
bc87fc5 Get rid of HfstFile wrapper.
30cde02 Try to preserve the names of transducers when converting.
90b3328 Expand prolog tests. Fix a small bug in naming transducers.
8e61537 Forgot to update testfile.
63a0456 Add a new function for reading prolog input and improve the way att input is handled.
9ef38f7 Split part of code related to reading att input into a separate function. Change the way transducer name is set when reading prolog input.
6d12e0d erge branch 'master' of https://github.com/hfst/hfst
22f1c47 Modify documentation, update tests, add new functions.
d14bc2f link to travis; bug tracker link sf→github
e8c7fd6 README is now symlink to README.rst; for more readable github
2258354 travis: no irc notes on success
1d8a1f4 travis: encrypt IRC channel for hfst/hfst
38c9544 Add HfstTokenizer examples as tests.
c635cd4 More documentation to HfstTokenizer.
c1ca3b6 Add more examples for HfstTokenizer.
0dc0cf4 Reorganize doxygen documentation to represent the new module structure of hfst python api.
95d658c Remove hfst.py, a directory structure is needed instead for doxygen.
5025aab Add new documentation for hfst python package.
4749745 Add more tests.
8f8b345 Split python tests to several files.
45aeb38 Skip one more failing pmatch test.
83dee76 Add missing files to dist.
a44b99e erge branch 'master' of https://github.com/hfst/hfst
b5ca1ec Add more documentation and modify functions.
579ee71 Merge branch 'master' of https://github.com/hfst/hfst
8050575 A PmatchContainer constructor taking a std::vector<HfstTransducer>. This should leave the passed vector untouched.
88aa682 Add more documentation.
ab82047 Add a simple test for XreCompiler.
a25eb2b A new test.
caf1ca8 XreCompiler::setOutputToConsole is needed, uncomment it.
f6acf2a Split c++ code to multiple files from libhfst.i. Update URL in setup.py.
780374a Merge branch 'master' of https://github.com/hfst/hfst
4c9ca19 Settle on a way to handle variable names with dashes (which look like minus signs)
967cc78 actually run the tokenize tests
5df87b6 Add more failing tests.
775a499 Fix some typos.
2e77e2d Add missing files to dist.
11ebca0 Merge branch 'master' of https://github.com/hfst/hfst
067e637 Merging master
4c9d43a Variable setting and reading
881b81c Use -Wno-deprecated also for tests.
85cb277 Modify messages that get printed during make check.
0761730 Skip tests that do not exist. Make prints more informative.
573324e Skip failing pmatch tests.
164e621 Add missing tests for ConvertTransducerFormat. Compile them with -Wno-deprecated.
f945e17 travis: irc notifications on build failure
0229baa Merge branch 'master' of https://github.com/hfst/hfst
93ed0db Remove const declarations from openfst back-end that cause compiler warnings.
5df2a2b Run g++ with -j2 (not -j) on travis; o/w runs out of memory
54b8f45 Comment out a function that is not defined.
48c91b8 Specify -Wno-deprecated also for tests in folders hfst-tagger and hfst-twolc.
810c330 https://travis-ci.org test rules
708171e doh, define multichar-symbols (for 2eb6888)
2eb6888 test for #320 / bf26db9
3aecdbc Merge branch 'master' of github.com:hfst/hfst
bf26db9 only change case if sf and lm case differ (#320)
b3782de Remove a rule that cannot be matched since a rule has already been defined for "DownCase(" earlier.
5ad3487 Add the keyword const to arrays used to store xfst commands.
79c7dfe Make compiler happy by using strdup instead of plain const char *.
f6d9953 Handle return values of fread in foma and sfst back-ends to prevent warnings during compilation.
214b7b6 Fix a typo.
862e858 Replace foma binary files that are used for testing results from lexc with equivalent files in prolog format. Then the tests will not depend on the foma back-end.
680389b tokenize tests
2b7cf7e tokenize: new --gtd Giellatekno/Divvun mode
adb37d4 tokenise: fix #318, header parsing for TOP
76e9069 Fix a bug in flag elimination involving multiple flags. Fixes bug #315.
f653780 Evaluate transducers (as opposed to argumentless functions) where declared
e6c6e82 (tag: v3.10.0) Ready for release 3.10.0.
97c1285 Swap implementations of commands 'apply up' and 'apply down' in hfst-xfst, making them work as they do in foma and xfst.
85ef34c Forgot to add before latest commit...
13280ea Edit warning about converting native foma transducers.
b31a6c1 Fix typo: swap left and right.
808d984 Add hfst-flookup to windows scripts.
66db285 Update bug tracker url and copyright year.
b127ab3 Git ignore hfst-flookup.
1b40de6 Add tests for hfst-flookup.
12e4cca Trandsducer operations no longer return a reference to the transducer, fix this in tests and interface operations.
a128c7f Add a tool hfst-flookup which does lookup from right to left, i.e. in the same way as foma's flookup and xfst's lookup. hfst-lookup and hfst-optimized-lookup stay as they are, but mention in their help message that they do lookup in the opposite direction.
cea21b4 Add a comment about the way gzipped input is handled.
da7ebcc Make HfstInputStream constructor throw a FileIsInGZFormatException if input is in gzipped format. In hfst-fst2fst, issue an error message that asks user to unzip the transducer.
5c2755b Merge commit frankier:add-gitignore manually and also add more files to be ignored by git.
99b470d Merge pull request #313 from frankier/escape-dot-labels
f6905a1 Improve handling of argumentless functions and funcalls
d804d3d Remove some memory leaks in compilation
ef740e0 Fix nested funcall issues
61e1e0a Add optional versions of cap functions and AnyCase().
68e078b Corrections and additions to casing functions, and add Implode() and Explode()
2484633 Add a facility for detecting when hfst-tokenize is given a "bare" dictionary and generating a somewhat intelligent tokenizer when that happens. Due to awkward architectural reasons this involves changes and additions in many parts of the library, but they are not major.
089f3ca Add .gitignore
d1d81db Merge branch 'master' of https://github.com/hfst/hfst
485f575 Fix bug that caused exclusionary lists to be missed much of the time.
03ff992 Escape " in dot labels.
9984096 Factor out replace_all into string-utils.cc.
ca10d77 Revert swapping of hfst-xfst's 'apply up' and 'apply down' commands.
7352cf7 Swap implementations of 'apply up' and 'apply down' in hfst-xfst, so that these commands work in the same way as in foma and xfst.
db2d4dc Add an example for HfstTransducer constructor.
f37b8d1 Tentatively implement lookup also for transducers that are not in optimized-lookup format using composition and path extraction.
1cb2992 Add an HfstTransducer constructor that takes a StringVector as argument.
f32c049 Support python 2 unicode strings in lookup.
8d2c6b8 Fix a typo
f442b91 Make HfstInputStream iterable by implementing functions __iter__, __next__ (for Python 3) and next (for Python 2).
2c660ff Add functions in namespace hfst::rules to python API.
947bb62 Add some information about Python interface in README and advise the users to see file python/README.
067b0c2 Remove slashes from headers used when processing the file.
c4b5ce8 Update documentation.
0e4e8a7 Redefine functions which return a reference to themselves as void to avoid problems on the python side.
5f97eac Tentatively add a C++ function 'std::vector<hfst::HfstTransducer> compile_pmatch_expression(const std::string & pmatch)', mostly copied from hfst-pmatch2fst.cc.
21da500 Add a const declaration to string& argument of PmatchContainer::process and PmatchContainer::match.
cd7f2f8 Tentatively add some pmatch functions. Add a missing typedef for HfstTransducerVector.
a9fbd44 Add a function to HfstInputStream for finding out whether hfst headers are included in transducers. When reading native foma transducers in hfst-fst2fst, warn that inversion may be needed for lookup to work as expected.
fa4565e (tag: v3.9.2) Release 3.9.2.
3f7a57e Update hfst binary file used for tests.
5cd8a71 Add option --pythonpath to python tests. Document the test tools better.
927da9c Make fst2txt functionality test silent.
ad8c82c Update installation check scripts.
ddf749d Remove test.py from EXTRA_DIST, the tests have been rewritten.
dd7d0be Merge pull request #309 from unhammer/pmatch-input-marks
97b6e3f Merge pull request #308 from unhammer/tokenise-minor-cg-fix
3781806 Make small modifications to documentation and typedefs used in function declarations/definitions, so that doxygen can generate documentation right.
00d89ce Update Python and C++ API documentation files.
4319b94 Move api documentation file under folder 'doc' too.
67e1e8c Move C++ documentation file under folder 'doc'.
8e2e651 Do not catch exceptions derived from HfstException inside the constructor of HfstInputStream and throw them as HfstException. Instead, throw them as such.
56ffca8 pmatch: special symbol @PMATCH_INPUT_MARK@ for input marks
90aa695 pmatch: on seeing a special symbol, mark the offset in the input string
bc345c6 CG forms always go in "<>", not before the tab
8dceb24 Merge pull request #307 from unhammer/tokenise-short-options-missing
2c207de short option -m missing from getopt_long
a9314ad Add m4 for older distros
6c944ba Bump patch version since revision has decreased
859f8da Add builting function Interpolate()
c6f9908 Fixed condition for including symbols in exclusionary lists Rather than the printability vector, which may not be completely set yet, check the symbol itself for printability.
780b9af Fix bug involving stale caches of function call results
26d3d8f Change python3 into python in README as version 2 works now too.
7af6f3f Move tools/src/hfst-string-conversions.[cc|h] under libhfst/src/.
a1b5bb2 Update version number to 3.9.1.
d23eb93 Do something useful in the case of a missing root rule
9e4e663 Make Makefiles in openfst back-end licensed as LGPL.
03e1949 Candidate for release 3.9.1.
27f80ee Fix names of files whose names were changed.
8e4f324 Set server based search to NO.
5b2c698 Add word boundaries in xerox mode.
6e78e1f Fix copypasted typo
b6970ab Fix ToLower and ToUpper, which had transposed names and flawed implementations
c939413 Use iterator instead of range-based loop on windows.
e5897fb Fix a couple of typos inside 'ifdef _MSC_VER' detected when compiling on windows.
7694ee0 License for the library code is now lgpl 3.
b8a5b98 LGPL license
a031526 Compatibility with python2.
1baee13 Add exclusionary lists (eg. Exc(Whitespace) matches symbols not in Whitespace)
a92e1d0 -X option fixed in hfst-pair-test. Additionally, the user gets an error message if unescaped :-symbols were found.
8d0a028 Fix a bug in HfstTransitionGraph::find_replacements. Implement compile-replace without composition in XfstCompiler to avoid alignment issues in result.
887381e Fix epsilon handling in compile-replace expressions.
2891a27 Allow epsilons inside compile-replace expressions as xfst does.
bcfea4a Warn about '@_EPSILON_SYMBOL_@' not being an ordinary symbol in hfst only in verbose mode.
5fc456b Allow space after operator '^' in regular expressions.
0758730 Fix binding precedence of :
9f4628e Add throw to appease rpmlint no-return-in-nonvoid-function
78efc5d Fix a bug in function 'fst'.
23a5285 Print a blank line even if there were no results or if tokenization failed
73d317b One && too many
5f4a8fc Rename hfst-proc2 to hfst-tokenize, with hfst-proc2 and hfst-tokenise as aliases
732e37e Add tests for python bindings.
51cbc5e Complete rewrite of pmatch compilation This is a cleanup and rewrite of pmatch compilation, hopefully allowing easier extensibility and maintainability in the future. While this has been reasonably well tested, it is likely that there are some regressions; please report any. Various additional features are pending.
31de91e Move scripts under subdirectories and remove scripts that are not needed.
6156215 Move windows scripts under scripts/windows. Get rid of libhfst_win.i.
0bf33d7 Remove HfstFastTransition.h also from dist.
02fa326 Remove HfstException.cc and HfstException.h files, HfstExceptionDefs.cc and HfstExceptionDefs.h are used instead.
9356517 Remove class HfstFastTransitionData and the template HfstFastTransducer that uses it. HfstBasicTransducer is the template of HfstTransitionGraph that is used everywhere and it is fast enough.
dd6873e Remove tentative reentrant version of xre compiler. The normal xre compiler can handle expressions that the reentrant one was originally intended for.
05c78d8 Update python API documentation.
f933525 Remove old commented code. Add a warning about python bindings not being under autotools; advise users to see python/README.
b127d48 Set -std=c++0x unless a higher standard is required with the -std option.
f2738be Do not generate hfst-scanner header file. Make hfst-scanner depend on hfst-compiler header file to make sure they are generated in the right order.
a44eb61 Check automake version in configure to determine whether header files generated by yacc/bison should use extension .h (automake < 1.12) or .hh (automake >= 1.12).
7f206c0 call_counter is a member of the container, not the transducer
2aeda9f Reweight arcs iff all conditions given with --input-symbol, --output-symbol and --symbol are met.
fddc9ea Forgot to increment call counter
a56089b Only call clock() every millionth time
1c4c871 Add missing declarations
a710ddd Only check clock() every million calls to get_analyses() to save time
8cffb74 Move start_clock setting to where the line is read to make the time limit per-analysis
6488877 Add short option for time cutoff (-t)
b6fdeb6 In pmatch, make the time cutoff "soft" to ensure we at least return something (this was an issue for rulesets with RTNs only - all the time could be spent in one leaving nothing for anything else, causing needless failure)
70e025f Ignore previous commit, I had some old library code confusing me
8dfcd66 Add time cutoff options
8a9f0c3 Fix what seems like a typo in hfst-lookup, &infinite_cutoff for infinite_cutoff It seems like the size_t and size_t* can substitute for each other, but the prototype is for size_t.
2134ef4 Fix time cutoff check in ol library code
8ff4aa3 Omit infinitely ambiguous check when we have a time cutoff
969d479 Fix time cutoff handling
dd30c33 Switch from !€ / !$ to !!€ / !!$ as the test case prefixes, to bring it in line with the conventions in the rest of the Giella infrastructure (where this testing setup is used).
fd175f1 Fix option --to-transducer of hfst-substitute.