Code

Taggers for URL filtering#112

Code

Merged

soldni merged 27 commits intomainallenai/dolma:mainfrom soldni/trainallenai/dolma:soldni/trainCopy head branch name to clipboardFeb 10, 2024

+1,991-25

Commits

Commits on Feb 5, 2024

• wip

  soldni

  committed
  170321f

  Copy full SHA for 170321f

Commits on Feb 6, 2024

• .

  soldni

  committed
  8f971eb

  Copy full SHA for 8f971eb

• tests

  soldni

  committed
  f766302

  Copy full SHA for f766302

• url testing

  soldni

  committed
  71b3858

  Copy full SHA for 71b3858

• implementing taggers

  soldni

  committed
  5ab2bd5

  Copy full SHA for 5ab2bd5

• missing metadata dir

  soldni

  committed
  f45c958

  Copy full SHA for f45c958

• taggers loading

  soldni

  committed
  038e811

  Copy full SHA for 038e811

• fixed tests

  soldni

  committed
  a38ffb1

  Copy full SHA for a38ffb1

• parsed out

  soldni

  committed
  91544e5

  Copy full SHA for 91544e5

• download domain

  soldni

  committed
  9430d35

  Copy full SHA for 9430d35

Commits on Feb 7, 2024

• Merge branch 'main' into soldni/train

  soldni

  authored
  609c995

  Copy full SHA for 609c995

• Merge branch 'main' into soldni/train

  soldni

  committed
  e524110

  Copy full SHA for e524110

Commits on Feb 8, 2024

• url

  soldni

  committed
  10c4c35

  Copy full SHA for 10c4c35

• source_url

  soldni

  committed
  f773e6b

  Copy full SHA for f773e6b

• added util to replace cached_path

  soldni

  committed
  dfa10cd

  Copy full SHA for dfa10cd

• added tests

  soldni

  committed
  2962f01

  Copy full SHA for 2962f01

• minor logger changes

  soldni

  committed
  1d13933

  Copy full SHA for 1d13933

• logging features

  soldni

  committed
  7e556bb

  Copy full SHA for 7e556bb

• tagger

  soldni

  committed
  589de50

  Copy full SHA for 589de50

• tagger

  soldni

  committed
  cd0eace

  Copy full SHA for cd0eace

• tagger

  soldni

  committed
  c0c2f12

  Copy full SHA for c0c2f12

• fixing tests older version

  soldni

  committed
  69cbb4a

  Copy full SHA for 69cbb4a

Commits on Feb 9, 2024

• parallel

  soldni

  committed
  5c939be

  Copy full SHA for 5c939be

• new lists

  soldni

  committed
  2abbbb5

  Copy full SHA for 2abbbb5

• new urls

  soldni

  committed
  8a7f998

  Copy full SHA for 8a7f998

• style

  soldni

  committed
  cee0c87

  Copy full SHA for cee0c87

Commits on Feb 10, 2024

• final filters

  soldni

  committed
  6ee9b31

  Copy full SHA for 6ee9b31