Code-prose-composition tagger.

[no0p]

Tagger for Code Prose Composition

Add a tagger that adds attributes for code-prose-other composition of files based on line classifications.

Produces tags for

• code/prose/other composition as a percent of the document
• code/prose/other mean entropy
• code/prose/other line counts
• coder-prose boundary count

Recommended filter for mixed prose/code content based on these tags is:

exp__code_prose_composition__code > 0.05
exp__code_prose_composition__prose > 0.3
exp__code_prose_composition__code_count >= 8
exp__code_prose_composition__code_mean_entropy < 0.5

The code entropy adjusts for bias towards code for short string including "code-y" characters like (, ), [, ], : etc due to a lack of nice negative examples. Until time for an improved classifier is available, including a filter for high confidence code predictions via mean entropy works sufficiently well.

One More Thing

Updated a pre-suite hook to set the multiprocessing start method to spawn to prevent a side effect where test case dependencies may set it to the default fork, violating runtime assertions.

[Whattabatt]

Generated attribute line example:

{"id":"http://apple.stackexchange.com/questions/66593/how-to-enable-repair-disk-button?answertab=votes","attributes":{"code_prose_composition__code_prose_composition__code":[[0,2748,0.17]],"code_prose_composition__code_prose_composition__prose":[[0,2748,0.7]],"code_prose_composition__code_prose_composition__other":[[0,2748,0.13]],"code_prose_composition__code_prose_composition__code_count":[[0,2748,5.0]],"code_prose_composition__code_prose_composition__code_mean_entropy":[[0,2748,0.84149]],"code_prose_composition__code_prose_composition__prose_count":[[0,2748,21.0]],"code_prose_composition__code_prose_composition__prose_mean_entropy":[[0,2748,0.15337]],"code_prose_composition__code_prose_composition__other_count":[[0,2748,4.0]],"code_prose_composition__code_prose_composition__other_mean_entropy":[[0,2748,0.04893]]},"source":"dclm"}

I think you might be able to keep this shorter by cutting the key in line_label, maybe elsewhere

[Whattabatt]

LGTM once you see if you can make the attribute keys shorter

[Whattabatt]

Other taggers use underscores instead of dashes

[no0p]

It looks like the format produced is as follows.

"{experiment_name}__{tagger_name}__{prediction_label}"

In the posted example, the experiment name matching the tagger name makes it look worse. Experiment name is somewhat out of our control. Nonetheless, there is room for improvement, so I made the following changes in the interest of brevity.

code_prose_composition -> code_composition

code_count -> code
prose_count -> prose
other_count -> other

code_composition -> code_pct
prose_composition -> prose_pct
other_composition -> other_pct

code_mean_entropy -> code_entropy
prose_mean_entropy -> prose_entropy
other_mean_entropy -> other_entropy

code_prose_boundaries -> boundaries

This results in records like the following:

{"id":"1","attributes":{
"my_experiment__code_composition__boundaries":[[0,616,0.0]],
"my_experiment__code_composition__other_pct":[[0,616,0.33]],
"my_experiment__code_composition__other":[[0,616,1.0]],
"my_experiment__code_composition__other_entropy": [[0,616,1.22949]],
"my_experiment__code_composition__prose_pct": [[0,616,0.67]],
"my_experiment__code_composition__prose": [[0,616,2.0]],
"my_experiment__code_composition__prose_entropy":[[0,616,0.002]]},
"source":"fake"}

I think it's a little more manageable and decent trade-off between explicitness and gigantic label keys. If you see more room for improvement, please let me know.

[Whattabatt]

I think it's a little more manageable and decent trade-off between explicitness and gigantic label keys. If you see more room for improvement, please let me know.

Yeah, good enough, dolma attributes are always a bit ugly for human reading