Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Options to skip small files and not recurse on input paths #90

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open
Prev Previous commit
Rename TEMP_FILE_EXTENSION to TEMP_FILE_PATH and use toString for che…
…cking Path names.
gsteelman committed Dec 17, 2015
commit c1968fa90b84700177bf16b76d8527f46ad5efb7
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ public class DistributedLzoIndexer extends Configured implements Tool {
public static final long LZO_INDEXING_SMALL_FILE_SIZE_DEFAULT = 0;
public static final String LZO_INDEXING_RECURSIVE_KEY = "lzo.indexing.recursive.enabled";
public static final boolean LZO_INDEXING_RECURSIVE_DEFAULT = true;
private static final String TEMP_FILE_EXTENSION = "/_temporary";
private static final String TEMP_FILE_NAME = "/_temporary";

private boolean lzoSkipIndexingSmallFiles = LZO_INDEXING_SKIP_SMALL_FILES_DEFAULT;
private boolean lzoRecursiveIndexing = LZO_INDEXING_RECURSIVE_DEFAULT;
@@ -49,12 +49,12 @@ public class DistributedLzoIndexer extends Configured implements Tool {
private Configuration conf = getConf();

/**
* Accepts paths which don't end in TEMP_FILE_EXTENSION
* Accepts paths which don't end in TEMP_FILE_NAME
*/
private final PathFilter nonTemporaryFilter = new PathFilter() {
@Override
public boolean accept(Path path) {
return !path.getName().endsWith(TEMP_FILE_EXTENSION);
return !path.toString().endsWith(TEMP_FILE_NAME);
}
};