the output of ./build.py is difficult to handle and contains many false positives
./build.py
the data cleanups and manipulation are now automated using the python pandas library the CLDR data are used to add english names
$ ./build_stats.py * CSV loaded → we now have 44298 rows * * duplicated headers are removed → we now have 43899 rows * * duplicated po files are removed → we now have 43588 rows * Deduplication is done → we now have 43588 rows * * remove pot files → we now have 43381 rows * * remove gmo files → we now have 39208 rows * * remove files with 'totalMessage'=0 → we now have 39148 rows * Removal is done → we now have 39148 rows * * remove if lang endswith encoding values → we now have 39132 rows * * remove if lang contains a point → we now have 39007 rows * bcp47 data are guessed → we now have 39007 rows * * remove if len(region)>2 → we now have 38881 rows * * remove if len(language)>3 → we now have 26288 rows * * remove if language.isdigit() → we now have 23984 rows * bcp47 data are cleaned → we now have 23984 rows * cldr data are added → we now have 23984 rows * * remove languages non existing in CLDR → we now have 23087 rows * cldr data are cleaned → we now have 23087 rows
2 new commits added
add build_stats in readme
add language, region and territory name from cldr
add f30 results
use results folder
@petersen @suanand : can you please look at the build_stats.py and give your feedback?
for example, I remove files ending with ".big5", ".cp936", etc. Is it indeed encoding only?
finale result file is: results/f30/3.result.csv
1 new commit added
add CLDR ideas
add territory-language consistency check using population data
improve documentation
rename *region* in *territory*
10 new commits added
add CLDR data
use pandas to clean data
Pull-Request has been merged by jibecfed
problem
the output of
./build.pyis difficult to handle and contains many false positivessolution
the data cleanups and manipulation are now automated using the python pandas library
the CLDR data are used to add english names
comment