PR#23: check if l10n folder exists - fedora-docs/translations-scripts

fedora-docs / translations-scripts

#23 check if l10n folder exists

Merged 2 years ago by darknao. Opened 2 years ago by jibecfed.

21-prevent-stats-erasing into master

don't pass find result as a single string

Francois Andrieu • 2 years ago

0260240

#21 check if l10n folder exists

Jean-Baptiste Holcroft • 2 years ago

6347c97

stats/compute.sh

file modified

+17 -8

		`@@ -2,20 +2,29 @@`
		`echo "**********************************"`
		`echo "* Refresh fedora-docs/localization"`

		`- # make sure we are in the folder of the current cript`
		`- pushd $(dirname $0) > /dev/null \|\| exit`
		`+ # we expect a l10n folder containing all po/pot files in the parent folder`
		`+ RELATIVE_PATH="../"`
		`+ FOLDER_TO_SCAN="l10n"`

		`- echo "* Runnning po Count"`
		`- pushd "../" > /dev/null \|\| exit`
		`+ # make sure we are in the folder of the current script`
		`+ pushd "$(dirname "$0")" > /dev/null \|\| exit`
		`+`
		`+ if [[ ! -d "$RELATIVE_PATH$FOLDER_TO_SCAN" ]]; then`
		`+ echo "The folder $FOLDER_TO_SCAN do not exist, exiting..."`
		`+ exit 1`
		`+ fi`
		`+`
		`+ echo "* Running po Count"`
		`+ pushd "$RELATIVE_PATH" > /dev/null \|\| exit`
		`# pocount requires the installation of translate-toolkit`
		`- pocount --csv $(find l10n -name '*.po') >> stats/results.csv 2>> stats/error.log`
		`- pocount --csv $(find l10n -name '*.pot') \| tail -n+2 >> stats/results.csv 2>> stats/error.log`
		`+ pocount --csv $(find "$FOLDER_TO_SCAN" -name '*.po') >> stats/results.csv 2>> stats/error.log`
		`+ pocount --csv $(find "$FOLDER_TO_SCAN" -name '*.pot') \| tail -n+2 >> stats/results.csv 2>> stats/error.log`
		`popd > /dev/null \|\| exit`

		`echo "* Cloning"`
		`git clone https://pagure.io/fedora-docs/localization.git --quiet`

		`- echo "* Installing Python depedencies with pip"`
		`+ echo "* Installing Python dependencies with pip"`
		`python3 -m venv venv`
		`source venv/bin/activate`
		`pip install pandas`
		`@@ -29,7 +38,7 @@`
		`git remote set-url --push origin ssh://git@pagure.io/fedora-docs/localization.git`
		`git add .`
		`git status --porcelain`
		`- git commit -m "Automated update" --quiet`
		`+ git commit -m "𝄠 Automated update 𝄟" --quiet`
		`git push --quiet`
		`popd > /dev/null \|\| exit`

jibecfed commented 2 years ago

the stats scripts is run after the global l10n pot update, but if it completely fails, there is no l10n folder to take translations from

the motivation to link both script is to reuses the "l10n" directory containing all po files, saving time & bandwith, and making sure stats match the reality

Metadata Update from @jibecfed:
- Request assigned

2 years ago

darknao commented 2 years ago

lgtm
I'm pushing this on staging first ( 6884803 ) to see how it goes.

darknao commented 2 years ago

script failed on staging :

* Running po Count
./stats/compute.sh: line 21: /usr/bin/pocount: Argument list too long
* Cloning
* Installing Python dependencies with pip
Collecting pandas
  Downloading pandas-1.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
Collecting python-dateutil>=2.7.3
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2017.3
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting numpy>=1.17.3
  Downloading numpy-1.21.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.8 MB)
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, pytz, python-dateutil, numpy, pandas
Successfully installed numpy-1.21.2 pandas-1.3.2 python-dateutil-2.8.2 pytz-2021.1 six-1.16.0
WARNING: You are using pip version 21.0.1; however, version 21.2.4 is available.
You should consider upgrading via the '/workspace/stats/venv/bin/python3 -m pip install --upgrade pip' command.
* Generating stats
Traceback (most recent call last):
  File "/workspace/stats/./build.py", line 270, in <module>
    main()
  File "/workspace/stats/./build.py", line 11, in main
    parse(file)
  File "/workspace/stats/./build.py", line 30, in parse
    data = pandas.read_csv(file)
  File "/workspace/stats/venv/lib64/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/workspace/stats/venv/lib64/python3.9/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/workspace/stats/venv/lib64/python3.9/site-packages/pandas/io/parsers/readers.py", line 482, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/workspace/stats/venv/lib64/python3.9/site-packages/pandas/io/parsers/readers.py", line 811, in __init__
    self._engine = self._make_engine(self.engine)
  File "/workspace/stats/venv/lib64/python3.9/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
    return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
  File "/workspace/stats/venv/lib64/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 69, in __init__
    self._reader = parsers.TextReader(self.handles.handle, **kwds)
  File "pandas/_libs/parsers.pyx", line 549, in pandas._libs.parsers.TextReader.__cinit__
pandas.errors.EmptyDataError: No columns to parse from file
* Commit changes