[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KenLM based content detection #72

Merged
merged 19 commits into from
Dec 13, 2022
Merged

KenLM based content detection #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
86d58c5
WIP: LRU cache for multiple KenLMs
Uinelj Nov 25, 2022
123ae34
make annotator immutable
Uinelj Nov 28, 2022
a84bc53
simplify models struct, add LRU later on
Uinelj Nov 29, 2022
da787c6
thread safe model hashmap
Uinelj Nov 30, 2022
2641f71
doc: add comments, link to ctclib fork
Uinelj Dec 1, 2022
670cc1d
ci: add cpp dependencies for ctclib/KenLM
Uinelj Dec 1, 2022
82e10b1
ci: add sudo on ctclib cpp deps install
Uinelj Dec 1, 2022
6bb7bb0
ci: change blocklist fetching from ftp to git
Uinelj Dec 1, 2022
5326c33
fix: add simple errors on no model found
Uinelj Dec 1, 2022
eeb1520
test: test loading of correct kenlm model
Uinelj Dec 2, 2022
e2f7a04
begin integration of kenlm in main pipeline
Uinelj Dec 2, 2022
d68f740
feat(kenlm): remove error prone defaults, remove commented code, adap…
Uinelj Dec 5, 2022
68efc77
ci: add full stacktrace
Uinelj Dec 6, 2022
385059d
add fallback on arpa models if binary not found
Uinelj Dec 6, 2022
b0be72a
ci: fix ci script and code to run properly the rebuilding integration…
Uinelj Dec 8, 2022
f1c52ba
feat: make kenlm an optional dep, add readme instructions
Uinelj Dec 8, 2022
17cceda
make kenlm an optional feature
Uinelj Dec 12, 2022
a4364c0
feat(kenlm): builders list now built at pipeline start.
Uinelj Dec 12, 2022
91a0d00
fix(logs): change log level of adult doc id from info to debug
Uinelj Dec 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 24 additions & 15 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:
branches: [master, dev]
env:
CARGO_TERM_COLOR: always

jobs:
cache_test_data:
runs-on: ubuntu-latest
Expand All @@ -34,32 +34,41 @@ jobs:
- name: Fetch identification bins
if: steps.cache-lid.outputs.cache-hit != 'true'
run: |
wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin
wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.189.bin
wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin
wget https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.189.bin
- name: Fetch blocklist
if: steps.cache-blocklist.outputs.cache-hit != 'true'
run: |
mkdir -p res/blocklist
wget ftp://ftp.ut-capitole.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz
tar xvzf blacklists.tar.gz
mv blacklists/* res/blocklist
rm blacklists.tar.gz
rmdir blacklists
mkdir -p res/blocklist
wget https://github.com/olbat/ut1-blacklists/archive/refs/heads/master.zip
unzip master.zip
mv ut1-blacklists-master/blacklists/* res/blocklist
gzip -d res/blocklist/adult/domains.gz #adult blocklist is compressed
rm -r ut1-blacklists-master
- name: Fetch CC shards
if: steps.cache-shards.outputs.cache-hit != 'true'
run: |
mkdir -p res/shards
wget -O res/shards/0.txt.gz https://data.commoncrawl.org/crawl-data/CC-MAIN-2022-33/segments/1659882570651.49/wet/CC-MAIN-20220807150925-20220807180925-00000.warc.wet.gz
wget -O res/shards/1.txt.gz https://data.commoncrawl.org/crawl-data/CC-MAIN-2022-33/segments/1659882570651.49/wet/CC-MAIN-20220807150925-20220807180925-00001.warc.wet.gz
mkdir -p res/shards
wget -O res/shards/0.txt.gz https://data.commoncrawl.org/crawl-data/CC-MAIN-2022-33/segments/1659882570651.49/wet/CC-MAIN-20220807150925-20220807180925-00000.warc.wet.gz
wget -O res/shards/1.txt.gz https://data.commoncrawl.org/crawl-data/CC-MAIN-2022-33/segments/1659882570651.49/wet/CC-MAIN-20220807150925-20220807180925-00001.warc.wet.gz
- name: Install KenLM dependencies
run: sudo apt install -y libboost-all-dev libeigen3-dev
- name: Get sample KenLM model
run: |
mkdir -p res/kenlm
wget -O res/kenlm/en.arpa https://raw.githubusercontent.com/agatan/ctclib/main/data/overfit.arpa
- name: Build
run: cargo build --verbose
- name: Create test directories
run: mkdir res/corpus res/rebuilt
run: |
mkdir res/corpus
mkdir res/rebuilt
mkdir -p res/corpus/rebuild
ls res/
- name: Run tests
run: cargo test --verbose
run: RUST_BACKTRACE=1 cargo test --verbose
- name: Run cargo-tarpaulin
uses: actions-rs/tarpaulin@v0.1
continue-on-error: true
- name: Upload to codecov.io
uses: codecov/codecov-action@v1

1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ Cargo.lock
/dst*
/result*
lid*
res/
Loading