tiny-lang-detect, branch HEADGenerate tiny models for language detectione9340ad6f80c3c09853e5860ef694ba036de52ba2026-01-28T07:30:27Z2026-01-28T07:30:27Zadd long cli argumentsTobias Bengforttobias.bengfort@posteo.deadd long cli arguments
1698810e8a923192a8f15347644155f7a39eafe52025-11-08T08:14:25Z2025-11-08T08:14:25ZREADME: typosTobias Bengforttobias.bengfort@posteo.deREADME: typos
7ce818b79da3ba429b8c3f176271cd51997124e12025-05-26T18:53:15Z2025-05-26T18:53:41Zdemo: update modelTobias Bengforttobias.bengfort@posteo.dedemo: update model
1a380f7c5ba83ef5a151e6b0bf4efa431d08db8a2025-05-26T18:17:10Z2025-05-26T18:17:10ZtypoTobias Bengforttobias.bengfort@posteo.detypo
fe18df939733a8d8e6286631930220e1ca5c13152025-05-26T15:40:33Z2025-05-26T18:15:58Zprobability: assume frequencies to be independentTobias Bengforttobias.bengfort@posteo.deprobability: assume frequencies to be independent
For dependent probabilities that sum to 1, we get the total probability
by calculating:
n! * \prod(q_i^{k_i} / k_i!)
If we want to compare two of these probabilities, we can leave out
anything that does not depend on q. With $p_i = k_i/n$, this results in:
\prod(q_i^{p_i})
This is what we have done so far. However, our probabilities do not sum to
1. If we only look at 1-grams, we can add an "everything else" factor:
(1 - \sum q_i)^{1 - sum p_i}
While 1-grams, 2-grams and 3-grams are not totally independent, they are
also not dependent in the sense that one implies not-the-other. So we
could caluclate the probability of each group individually and multiply
the results.
In practice, just treating everything as independent seems to work just
as well and is much simpler.
9c128468b32a9d10a1ac0dea0908e38d036832df2025-05-26T15:37:12Z2025-05-26T15:37:24Zallow utf-8 in json outputTobias Bengforttobias.bengfort@posteo.deallow utf-8 in json output
6d8c914ce3795f6e5d2fc26800b3514000c8e4dd2025-05-19T11:36:43Z2025-05-19T11:44:47Zfurther simplifyTobias Bengforttobias.bengfort@posteo.defurther simplify
c70e23e44137b076ce3db100ac66b940358e4bd82025-05-19T09:17:30Z2025-05-19T11:08:03Zsimplify distanceTobias Bengforttobias.bengfort@posteo.desimplify distance
`prod (pi/qi)**pi` is how much more likely it is that observation p was
created by model p than by model q. Applying log() turns this into the
KL-divergence.
`prod qi**pi` is the probability that observation p was created by model
q.
`prod (pi/qi)**pi * prod qi**pi = prod pi**pi` is the same for each q.
8184ad526ec1080060f790a1fa99b0533fdb1b8a2025-05-14T05:10:30Z2025-05-14T05:43:59Zswitch from euclidean distance to Kullback–Leibler divergenceTobias Bengforttobias.bengfort@posteo.deswitch from euclidean distance to Kullback–Leibler divergence
This has better results and is also more in line with the Bayesian
approach used by langdetect.
2f92e8aef5503af2ab121000b74fe1c1b3bf160a2025-05-12T07:08:10Z2025-05-12T07:11:03ZREADME: add single-ngram classifierTobias Bengforttobias.bengfort@posteo.deREADME: add single-ngram classifier
27981b2f5adbba86e975add9b04cc0df32c62c9e2025-05-12T07:07:54Z2025-05-12T07:08:06ZREADME: tweak "how does it work"Tobias Bengforttobias.bengfort@posteo.deREADME: tweak "how does it work"
d4608364ed033cd73186aa7f05ec76b777bbf42c2025-05-12T07:07:38Z2025-05-12T07:07:47ZREADME: inline classifier codeTobias Bengforttobias.bengfort@posteo.deREADME: inline classifier code
430f0819379897f02e6e8af12bb744f17030b2e12025-05-10T18:34:26Z2025-05-10T18:34:26Ztweak readmeTobias Bengforttobias.bengfort@posteo.detweak readme
e160d8cb04fb6d6e6c762edcb27cbc4f704fb1d12025-05-10T18:32:16Z2025-05-10T18:32:16Zgen_model: most significant firstTobias Bengforttobias.bengfort@posteo.degen_model: most significant first
315458d0a09b6e790689e7e56ccb0428eca4a4c22025-05-10T18:29:51Z2025-05-10T18:29:51Zgen_model: allow to pass more than two languagesTobias Bengforttobias.bengfort@posteo.degen_model: allow to pass more than two languages
281071942a7647e36686aae2deb2511b2079b6372025-05-10T18:07:41Z2025-05-10T18:07:41Zadd some explanationTobias Bengforttobias.bengfort@posteo.deadd some explanation
2479e1d6933082faf7d4fc5527d76f55c5087f3e2025-05-10T16:50:14Z2025-05-10T16:50:14Zlimit precisionTobias Bengforttobias.bengfort@posteo.delimit precision
127bc181a6b4c10fd266504951d68dbc2e9aa4ef2025-05-10T15:40:52Z2025-05-10T15:40:52Ztweak test outputTobias Bengforttobias.bengfort@posteo.detweak test output
3f86e6fb263df69ee5a826710b4c22496916add02025-05-10T15:36:15Z2025-05-10T15:36:15ZREADME: rm redundant lineTobias Bengforttobias.bengfort@posteo.deREADME: rm redundant line
49e8c884b8b27f1ae43ed1908aa228897c6c60072025-05-06T06:30:07Z2025-05-06T06:30:07Zdemo: add headingTobias Bengforttobias.bengfort@posteo.dedemo: add heading
5698b03845de35c9d8a99ede35bc7a090d5f40d22025-05-06T06:29:17Z2025-05-06T06:29:17Zinclude example model in READMETobias Bengforttobias.bengfort@posteo.deinclude example model in README
60da7d4b74c54273652a4bbdb84408bb9f0558862025-05-06T06:22:42Z2025-05-06T06:25:00Zadd READMETobias Bengforttobias.bengfort@posteo.deadd README
a3e48ff6c5886c46a18cbb9fa15646a3b1ce40412025-05-06T06:10:24Z2025-05-06T06:25:00Zadd js demoTobias Bengforttobias.bengfort@posteo.deadd js demo
1bf85abcec4588d0e3f0806016148a68ffb139182025-05-06T05:43:34Z2025-05-06T06:25:00Zadd test.pyTobias Bengforttobias.bengfort@posteo.deadd test.py
39930b5dc497f10d174802bfc943c5c18cdb4c6f2025-05-06T05:20:08Z2025-05-06T06:25:00Zadd gen_model.pyTobias Bengforttobias.bengfort@posteo.deadd gen_model.py
3db712ebeea5d1322b6ef9d9470955a5667102622025-05-06T05:06:16Z2025-05-06T06:25:00Zconvert to shell scriptTobias Bengforttobias.bengfort@posteo.deconvert to shell script
c2755acb5d4af6c990d63f827e18c83e33e20c692025-05-06T05:03:50Z2025-05-06T06:25:00Zdownload dataTobias Bengforttobias.bengfort@posteo.dedownload data