Received-SPF: pass (google.com: domain of gleki.is.my.name@gmail.com designates 2a00:1450:400c:c09::229 as permitted sender) client-ip=2a00:1450:400c:c09::229;
MIME-Version: 1.0
From: Gleki Arxokuna <gleki.is.my.name@gmail.com>
Date: Mon, 8 Feb 2016 16:35:14 +0300
Message-ID: <CAO7bV+gTJJshqBp1yq4U1ON57S=_s6dwnh4OmKTa3oa8+VAzCw@mail.gmail.com>
Subject: [lojban] N-grams of Lojban corpus
To: "lojban@googlegroups.com" <lojban@googlegroups.com>
Content-Type: multipart/alternative; boundary=001a1145bc8ab0700f052b42470e
Reply-To: lojban@googlegroups.com
Precedence: list
Mailing-list: list lojban@googlegroups.com; contact lojban+owners@googlegroups.com
Sender: lojban@googlegroups.com
X-Spam_score: -1.7
X-Spam_score_int: -16
X-Spam_bar: -

--001a1145bc8ab0700f052b42470e
Content-Type: text/plain; charset=UTF-8

For various reasons we may need stats of N-grams from Lojban corpus.

Not that it's hard to generate such stats.

But we first need to preprocess the log of our history:
http://www.lojban.org/irclogs/irclogs.zip

Definitely, messages from "mensi", "livla" must be removed.

Anything else?

I'd like to eventually develop an algorithm of preprocessing this log.
Any help is welcomed.


I started adding different lists of N-grams here:
https://mw.lojban.org/papri/N-grams_of_Lojban_corpus

But spreadsheets might be needed instead since list can be long.

PS. If you wonder where N-grams might be needed the immediate application
is "collect most frequent phrases in Lojban and make a phrasebook out of
that".

-- 
You received this message because you are subscribed to the Google Groups "lojban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lojban+unsubscribe@googlegroups.com.
To post to this group, send email to lojban@googlegroups.com.
Visit this group at https://groups.google.com/group/lojban.
For more options, visit https://groups.google.com/d/optout.

--001a1145bc8ab0700f052b42470e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">For various reasons we may need stats of N-grams from Lojb=
an corpus.<div><br></div><div>Not that it&#39;s hard to generate such stats=
.</div><div><br></div><div>But we first need to preprocess the log of our h=
istory:</div><div><a href=3D"http://www.lojban.org/irclogs/irclogs.zip">htt=
p://www.lojban.org/irclogs/irclogs.zip</a><br></div><div><br></div><div>Def=
initely, messages from &quot;mensi&quot;, &quot;livla&quot; must be removed=
.</div><div><br></div><div>Anything else?</div><div><br></div><div>I&#39;d =
like to eventually develop an algorithm of preprocessing this log.</div><di=
v>Any help is welcomed.</div><div><br></div><div><br></div><div>I started a=
dding different lists of N-grams here:=C2=A0<a href=3D"https://mw.lojban.or=
g/papri/N-grams_of_Lojban_corpus">https://mw.lojban.org/papri/N-grams_of_Lo=
jban_corpus</a></div><div><br></div><div>But spreadsheets might be needed i=
nstead since list can be long.</div><div><br></div><div>PS. If you wonder w=
here N-grams might be needed the immediate application is &quot;collect mos=
t frequent phrases in Lojban and make a phrasebook out of that&quot;.</div>=
</div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;lojban&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:lojban+unsubscribe@googlegroups.com">lojban+unsub=
scribe@googlegroups.com</a>.<br />
To post to this group, send email to <a href=3D"mailto:lojban@googlegroups.=
com">lojban@googlegroups.com</a>.<br />
Visit this group at <a href=3D"https://groups.google.com/group/lojban">http=
s://groups.google.com/group/lojban</a>.<br />
For more options, visit <a href=3D"https://groups.google.com/d/optout">http=
s://groups.google.com/d/optout</a>.<br />

--001a1145bc8ab0700f052b42470e--