Envelope-to: lojban-list-archive@lojban.org
Delivery-date: Tue, 14 Jun 2022 12:33:31 -0700
Sender: lojban@googlegroups.com
Received-SPF: pass (google.com: domain of scope845hlang343jbo@icebubble.org designates 2607:f2f8:a1d8::b19:0:f0b as permitted sender) client-ip=2607:f2f8:a1d8::b19:0:f0b;
From: scope845hlang343jbo@icebubble.org
To: lojban@googlegroups.com
Subject: [lojban] Re: Lojban tokenizer for machine learning, first version
References: <d1a72031-b3ed-4164-bfba-bfa5fa65893bn@googlegroups.com>
Date: Tue, 14 Jun 2022 14:58:57 +0000
In-Reply-To: <d1a72031-b3ed-4164-bfba-bfa5fa65893bn@googlegroups.com> (Oleg
	Parashchenko's message of "Sun, 12 Jun 2022 01:32:55 -0700 (PDT)")
Message-ID: <86letzxpe6.fsf@cmarib.ramside>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Reply-To: lojban@googlegroups.com
Precedence: list
Mailing-list: list lojban@googlegroups.com; contact lojban+owners@googlegroups.com
X-Spam_score: -7.8
X-Spam_score_int: -77
X-Spam_bar: -------

Oleg Parashchenko <olpa@uucode.com> writes:

> I've just released the first version of a lojban tokenizer. It is intended 
> for use in machine learning applications and therefore is a bit different 
> from a linguistic tokenizer. In particular, it does sub-word tokenization.
>
> Additionally, there is a lexer, which can be used to develop alternative 
> tokenizers.

.uanai How is that different from any of the other Lojban parsers that
have been written?  I am interested in your lexer, however.  Which
version of the grammar did you use?  The PEG?  I'd be very curious to
see how your lexer distinguishes between lujvo and fu'ivla.

-- 
You received this message because you are subscribed to the Google Groups "lojban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lojban+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lojban/86letzxpe6.fsf%40cmarib.ramside.