X-Authentication-Warning: delorie.com: mail set sender to djgpp-bounces using -f
Date: Sat, 15 May 2004 14:54:52 +0200
From: "Eli Zaretskii" <eliz AT gnu DOT org>
Sender: halo1 AT zahav DOT net DOT il
To: djgpp AT delorie DOT com
Message-Id: <6654-Sat15May2004145451+0300-eliz@gnu.org>
X-Mailer: emacs 21.3.50 (via feedmail 8 I) and Blat ver 1.8.9
In-reply-to: <10abt7pn4699h2a@corp.supernews.com> (nospam@nspam.net)
Subject: Re: Tokenizing
References:  <10abt7pn4699h2a AT corp DOT supernews DOT com>
Reply-To: djgpp AT delorie DOT com
Errors-To: nobody AT delorie DOT com
X-Mailing-List: djgpp AT delorie DOT com
X-Unsubscribes-To: listserv AT delorie DOT com
Precedence: bulk

> From: "Bill Cunningham" <nospam AT nspam DOT net>
> Newsgroups: comp.os.msdos.djgpp
> Date: Sat, 15 May 2004 06:47:12 -0400
> 
>     I've just learned of something new called tokenizing. Is this done by
> djgpp before parsing?

It's also called ``lexical analysis''.  And yes, GCC, like any other C
compiler, subdivides the source into tokens before it parses the code.