Emoji AST Interpreter Lexer Stage - Part 1

Before a computer can add a banana to an apple, it has to stop seeing a wall of characters and start seeing meaning, and that first translation from raw text into labeled pieces is where every compiler and interpreter quietly begins its real work.

This series builds a tiny playful language where fruit emoji stand in for numbers and plain symbols do the actual math, and this first part focuses only on the lexer, the humble stage that slices raw source text into a clean orderly stream of tokens.

Nothing here evaluates anything yet, the goal is simply to turn a messy string into a tidy list of tokens the parser can trust.

What a Lexer Really Does

A lexer, sometimes called a tokenizer or scanner, reads the source one character at a time and groups those characters into meaningful units called tokens, so an apple emoji becomes a single number token, a plus sign becomes an operator token, and a run of spaces simply vanishes because whitespace carries no meaning here and only exists to keep the source readable.

Think of it like reading a sentence out loud, the eyes do not consciously process the individual little ink dots on the page, they recognize whole words at a glance, and the lexer hands the rest of the compiler clean words instead of raw letters.

The output is a flat list, no nesting and no structure yet, just tokens in the exact order they appeared in the original text.

Designing the Token

Ahead of scanning anything, the shape of a token needs to be defined, because every stage after the lexer will depend on that shape, and a token in this language only needs two things, a kind that says what category it belongs to and a value that holds the literal text that was matched from the source, so nothing about the original input is ever lost along the way.

Three token kinds cover the language, a number for fruit emoji, an operator for symbols, and an end marker that closes the input.

type TokenKind = 'number' | 'operator' | 'eof'

interface Token {
  kind: TokenKind
  value: string
}

const FRUIT = new Set(['🍎', '🍌', '🍒', '🍇', '🍓'])
const OPERATORS = new Set(['+', '-', '*', '/', '(', ')'])

Scanning Character by Character

With the token shape settled, the scanner itself is surprisingly small, it keeps a cursor at the current position, looks at the character under that cursor, decides which category it falls into, produces the matching token, and then advances the cursor past whatever it just consumed, repeating that loop until the entire source string has been walked start to finish.

One subtle detail trips up almost everyone the first time, an emoji is not a single character the way a letter is, many emoji occupy two code units in a JavaScript string, so reading one character at a time splits an apple in half and makes garbage.

Iterating with the string built in iterator fixes this cleanly, since it walks by whole code points, so each fruit stays one unit.

function tokenize(source: string): Token[] {
  const tokens: Token[] = []
  for (const char of source) {
    if (char === ' ' || char === '\t' || char === '\n') {
      continue
    }
    if (FRUIT.has(char)) {
      tokens.push({ kind: 'number', value: char })
      continue
    }
    if (OPERATORS.has(char)) {
      tokens.push({ kind: 'operator', value: char })
      continue
    }
    throw new SyntaxError(`Unexpected character: ${char}`)
  }
  tokens.push({ kind: 'eof', value: '' })
  return tokens
}

Trying It Out

Running the lexer on a small expression like an apple plus two bananas returns a neat array of tokens, a number token carrying the apple, an operator token carrying the plus, another number token carrying the banana, and finally the end marker, and seeing that flat ordered list appear is the first moment the tiny language stops being a string and starts feeling real.

Notice what the lexer deliberately does not do, it never checks whether the expression actually makes any sense at all, a stray plus with nothing after it still tokenizes perfectly cleanly because catching that mistake is squarely the parser's job, not the lexer's, and keeping these two responsibilities separate is what keeps each stage small and easy to reason about.

That separation is really the whole point, the lexer answers what are the pieces, and the next part answers how the pieces fit.

console.log(tokenize('🍎 + 🍌'))
// [
//   { kind: 'number', value: '🍎' },
//   { kind: 'operator', value: '+' },
//   { kind: 'number', value: '🍌' },
//   { kind: 'eof', value: '' }
// ]