Also improve the various self.key.possible calls to calculate more
correctly whether a key is required.
This commit adds support for the Scanner to pick up
Flow{Mapping,Sequence}{Start,End} tokens, but _does not_ yet allow for
FlowEntry tokens.
I misunderstood YAML key semantics around implicit keys, thinking that
only plain (e.g non flow / block) scalars required length and line limit
checking, but it turns out that _all_ possible implicit key variants
have these limits, so I've added checks here, and corrected a bad test
The external Scanner API now requires a .tokens list to add tokens
into.
Where possible, it will still attempt to only store one token per call,
however it may store more if the situation requires, typically when
calculating if a given scalar is a key, or when handling indentation
tokens.
This is a part of an API changes I"ll be making, which will allow
allocations in the scanner code. This change is being made for a few
reasons.
1. Allows me to make the Scanner API nicer, as callers will only need
to pass in the underlying data being scanned, and will not be tied to
a mutable lifetime which limits them to scanning tokens one at a
time.
2. Makes the code simpler, as I no longer need to ensure the mutable
'owned' lifetime is honored throughout the call stack.
3. I'll need to allocate anyway for the indentation stack, and thus not
allocating in other places that are sensible is less important.
This module contains the beginnings of the state required to track and
store tokens which may be "found" out of sequence, notably when we first
need to parse a scalar (Token #1) then check if a value sequence follows
it (Token #2), and if so, return a key token first (Token #3), where the
correct order of tokens is:
Token #3 -> Token #1 -> Token #2
We also need to track in the scanner whether a key is even possible, e.g
if we just parsed a value token, the next scalar _is not_ a key
this allows callers to decide when to convert the range into a Token
ref, which will be important when we need to save a scalar because we
need to return a Key Token first
A struct for doing book keeping about where we are in the buffer:
1. How much we've read
2. How many lines we've seen
3. The current column
I'll likely add variants to advance!, as its the primary method used to
traverse the buffer
This will likely be passed as an extra argument down the various scan
call stacks, and care will need to be taken to ensure we're handling
line breaks correctly (because I bet we're not currently)
Tests will need to be updated to test that we're getting the stats we
expect.
An unfortunate glitch in the compiler requires that I use a match
statement over fall through if guards, as the borrow checker is to
restrictive, and will not allow the code to compile despite being
clearly correct.
This required me to lift the stream checks into next_token, which means
we now have a redundant check. Hopefully in the future the borrow
checker will become smarter.
This commit also refactors stream checks to use a const identifier shared
between the call site in next_token and the function proper, and misc
changes to Scanner.eat_whitespace
As we'll be using it throughout the various scanner/* modules, not just
on the scanner struct itself.
This commit also improves eat_whitespace to chomp any valid YAML
whitespace, not just newlines and spaces.
Split out tag scanning functions into their own module.
This commit includes two functions scan_tag_uri, and scan_tag_handle for
process prefix/suffix'es and handles respectively.
Note that these functions by themselves cannot properly parse either %TAG
directives or YAML node tags; but higher level functions can use these to
correctly scan both.
Paul Stemmet