lib/scanner: fix missing token production for zero indents

So, the underlying bug here is caused because we check for zero indents
*before* unwinding the indentation stack. Take the following example:

```
 0|Map1:
 1|- Map2: value
 2|  Map2.1: value
 3|Map3: ...
  ----------
  0123456789
```

At (1,0) we encounter to indentation level increases -- map, sequence --
but zero actual indentation increase until (1,3) ('M' in 'Map2').

So, when we arrive at (3,0), Scanner::unroll_indent/2 only unrolls one
indent level... the second one (sequence) which contains the information
we use to determine whether an indent level has a zero indent that needs
to be added to the queue. Thus, when we hit the call to
Scanner::pop_zero_indent_sequence/2 it looks at the current indent
(map), which it correctly decides isn't zero indented.

The fix is simple, just reorder the operations to check for zero indents
before we unroll.

That said, I probably should create some additional test cases to check
that we handle these edge cases correctly.

1. Zero indent before and after real indent: (Z,N,Z)
2. Double zero indent -- not sure this is possible in YAML (Z,Z)

Issue: #54
This commit is contained in:
Paul Stemmet 2022-11-02 10:49:10 +00:00
parent 1dfa3def22
commit 835ceb0b97
Signed by: Paul Stemmet
GPG Key ID: EDEA539F594E7E75
1 changed files with 1 additions and 1 deletions

View File

@ -139,8 +139,8 @@ impl Scanner
self.expire_stale_saved_key()?; self.expire_stale_saved_key()?;
// Handle indentation unrolling // Handle indentation unrolling
self.pop_zero_indent_sequence(*base, tokens)?;
self.unroll_indent(tokens, self.stats.column)?; self.unroll_indent(tokens, self.stats.column)?;
self.pop_zero_indent_sequence(*base, tokens)?;
// Is it the end of a stream? // Is it the end of a stream?
if base.is_empty() || self.state == StreamState::Done if base.is_empty() || self.state == StreamState::Done