Feature/scanner/keys #16

Merged

bazaah merged 20 commits from feature/scanner/keys into master

2021-07-25 11:41:57 +00:00

bazaah commented

2021-07-25 11:05:20 +00:00

(Migrated from github.com)

Components

Adds the ability to scan certain implicit keys
Modifies the Scanner public API to take a Tokens<'de> list to read tokens into
Removes the requirement to pass scratch space into the Scanner, as it will generate it's own where required

API Changes

The Tokens<'de> typedef holds parsed tokens which the caller can read from at their leisure. This change is important for a couple reasons. First it allows the Scanner to return >1 token per call, which greatly simplifies certain aspects of the Scanner, notably how it handles Token::Key generation. However this will also positively impact the handling of graph depth tokens generally (both flow and block), and likely provide room for optimizations around multi-doc streams.

This change also decouples each Token's lifetime from the previously provided &'c mut scratch space, which lifetimes restricted to only a single token. Instead we now utilize Slice<'de>'s ability to transport an owned String out thus removing the lifetime interaction, and allowing callers to retrieve (and thus parse) multiple tokens at once, which is a win for usability.

Of course, the downside of this is that:

This lib becomes alloc (and therefore std) dependent
Callers have less control over Scanner memory usage

However 1. would happen anyway (due to how block indents must be stored) and may be fixable later by providing some feature that replaces the Vec and Strings with a trait over fallible allocation... or it may just be fixed for us by the boys over in Rust Kernel land merging their changes to alloc upstream.

This PR adds limited support for scanning implicit YAML keys, currently only support flow scalar nodes. It expands test coverage to cover the new functionality, and also makes an API change to the scanner proper. ## Components - Adds the ability to scan certain implicit keys - Modifies the Scanner public API to take a `Tokens<'de>` list to read tokens into - Removes the requirement to pass scratch space into the Scanner, as it will generate it's own where required ## API Changes The `Tokens<'de>` typedef holds parsed tokens which the caller can read from at their leisure. This change is important for a couple reasons. First it allows the Scanner to return >1 token per call, which greatly simplifies certain aspects of the Scanner, notably how it handles `Token::Key` generation. However this will also positively impact the handling of graph depth tokens generally (both flow and block), and likely provide room for optimizations around multi-doc streams. This change also decouples each Token's lifetime from the previously provided `&'c mut scratch` space, which lifetimes restricted to only a single token. Instead we now utilize `Slice<'de>`'s ability to transport an owned `String` out thus removing the lifetime interaction, and allowing callers to retrieve (and thus parse) multiple tokens at once, which is a win for usability. Of course, the downside of this is that: 1. This lib becomes `alloc` (and therefore `std`) dependent 2. Callers have less control over Scanner memory usage However 1. would happen anyway (due to how block indents must be stored) and may be fixable later by providing some feature that replaces the `Vec` and `String`s with a trait over fallible allocation... or it may just be fixed for us by the boys over in Rust Kernel land merging their changes to `alloc` upstream.