Provide a mechanism to disable variables

[parasyte]

Motivation

Variable interpolation can be useful in some contexts, but causes issues in others. One example where variables are unwanted is using Gura as a simple flat file database. Gura is a good alternative to CSV, JSON, YAML, and some other options in this use case. The existence of variables causes some problems, however.

1. Variables make using the $ literal awkward. A ledger example highlights this shortcoming:

company: "Acme tools"
ledger: [
    date: "2022-06-22"
    description: "Cash"
    debit: "\$613.00"
    credit: "\$0.00",

    date: "2022-06-22"
    description: "Sales"
    debit: "\$0.00"
    credit: "\$803.53",

    date: "2022-06-22"
    description: "Sales Tax Payable"
    debit: "\$0.00"
    credit: "\$200.88",
]

2. It is easy to forget about (or be unaware of) escaping the $ character. The parser raises VariableNotDefinedError in this situation.

3. Automatically pulling information from the environment may be a privacy or security violation in sensitive environments. Some examples include a website that accepts Gura file uploads and parses them on the backend for validation, or a native application that uses Gura for its embedded database and users decide to update the database from untrusted sources. (See also Provide a mechanism to disable imports #13.)

4. Some platforms may not provide access to environment variables at all, such as WASM VMs and web browsers.

Proposal

Each implementation must provide a mechanism to disable variable interpolation with environment variables, and also to disable variable interpolation altogether. The configuration option has three states:

1. Variables are enabled and using environment variables is allowed.
2. Variables are enabled and using environment variables are not allowed.
3. Variables are disabled.

The first state acts as Gura does today. The second state acts as it does today with the exception that the environment is never accessed; all variables must be defined in the Gura tree.

The third state treats all $ characters as literals, and all escaped \$ as the same literal $ character. In other words, existing Gura files continue to parse correctly, except variables are no longer interpolated. This state also allows unescaped $ literals so that strings like "$613.00" become valid. These files will not parse when variables are enabled. The parser raises VariableNotDefinedError in this situation.

Example in Python:

import gura

gura_str = '''
dollars: "$3.50"
'''

gura.loads(gura_str) # This will throw a VariableNotDefinedError
gura.loads(gura_str, variables="enabled") # Same as above
gura.loads(gura_str, variables="env_disabled") # Same as above
gura.loads(gura_str, variables="disabled") # Parses with literal "$3.50"

Considerations

• Should the configuration be simplified to a boolean? I.e. do not implement "state 2".
• Should the default be all variables disabled? This seems like a sane default (default-deny policy) but is not backward compatible.
• Allowing unescaped $ characters to be treated as literals will bifurcate the Gura ecosystem into applications that support these files and applications which do not. Is this an acceptable tradeoff? Is there a better alternative?
  • Variables could be removed entirely. An application would need to post-process the parsed structure if they wish to continue using variables.

Version

Gura 2.0.0 because it contains many other breaking changes, including the aforementioned #13.

Changes

1. Add a configuration to disable environment variable access and variable interpolation.
2. Include changes in a new Changelog section.

[facundoq]

1. Variables make using the $ literal awkward. A ledger example highlights this shortcoming:

This is true, but will be so with any syntax. Using other syntax such as {variable} requires escaping both { and }. The $ character was chosen because of it's also used in other languagues such as bash/php, but I agree that maybe a better character could be found.

2. It is easy to forget about (or be unaware of) escaping the $ character. The parser raises VariableNotDefinedError in this situation.

I think editor support could help on this.

3. Automatically pulling information from the environment may be a privacy or security violation in sensitive environments. Some examples include a website that accepts Gura file uploads and parses them on the backend for validation, or a native application that uses Gura for its embedded database and users decide to update the database from untrusted sources. (See also Provide a mechanism to disable imports #13.)

We agree, but this issue is orthogonal to variables. You can have variables without imports and viceversa.

4. Some platforms may not provide access to environment variables at all, such as WASM VMs and web browsers.

In that case, no harm done, you just have to be aware that your config file needs to be self contained.

Considerations

• Should the configuration be simplified to a boolean? I.e. do not implement "state 2".

This should be the default, yes, for security purposes.

• Should the default be all variables disabled? This seems like a sane default (default-deny policy) but is not backward compatible.

I don't think so. Just don't use variables and it's the same!

• Allowing unescaped $ characters to be treated as literals will bifurcate the Gura ecosystem into applications that support these files and applications which do not. Is this an acceptable tradeoff? Is there a better alternative?

I guess Gura could exist without variables, but I feel having to escape $ is a very low price to pay for having access to variables which can simplify many, many configuration files. I think we are open to changing our minds on this, but I don't see the use case you are mentioning as a strong counter example. Just remove the $ from the gura file and interpret those numbers as currency. Better yet, add fields named "debit_currency" and "credit_currency", with string values such as "USD", "EUR", etc.

• Variables could be removed entirely. An application would need to post-process the parsed structure if they wish to continue using variables.

I feel this is unnecessary, applications can escape $ signs when needed. The same goes for any keyword in Gura.

[parasyte]

We agree, but this issue is orthogonal to variables. You can have variables without imports and viceversa.

Maybe a small misunderstanding, here. For clarity, I linked to the thread about imports as an example of a similar security issue. In this case, the issue is with exposing environment variables, and is indeed unrelated to imports.

[parasyte]

Just remove the $ from the gura file and interpret those numbers as currency.

This is not a usable workaround. Raw currency is not the issue but using a currency representation in any text is a papercut. A string like "I owe \$100 by tomorrow" does not read nicely and cannot trivially be transformed to a number.

[facundoq]

Did you mean ¿environmental variable in every instance of the variable word in your op? If not, env variables and imports work similarly and both have security issues. But they are orthogonal to variables per se. Why disable all variables if env variables are the issue? El mié., 22 jun. 2022 12:34, Jay Oster ***@***.***> escribió:

…

We agree, but this issue is orthogonal to variables. You can have variables without imports and viceversa. Maybe a small misunderstanding, here. For clarity, I linked to the thread about imports as an example of a similar security issue. In this case, the issue is with exposing *environment variables*, and is indeed unrelated to imports. — Reply to this email directly, view it on GitHub <#18 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMA6B2ROGQBUPFTLDJWCBLVQMW77ANCNFSM5ZQR26UA> . You are receiving this because you commented.Message ID: ***@***.***>

[Genarito]

I agree with both parties:

First of all, it is true that it may represent security issues since any file that is parsed server-side would access the global variables exposing them in the parsed result. So it seems good to me to provide in the technical documentation a section indicating that tools implementing Gura serialization should offer the mentioned options:

• Both types of variables are enabled
• Local only
• ENV only
• Both types of variables are disabled

Secondly, regarding the default option, wouldn't option 2 (only local variables enabled) be better? Since that does not represent a security problem, it only generates an inconvenience in the case of having to define several times some string with the $ character, but there I agree with Facundo in that it is a problem that both the IDE, as the new options can help to solve. In any case, we can only guide those who implement the tools, which options we consider better by default, but it will remain at their criterion.

[parasyte]

I would like to make a stronger case for considering the deprecations and removal of variables. I'm not convinced this would be accepted, which is why it was left as a footnote in the original proposal.

Multiline literal strings already solve the problem with literal $ characters, but they look out of place when mixed with basic strings. Using multiline literal strings everywhere for consistency makes the entire file format look alien. One of the most attractive things about Gura, IMHO, is that it makes configuration reasonable clean and readable. Using multiline literal strings to work around variable escaping would detract from the cleanliness and readability, but it's an option.

Escaping the character within basic strings is a better choice, even if it also detracts from the cleanliness and readability... I just can't trust my users to always remember to add escapes. They don't care about the peculiarities of the spec or parser; they just want the configuration to be human readable and writable without special tools like an IDE extension.

I understand that my audience is not the target for Gura, and I might be trying to purpose it for a task that isn't a great fit. Right now, I am using YAML and it has been rather painful, despite being readable when features are used sparingly. Gura will solve many of the challenges I've faced at the cost of adding new ones (like accidental magic variables).

On the other hand, I believe that variables are going to be more often undesirable than useful. I say this because most of the configuration templating I've done starts out with something simple like variable substitution and slowly grows a need for iteration and computation like some template processors support. Ultimately, these are all things that belong in a DSL, not in my config files. With this perspective, it makes sense to not only disable variable interpolation (for security reasons or otherwise), but also to stop supporting any syntax for it entirely.

The other option, of course, is applications like mine writing a preprocessor to escape any unescaped $ characters in the config before handing it to the parser.

[Genarito]

There are several interesting points in your response. But I don't see the consideration of Literal Strings (no multi-lines, just single quotes ', read the String section for more). Your example could look like this:

company: 'Acme tools'
ledger: [
    date: '2022-06-22'
    description: 'Cash'
    debit: '$613.00'
    credit: '$0.00',

    date: '2022-06-22'
    description: 'Sales'
    debit: '$0.00'
    credit: '$803.53',

    date: '2022-06-22'
    description: 'Sales Tax Payable'
    debit: '$0.00'
    credit: '$200.88',
]

So no need to resort to multi-line literal strings or escaped basic strings for your requirement :D.

With this in mind, I still think it's a good idea to have a mechanism to disable variable consideration to prevent all the problems you mentioned. But as far as readability is concerned there are the following solutions:

• Use Literal Strings (although perhaps basic strings and literal strings will be mixed).
• Disable the use of variables in your tool so that, in case the configuration is defined by a user, he/she cannot insert a variable by accident.
• Set up the pre-processor you are talking about, but it seems to me unnecessary work since you have the two options above.

The use of variables was born from some annoyances I had when I had to define services in Docker. For example:

version: '3'

$user: 'test'
$pass: 'test'
$db: 'test'

services:
    # PostgreSQL
    db:
        image: postgres:13
        environment:
            POSTGRES_USER: $user
            POSTGRES_PASSWORD: $pass
            POSTGRES_DB: $db

    # Some service that needs to connect to PostgreSQL
    other_service:
        image: my_img
        environment:
            POSTGRES_USER: $user
            POSTGRES_PASSWORD: $pass
            POSTGRES_DB: $db

Now, scale the example to 5 interconnected services! The use of variables helps in this kind of problem. Of course, if security is compromised, the implementation must offer the mechanism to disable variables.

[parasyte]

I didn’t consider single-line literal strings because they do not allow using the single quote character ' at all. It is commonly used in English contractions, so it is expected to appear in strings.

The use of variables was born from some annoyances I had when I had to define services in Docker.

Yeah, I have done enough service configuration to be familiar with the problem and how variables address it. It happens everywhere. Helm, Ansible, Chef, …

[facundoq]

¿Why disable variables? Defined in the same file, they are self-contained. W By adding API ability to disable variables, you are effectively making many Gura config files invalid. Worse, the error will possibly be parsed as a correct string! e've already discussed additional templating mechanisms and decided Gura will not include those. It will stand in the middle between a simple config file and a complex templating system. @jay, one of your original concerns is about having to escape $. As Genaro mentioned, you can quote those, or as you said, you can escape them in your string before saving to a Gura file. This is not something unheard of, and you will also have to escape other characters such as "], etc. The other one is about safety with env variables. This is 100% correct. But disabling variables altogether is both unnecessary and dangerous.

…

On Thu, Jun 23, 2022 at 10:48 AM Genaro Camele ***@***.***> wrote: There are several interesting points in your response. But I don't see the consideration of Literal Strings (no multi-lines, just single quotes ', read the String section <https://gura.netlify.app/docs/spec#string> for more). Your example could look like this: company: 'Acme tools'ledger: [ date: '2022-06-22' description: 'Cash' debit: '$613.00' credit: '$0.00', date: '2022-06-22' description: 'Sales' debit: '$0.00' credit: '$803.53', date: '2022-06-22' description: 'Sales Tax Payable' debit: '$0.00' credit: '$200.88',] So no need to resort to multi-line literal strings or escaped basic strings for your requirement :D. With this in mind, I still think it's a good idea to have a mechanism to disable variable consideration to prevent all the problems you mentioned. But as far as readability is concerned there are the following solutions: - Use Literal Strings (although perhaps basic strings and literal strings will be mixed). - Disable the use of variables in your tool so that, in case the configuration is defined by a user, he/she cannot insert a variable by accident. - Set up the pre-processor you are talking about, but it seems to me unnecessary work since you have the two options above. The use of variables was born from some annoyances I had when I had to define services in Docker. For example: version: '3' $user: 'test'$pass: 'test'$db: 'test' services: # PostgreSQL db: image: postgres:13 environment: POSTGRES_USER: $user POSTGRES_PASSWORD: $pass POSTGRES_DB: $db # Some service that needs to connect to PostgreSQL other_service: image: my_img environment: POSTGRES_USER: $user POSTGRES_PASSWORD: $pass POSTGRES_DB: $db Now, scale the example to 5 interconnected services! The use of variables helps in this kind of problem. Of course, if security is compromised, the implementation must offer the mechanism to disable variables. — Reply to this email directly, view it on GitHub <#18 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMA6B3RZF2CYTRWCLO2U5TVQRTJ7ANCNFSM5ZQR26UA> . You are receiving this because you commented.Message ID: ***@***.***>

[Genarito]

Yep, the incompatibility issue with the same Gura file sounds bad... So @facundoq, you propose only allowing to disable ENV vars in parsers? And the rest of the implementation specs would remain the same, wouldn't it?

Also, @parasyte if a Gura file is being generated programmatically, a tokenization section will be available in Gura 2.0 that will allow new implementations to generate a document structure safely.

If we decide, this week I will add the ENV vars disabling documentation along with a new standardized error for that case (just like ImportDisabledError when imports are disabled).

[parasyte]

There are certainly many ways to avoid having to escape $ (I think they have all been exhaustively mentioned here now) and they all have downsides. It’s a tradeoff that makes using Gura as a human readable config format annoying. In my case ” is expected far less frequently than $ in strings, so I could live with single line basic strings if I didn’t have to escape the more common character.

These files are hand-written (or at least maintained by hand) so tools don’t really help, except I can avoid VariableNotFoundError with a preprocessor if a user forgets to (or doesn’t want to) escape $.

Worse, the error will possibly be parsed as a correct string!

haha! That’s pretty much what I want! https://xkcd.com/1172/

[Genarito]

haha! That’s pretty much what I want! https://xkcd.com/1172/

Hahahahahah exactly, let's try to break the previous documents as little as possible 😄

Regarding the use of apostrophes in strings, it is a problem that occurs in any configuration language and ends up being irremediable: for example, in JSON only double quotes can be used, so its use (very common in Spanish, for example) must always be escaped by the writer. The same happens with the words "yes" or "no" in YAML.
The important thing here is that the use of apostrophes in Literal Strings will throw a syntax error, so all problems are solved! So IMHO Literal Strings are the cleanest solution for your problem, using them:

1. Your documents remain homogeneous (no mix of Basic Strings and Literal Strings). Only single quotes!
2. You can use the $ sign without worrying about variable interpretation.
3. In case you need to write an apostrophe and the user forgets to escape it, any IDE will indicate the error, and in the case of reaching the program you will end up with a clear syntax problem that is easily solvable and whose error message is explicit.
4. Unfortunately you will not be able to avoid the problem of escaping characters, this happens with SQL queries and any other configuration language deserialization tool.

The problem with ENVs vars is solved with the new standard error and we do not introduce the problems mentioned by Facundo 💯

[parasyte]

The conclusion from this discussion I think is that the best way to avoid parser errors with fallible human authors is to recommend mixing single-line and multiline literal strings in their documents. This is probably an adequate compromise, even though the "choose your own adventure" feeling of which string format to use could still be a problem for noobs and the incorrigible.

As you showed in #18 (reply in thread), the document still looks readable with literal strings, and multiline literal strings can be used when apostrophes are needed. This is probably a good practice anyway, to avoid accidental variable interpolation.

Thanks for the back and forth @Genarito and @facundoq!

[Genarito]

Thank you @parasyte for making Gura better! 🍻 🚀

[facundoq]

On Thu, Jun 23, 2022 at 11:59 AM Genaro Camele ***@***.***> wrote: Yep, the incompatibility issue with the same Gura file sounds bad... So @facundoq <https://github.com/facundoq>, you propose only allowing to disable ENV vars in parsers? And the rest of the implementation specs would remain the same, wouldn't it?

Exactly. As Jay says, it would be safer to disable it by default so that variables are not read inadvertently.

…

Message ID: ***@***.*** com>

[facundoq]

I think it's important to point out that you will ALWAYS need to escape something. $ and ' are some of the cases, but not the only ones. Perhaps an API function to safely do this that will automatically update with new versions would come in handy. But there's no going around the fact that the file itself will show the escaped characters. El jue., 23 jun. 2022 14:30, Jay Oster ***@***.***> escribió:

…

I didn’t consider single-line literal strings because they do not allow using the single quote character ' at all. It is commonly used in English contractions, so it is expected to appear in strings. The use of variables was born from some annoyances I had when I had to define services in Docker. Yeah, I have done enough service configuration to be familiar with the problem and how variables address it. It happens everywhere. Helm, Ansible, Chef, … — Reply to this email directly, view it on GitHub <#18 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMA6BZ5Y6VCZPALTIYFWW3VQSNNDANCNFSM5ZQR26UA> . You are receiving this because you were mentioned.Message ID: ***@***.***>