diff --git a/docs/src/starks/protocol.md b/docs/src/starks/protocol.md index 4aa91760d..1fb4c5626 100644 --- a/docs/src/starks/protocol.md +++ b/docs/src/starks/protocol.md @@ -4,14 +4,15 @@ In this section we describe precisely the STARKs protocol used in Lambdaworks. We begin with some additional considerations and notation for most of the relevant objects and values to refer to them later on. -### Grinding -This is a technique to increase the soundness of the protocol by adding proof of work. It works as follows. At some fixed point in the protocol, the verifier sends a challenge $x$ to the prover. The prover needs to find a string $y$ such that $H(x || y)$ begins with a predefined number of zeroes. Here $x || y$ denotes the concatenation of $x$ and $y$, seen as bit strings. -The number of zeroes is called the *grinding factor*. The hash function $H$ can be any hash function, independent of other hash functions used in the rest of the protocol. In Lambdaworks we use Keccak256. - ### Transcript The Fiat-Shamir heuristic is used to make the protocol noninteractive. We assume there is a transcript object to which values can be added and from which challenges can be sampled. +### Grinding +This is a technique to increase the soundness of the protocol by adding proof of work. It works as follows. At some fixed point in the protocol, a value $x$ is derived in a deterministic way from all the interactions between the prover and the verifier up to that point (the state of the transcript). The prover needs to find a string $y$ such that $H(x || y)$ begins with a predefined number of zeroes. Here $x || y$ denotes the concatenation of $x$ and $y$, seen as bit strings. +The number of zeroes is called the *grinding factor*. The hash function $H$ can be any hash function, independent of other hash functions used in the rest of the protocol. In Lambdaworks we use Keccak256. + + ## General notation - $\mathbb{F}$ denotes a finite field. @@ -33,7 +34,7 @@ These values are determined the program, the specifications of the AIR being us - $m$ is the total number of columns: $m := m' + m''$. - $P_k^T$ denote the transition constraint polynomials for $k=1,\dots,n_T$. We are assuming these are of degree at most 2. - $Z_j^T$ denote the transition constraint zerofiers for $k=1,\dots,n_T$. -- $b=2^l$ is the *[blowup factor](/starks/protocol_overview.html#general-case-the-blowup-factor)*. +- $b=2^l$ is the *[blowup factor](/starks/protocol_overview.html#fri)*. - $c$ is the *grinding factor*. - $Q$ is number of FRI queries. - We assume there is a fixed hash function from $\mathbb{F}$ to binary strings. We also assume all Merkle trees are constructed using this hash function. @@ -58,13 +59,16 @@ Both prover and verifier compute the following. Given a vector $A=(y_0, \dots, y_L)$. The operation $\text{Commit}(A)$ returns the root $r$ of the Merkle tree that has the hash of the elements of $A$ as leaves. -For $i\in[0,2^{n+k})$, the operation $\text{open}(A, i)$ returns $y_i$ and the authentication path $s$ to the Merkle tree root. +For $i\in[0,2^{n+k})$, the operation $\text{Open}(A, i)$ returns the pair $(y_i, s)$, where $s$ is the authentication path to the Merkle tree root. The operation $\text{Verify}(i,y,r,s)$ returns _Accept_ or _Reject_ depending on whether the $i$-th element of $A$ is $y$. It checks whether the authentication path $s$ is compatible with $i$, $y$ and the Merkle tree root $r$. In our cases the sets $A$ will be of the form $A=(f(a), f(ab), f(ab^2), \dots, f(ab^L))$ for some elements $a,b\in\mathbb{F}$. It will be convenient to use the following abuse of notation. We will write $\text{Open}(A, ab^i)$ to mean $\text{Open}(A, i)$. Similarly, we will write $\text{Verify}(ab^i, y, r, s)$ instead of $\text{Verify}(i, y, r, s)$. Note that this is only notation and $\text{Verify}(ab^i, y, r, s)$ is only checking that the $y$ is the $i$-th element of the commited vector. +##### Batch +As we mentioned in the [protocol overview](protocol_overview.html#batch). When committing to multiple vectors $A_1, \dots, A_k$, where $A_i = (y_0^{(i), \dots, y_L^{(i)}})$ one can build a single Merkle tree. Its $j$-th leaf is the concatenation of all the $j$-th coordinates of all vectors, that is, $(y_j^{(1)}||\dots||y_j^{(k)})$. The commitment to this batch of vectors is the root of this Merkle tree. + ## Protocol ### Prover @@ -92,15 +96,15 @@ In our cases the sets $A$ will be of the form $A=(f(a), f(ab), f(ab^2), \dots, f #### Round 2: Construction of composition polynomial $H$ -- Sample $\alpha_1^B,\dots,\alpha_{m}^B$ and $\beta_1^B,\dots,\beta_{m}^B$ in $\mathbb{F}$ from the transcript. -- Sample $\alpha_1^T,\dots,\alpha_{n_T}^T$ and $\beta_1^T,\dots,\beta_{n_T}^T$ in $\mathbb{F}$ from the transcript. +- Sample $\beta_1^B,\dots,\beta_{m}^B$ in $\mathbb{F}$ from the transcript. +- Sample $\beta_1^T,\dots,\beta_{n_T}^T$ in $\mathbb{F}$ from the transcript. - Compute $B_j := \frac{t_j - P^B_j}{Z_j^B}$. - Compute $C_k := \frac{P^T_k(t_1, \dots, t_m, t_1(gX), \dots, t_m(gX))}{Z_k^T}$. - Compute the _composition polynomial_ $$H := \sum_{k} \beta_k^TC_k + \sum_j \beta_j^BB_j$$ - Decompose $H$ as $$H = H_1(X^2) + XH_2(X^2)$$ -- Compute commitments $[H_1]$ and $[H_2]$. +- Compute commitments $[H_1]$ and $[H_2]$ (*Batch commitment optimization applies here*). - Add $[H_1]$ and $[H_2]$ to the transcript. #### Round 3: Evaluation of polynomials at $z$ @@ -116,35 +120,30 @@ In our cases the sets $A$ will be of the form $A=(f(a), f(ab), f(ab^2), \dots, f ##### Round 4.1.k: FRI commit phase -- Let $D_0:=D_{\text{LDE}}$, and $[p_0]:=\text{Commit}(p_0(D_0))$. -- Add $[p_0]$ to the transcript. +- Let $D_0:=D_{\text{LDE}}$. - For $k=1,\dots,n$ do the following: - Sample $\zeta_{k-1}$ from the transcript. - Decompose $p_{k-1}$ into even and odd parts, that is, $p_{k-1}=p_{k-1}^{odd}(X^2)+ X p_{k-1}^{even}(X^2)$. - Define $p_k:= p_{k-1}^{odd}(X) + \zeta_{k-1}p_{k-1}^{even}(X)$. - If $k < n$: - - Let $L$ such that $|D_{k-1}|=2L$. Define $D_{k}:=(d_1^2, \dots, d_L^2)$, where $D_{k-1}=(d_1, \dots, d_{2L})$. + - Let $L = |D_{k-1}|/2$. Define $D_{k}:=(d_0^2, \dots, d_{L-1}^2)$, where $D_{k-1}=(d_0, \dots, d_{2L-1})$. - Let $[p_k]:=\text{Commit}(p_k(D_k))$. - Add $[p_k]$ to the transcript. - $p_n$ is a constant polynomial and therefore $p_n\in\mathbb{F}$. Add $p_n$ to the transcript. ##### Round 4.2: Grinding -- Sample $x$ from the transcript. +- Let $x$ be the internal state of the transcript. - Compute $y$ such that $\text{Keccak256}(x || y)$ has $c$ leading zeroes. - Add $y$ to the transcript. ##### Round 4.3: FRI query phase - For $s=0,\dots,Q-1$ do the following: - - Sample random index $\iota_s \in [0, 2^{n+l}]$ from the transcript and let $\upsilon_s := \omega^{\iota_s}$. - - Compute $\text{Open}(p_0(D_0), \upsilon_s)$. - - Compute $\text{Open}(p_k(D_k), -\upsilon_s^{2^k})$ for all $k=0,\dots,n-1$. - - -##### Round 4.4: Open deep composition polynomial components -- For $s=0,\dots,Q-1$ do the following: - - Compute $\text{Open}(H_1(D_{\text{LDE}}), \upsilon_s)$, $\text{Open}(H_2(D_{\text{LDE}}), \upsilon_s)$. - - Compute $\text{Open}(t_j(D_{\text{LDE}}), \upsilon_s)$ for all $j=1,\dots, m$. + - Sample random index $\iota_s \in [0, 2^{n+l-1}]$ from the transcript and let $\upsilon_s := h\omega^{\iota_s}$. + - Compute $\text{Open}(t_j(D_{\text{LDE}}), \upsilon_s)$ and $\text{Open}(t_j(D_{\text{LDE}}), -\upsilon_s)$ for all $j=1,\dots, m$. + - Compute $\text{Open}(H_1(D_{\text{LDE}}), \upsilon_s)$ and $\text{Open}(H_1(D_{\text{LDE}}), -\upsilon_s)$. + - Compute $\text{Open}(H_2(D_{\text{LDE}}), \upsilon_s)$ and $\text{Open}(H_2(D_{\text{LDE}}), -\upsilon_s)$. + - Compute $\text{Open}(p_k(D_k), \upsilon_s^{2^k})$ and $\text{Open}(p_k(D_k), -\upsilon_s^{2^k})$ for all $k=1,\dots,n-1$. #### Build proof @@ -154,41 +153,48 @@ $$ \Pi = ( &\\ &\{[t_j], t_j(z), t_j(gz): 0\leq j < m\}, \\ &[H_1], H_1(z^2),[H_2], H_2(z^2), \\ -&\{[p_k]: 0\leq k < n\}, \\ +&\{[p_k]: 1\leq k < n\}, \\ &p_n, \\ &y, \\ -&\{\text{Open}(p_0(D_0), \upsilon_s): 0\leq s < Q\}), \\ -&\{\text{Open}(p_k(D_k), -\upsilon_s^{2^k}): 0\leq k< n, 0\leq s < Q\}, \\ +&\{\text{Open}(t_j(D_{\text{LDE}}), \upsilon_s): 0 \leq j< m, 0 \leq s < Q\}, \\ &\{\text{Open}(H_1(D_{\text{LDE}}), \upsilon_s): 0 \leq s < Q\}, \\ &\{\text{Open}(H_2(D_{\text{LDE}}), \upsilon_s): 0 \leq s < Q\}, \\ -&\{\text{Open}(t_j(D_{\text{LDE}}), \upsilon_s): 0 \leq j< m, 0 \leq s < Q\}, \\ +&\{\text{Open}(p_k(D_k), \upsilon_s^{2^k}): 1\leq k< n, 0\leq s < Q\}, \\ +&\{\text{Open}(t_j(D_{\text{LDE}}), -\upsilon_s): 0 \leq j< m, 0 \leq s < Q\}, \\ +&\{\text{Open}(H_1(D_{\text{LDE}}), -\upsilon_s): 0 \leq s < Q\}, \\ +&\{\text{Open}(H_2(D_{\text{LDE}}), -\upsilon_s): 0 \leq s < Q\}, \\ +&\{\text{Open}(p_k(D_k), -\upsilon_s^{2^k}): 1\leq k< n, 0\leq s < Q\}, \\ ) & \end{align} $$ ### Verifier +From the point of view of the verifier, the proof they receive is a bunch of values that may or may not be what they claim to be. To make this explicit, we avoid denoting values like $t_j(z)$ as such, because that implicitly assumes that the value was obtained after evaluating a polynomial $t_j$ at $z$. And that's something the verifier can't assume. We use the following convention. -#### Notation - -- Bold capital letters refer to commitments. For example $\mathbf{H}_1$ is the claimed commitment $[H_1]$. -- Greek letters with superscripts refer to claimed function evaluations. For example $\tau_j^z$ is the claimed evaluation $t_j(z)$. -- Gothic letters refer to authentication paths (e.g. $\mathfrak{H}_1$ is the authentication path of the opening of $H_1$). +- Bold capital letters refer to commitments. For example $\mathbf{T}_j$ is the claimed commitment $[t_j]$. +- Greek letters with superscripts refer to claimed function evaluations. For example $\tau_j^z$ is the claimed evaluation $t_j(z)$ and $\tau_j^{gz}$ is the claimed evaluation of $t_j(gz)$. Note that field elements in superscripts never indicate powers. They are just notation. +- Gothic letters refer to authentication paths. For example $\mathfrak{T}_j$ is the authentication path of a opening of $t_j$. +- Recall that every opening $\text{Open}(A, i)$ is a pair $(y, s)$, where $y$ is the claimed value at index $i$ and $s$ is the authentication path. So for example, $\text{Open}(t_j(D_{\text{LDE}}), \upsilon_s)$ is denoted as $(\tau_j^{\upsilon_s}, \mathfrak{T}_j)$ from the verifier's end. #### Input +This is the proof using the notation described above. The elements appear in the same exact order as they are in the [Prover](#build-proof) section, serving also as a complete reference of the meaning of each value. $$ \begin{align} \Pi = ( &\\ &\{\mathbf{T}_j, \tau_j^z, \tau_j^{gz}: 0\leq j < m\}, \\ &\mathbf{H}_1, \eta_1^{z^2},\mathbf{H}_2, \eta_2^{z^2}, \\ -&\{\mathbf{P}_k: 0\leq k < n\}, \\ +&\{\mathbf{P}_k: 1\leq k < n\}, \\ &\pi, \\ &y, \\ -&\{(\pi_0^{\upsilon_s}, \mathfrak{P}_0): 0\leq s < Q\}, \\ -&\{(\pi_k^{-\upsilon_s^{2^k}}, \mathfrak{P}_k): 0\leq k< n, 0\leq s < Q\}, \\ +&\{(\tau_j^{\upsilon_s}, \mathfrak{T}_j): 0 \leq j< m, 0 \leq s < Q\}, \\ &\{(\eta_1^{\upsilon_s}, \mathfrak{H}_1): 0 \leq s < Q\}\\ &\{(\eta_2^{\upsilon_s}, \mathfrak{H}_2): 0 \leq s < Q\},\\ -&\{(\tau_j^{\upsilon_s}, \mathfrak{T}_j): 0 \leq j< m, 0 \leq s < Q\}, \\ +&\{(\pi_k^{\upsilon_s^{2^k}}, \mathfrak{P}_k): 1\leq k< n, 0\leq s < Q\}, \\ +&\{(\tau_j^{-\upsilon_s}, \mathfrak{T}_j'): 0 \leq j< m, 0 \leq s < Q\}, \\ +&\{(\eta_1^{-\upsilon_s}, \mathfrak{H}_1'): 0 \leq s < Q\}\\ +&\{(\eta_2^{-\upsilon_s}, \mathfrak{H}_2'): 0 \leq s < Q\},\\ +&\{(\pi_k^{-\upsilon_s^{2^k}}, \mathfrak{P}_k'): 1\leq k< n, 0\leq s < Q\}, \\ ) & \end{align} $$ @@ -196,7 +202,7 @@ $$ #### Step 1: Replay interactions and recover challenges - Start a transcript -- (Strong Fiat Shamir) Commit to the set of coefficients of the transition and boundary polynomials, and add the commitments to the transcript. +- (Strong Fiat Shamir) Add all public values to the transcript. - Add $\mathbf{T}_j$ to the transcript for all $j=1, \dots, m'$. - Sample random values $a_1, \dots, a_l$ from the transcript. - Add $\mathbf{T}_j$ to the transcript for $j=m' +1, \dots, m' + m''$. @@ -206,15 +212,13 @@ $$ - Sample $z$ from the transcript. - Add $\eta_1^{z^2}$, $\eta_2^{z^2}$, $\tau_j^z$ and $\tau_j^{gz}$ to the transcript. - Sample $\gamma$, $\gamma'$, and $\gamma_1, \dots, \gamma_m, \gamma'_1, \dots, \gamma'_m$ from the transcript. -- Add $\mathbf{P}_0$ to the transcript - For $k=1, \dots, n$ do the following: - Sample $\zeta_{k-1}$ - If $k < n$: add $\mathbf{P}_k$ to the transcript - Add $\pi$ to the transcript. -- Sample $x$ from the transcript. - Add $y$ to the transcript. - For $s=0, \dots, Q-1$: - - Sample random index $\iota_s \in [0, 2^{n+l}]$ from the transcript and let $\upsilon_s := \omega^{\iota_s}$. + - Sample random index $\iota_s \in [0, 2^{n+l-1}]$ from the transcript and let $\upsilon_s := h\omega^{\iota_s}$. #### Verify grinding: Check that $\text{Keccak256}(x || y)$ has $c$ leading zeroes. @@ -230,35 +234,61 @@ Check that $\text{Keccak256}(x || y)$ has $c$ leading zeroes. #### Step 3: Verify FRI -- Check that the following are all _Accept_: - - $\text{Verify}((\upsilon_s, \pi_0^{\upsilon_s}), \mathbf{P}_0, \mathfrak{P}_0)$ for all $0\leq s < Q$. - - $\text{Verify}((-\upsilon_s^{2^k}, \pi_k^{-\upsilon_s^{2^k}}), \mathbf{P}_k, \mathfrak{P}_k)$ for all $0\leq k < n$, $0\leq s < Q$. -- For all $s=0,\dots,Q-1$: - For all $k=0,\dots,n-1$: - Solve the following system of equations on the variables $G, H$ +- Reconstruct the deep composition polynomial values at $\upsilon_s$ and $-\upsilon_s$. That is, define + $$\begin{align}\pi_0^{\upsilon_s}&:= + \gamma\frac{\eta_1^{\upsilon_s} - \eta_1^{z^2}}{\upsilon_s - z^2} + \gamma'\frac{\eta_2^{\upsilon_s} - \eta_2^{z^2}}{\upsilon_s - z^2} + \sum_j \gamma_j\frac{\tau_j^{\upsilon_s} - \tau_j^{z}}{\upsilon_s - z} + \gamma_j'\frac{\tau_j^{\upsilon_s} - \tau_j^{gz}}{\upsilon_s - gz}, \\ + \pi_0^{-\upsilon_s}&:= + \gamma\frac{\eta_1^{-\upsilon_s} - \eta_1^{z^2}}{-\upsilon_s - z^2} + \gamma'\frac{\eta_2^{-\upsilon_s} - \eta_2^{z^2}}{-\upsilon_s - z^2} + \sum_j \gamma_j\frac{\tau_j^{-\upsilon_s} - \tau_j^{z}}{-\upsilon_s - z} + \gamma_j'\frac{\tau_j^{-\upsilon_s} - \tau_j^{gz}}{-\upsilon_s - gz}. + \end{align} + $$ +- For all $s=0,\dots,Q-1$: + - For all $k=0,\dots,n-1$: + - Check that $\text{Verify}((\upsilon_s^{2^k}, \pi_k^{-\upsilon_s^{2^k}}), \mathbf{P}_k, \mathfrak{P}_k)$ and $\text{Verify}((-\upsilon_s^{2^k}, \pi_k^{-\upsilon_s^{2^k}}), \mathbf{P}_k, \mathfrak{P}_k')$ are _Accept_. + - Solve the following system of equations on the variables $G, H$ $$ \begin{aligned} \pi_k^{\upsilon_s^{2^{k}}} &= G + \upsilon_s^{2^k}H \\ \pi_k^{-\upsilon_s^{2^{k}}} &= G - \upsilon_s^{2^k}H \end{aligned} $$ - - Define $\pi_{k+1}^{\upsilon_s^{2^{k+1}}}:=G + \zeta_{k}H$ - - Check that $\pi_n^{\upsilon_s^{2^n}}$ is equal to $\pi$. - -#### Step 4: Verify deep composition polynomial is FRI first layer + - If $k < n - 1$, check that $\pi_{k+1}^{\upsilon_s^{2^{k+1}}}$ equals $G + \zeta_{k}H$ + - If $k = n$, check that $\pi$ equals $G + \zeta_{k}H$. + +#### Step 4: Verify trace and composition polynomials openings - For $s=0,\dots,Q-1$ do the following: - Check that the following are all _Accept_: + - $\text{Verify}((\upsilon_s, \tau_j^{\upsilon_s}), \mathbf{T}_j, \mathfrak{T}_j)$ for all $0\leq j < m$. - $\text{Verify}((\upsilon_s, \eta_1^{\upsilon_s}), \mathbf{H}_1, \mathfrak{h}_1)$. - $\text{Verify}((\upsilon_s, \eta_2^{\upsilon_s}), \mathbf{H}_2, \mathfrak{h}_2)$. - - $\text{Verify}((\upsilon_s, \tau_j^{\upsilon_s}), \mathbf{T}_j, \mathfrak{T}_j)$ for all $0\leq j < m$. - - Check that $\pi_0^{\upsilon_s}$ is equal to the following: - $$ - \gamma\frac{\eta_1^{\upsilon_s} - \eta_1^{z^2}}{\upsilon_s - z^2} + \gamma'\frac{\eta_2^{\upsilon_s} - \eta_2^{z^2}}{\upsilon_s - z^2} + \sum_j \gamma_j\frac{\tau_j^{\upsilon_s} - \tau_j^{z}}{\upsilon_s - z} + \gamma_j'\frac{\tau_j^{\upsilon_s} - \tau_j^{gz}}{\upsilon_s - gz} - $$ + - $\text{Verify}((-\upsilon_s, \tau_j^{\upsilon_s}), \mathbf{T}_j, \mathfrak{T}_j')$ for all $0\leq j < m$. + - $\text{Verify}((-\upsilon_s, \eta_1^{\upsilon_s}), \mathbf{H}_1, \mathfrak{h}_1')$. + - $\text{Verify}((-\upsilon_s, \eta_2^{\upsilon_s}), \mathbf{H}_2, \mathfrak{h}_2')$. + + +## Notes on Optimizations and variants +### Sampling of challenges variant +To build the composition the prover samples challenges $\beta_k^T$ and $\beta_j^B$ for $k = 1,\dots,n_T$ and $j=1,\dots,m$. A variant of this is sampling a single challenge $\beta$ and defining $\beta_k^T$ and $\beta_j^B$ as powers of $\beta$. That is, define $\beta_k^T := \beta^{k-1}$ for $k=1,\dots,n_T$ and $\beta_j^B := \beta^{j + n_T - 1}$ for $j =1, \dots, m$. + +The same variant applies for the challenges $\gamma, \gamma', \gamma_j, \gamma_j'$ for $j = 1, \dots, m$ used to build the deep composition polynomial. In this case the variant samples a single challenge $\alpha$ and defines $\gamma_j := \alpha^j$, $\gamma_j' := \alpha^{j + m - 1}$ for all $j=1,\dots,m$, and $\gamma := \alpha^{2m}, \gamma' := \alpha^{2m+1}$. + +### Batch inversion +Inversions of finite field elements are slow. There is a very well known trick to batch invert many elements at once replacing inversions by multiplications. See [here](https://en.wikipedia.org/wiki/Modular_multiplicative_inverse#Multiple_inverses) for the algorithm. + +### FFT +One of the most computationally intensive operations performed is polynomial division. These can be optimized by utilizing [Fast Fourier Transform](http://web.cecs.pdx.edu/~maier/cs584/Lectures/lect07b-11-MG.pdf) (FFT) to divide each field element in Lagrange form. + +### Ruffini's rule +In specific scenarios, such as dividing by a polynomial of the form $X-a$, for example when building the deep composition polynomial, [Ruffini's rule](https://en.wikipedia.org/wiki/Ruffini%27s_rule) can be employed to further enhance performance. + +### Bit-reversal ordering of Merkle tree leaves +As one can see from inspecting the protocol, there are multiple times where, for a polynomial $p$, the prover sends both openings $\text{Open}(p(D), h\omega^i)$ and $\text{Open}(p(D), -h\omega^i)$. This implies, a priori, sending two authentication paths. Domains can be indexed using bit-reverse ordering to reduce this to a single authentication path for both openings, as follows. + +The natural way of building a Merkle tree to commit to a vector $(p(h), p(h\omega), p(h\omega^2), \dots, p(h\omega^{2^k-1}))$, is assigning the value $p(h\omega^i)$ to leaf $i$. If this is the case, the value $p(h\omega^i)$ is at position $i$ and the value $p(-h\omega^i)$ is at position $i + 2^{k-1}$. This is because $-1$ equals $\omega{2^{k-1}}$ for the value $\omega$ used in the protocol. -# Other +Instead of this naive approach, a better solution is to assign the value $p(h\omega^{\sigma(i)})$ to leaf $i$, where $\sigma$ is the bit-reversal permutation. This is the permutation that maps $i$ to the index whose binary representation (padded to $k$ bits), is the binary representation of $i$ but in reverse order. For example, if $k=3$ and $i=1$, then its binary representation is $001$, which reversed is $100$. Therefore $\sigma(1) = 8$. In the same way $\sigma(0) = 0$ and $\sigma(2) = 4$. Check out the [wikipedia](https://en.wikipedia.org/wiki/Bit-reversal_permutation) article. With this ordering of the leaves, if $i$ is even, element $p(h\omega^{\sigma(i)})$ is at index $i$ and $p(-h\omega^{\sigma(i)})$ is at index $i + 1$. Which means that a single authentication path serves to validate both points simultaneously. -## Notes on Optimizations -- Inversions of finite field elements are slow. There is a very well known trick to batch invert many elements at once replacing inversions by multiplications. See [here](https://en.wikipedia.org/wiki/Modular_multiplicative_inverse#Multiple_inverses) for the algorithm. -- One of the most computationally intensive operations performed is polynomial division. These can be optimized by utilizing [Fast Fourier Transform](http://web.cecs.pdx.edu/~maier/cs584/Lectures/lect07b-11-MG.pdf) (FFT) to divide each field element in Lagrange form. -- In specific scenarios, such as dividing by a polynomial of the form (x-a), [Ruffini's rule](https://en.wikipedia.org/wiki/Ruffini%27s_rule) can be employed to further enhance performance. +### Redundant values in the proof +The prover opens the polynomials $p_k$ of the FRI layers at $\upsilon_s^{2^k}$ and $-\upsilon_s^{2^k}$ for all $k>1$. Later on, the verifier uses each of those pairs to reconstruct one of the values of the next layer, namely $p_{k+1}(\upsilon^{2^{k+1}})$. So there's no need to add the value $p_k(\upsilon^{2^{k+1}})$ to the proof, as the verifier reconstructs them. The prover only needs to send the authentication paths $\mathfrak{P}_k$ for them. +The protocol is only modified at Step 3 of the verifier as follows. Checking that $\text{Verify}((\upsilon_s^{2^k}, \pi_k^{\upsilon_s^{2^k}}), \mathbf{P}_k, \mathfrak{P}_k)$ is skipped. After computing $x := G + \zeta_{k}H$, the verifier uses it to check that $\text{Verify}((\upsilon_s^{2^k}, x), \mathbf{P}_k, \mathfrak{P}_k)$ is _Accept_, which proves that $x$ is actually $\pi_k^{\upsilon_s^{2^k}}$, and continues to the next iteration of the loop. diff --git a/docs/src/starks/protocol_overview.md b/docs/src/starks/protocol_overview.md index 17cad6d26..321329cbd 100644 --- a/docs/src/starks/protocol_overview.md +++ b/docs/src/starks/protocol_overview.md @@ -9,140 +9,155 @@ Then a single polynomial $F$ is constructed that encodes the set of all the poly Then the verifier chooses a random point $z$ and challenges the prover to reveal the values $F(z)$ and $H(z)$. Then the verifier checks that $H(z) = F(z)/G(z)$, which convinces him that the same relation holds at a level of polynomials and, in consequence, convinces the verifier that the private trace $T$ of the prover is valid. In summary, at a very high level, the STARK protocol can be organized into three major parts: + - Arithmetization and commitment of execution trace. -- Construction of composition polynomial $H$. +- Construction and commitment of composition polynomial $H$. - Opening of polynomials at random $z$. # Arithmetization + As the Recap mentions, the trace is a table containing the system's state at every step. In this section, we will denote the trace as $T$. A trace can have several columns to store different aspects or features of a particular state at a specific moment. We will refer to the $j$-th column as $T_j$. You can think of a trace as a matrix $T$ where the entry $T_{ij}$ is the $j$-th element of the $i$-th state. -Most proving systems' primary tool is polynomials over a finite field $\mathbb{F}$. Each column $T_j$ of the trace $T$ will be interpreted as evaluations of such a polynomial $t_j$. Consequently, any information about the states must be encoded somehow as an element in $\mathbb{F}$. +Most proving systems' primary tool is polynomials over a finite field $\mathbb{F}$. Each column $T_j$ of the trace $T$ will be interpreted as evaluations of such a polynomial $t_j$. Consequently, any information about the states must be encoded somehow as an element in $\mathbb{F}$. To ease notation, we will assume here and in the protocol that the constraints encoding transition rules depend only on a state and the previous one. Everything can be easily generalized to transitions that depend on many preceding states. Then, constraints can be expressed as multivariate polynomials in $2m$ variables $$P_k^T(X_1, \dots, X_m, Y_1, \dots, Y_m)$$ A transition from state $i$ to state $i+1$ will be valid if and only if when we plug row $i$ of $T$ in the first $m$ variables and row $i+1$ in the second $m$ variables of $P_k^T$, we get $0$ for all $k$. In mathematical notation, this is $$P_k^T(T_{i, 0}, \dots, T_{i, m}, T_{i+1, 0}, \dots, T_{i+1, m}) = 0 \text{ for all }k$$ -These are called *transition constraints* and check the trace's local properties, where local means relative to specific rows. There is another type of constraint, called *boundary constraint*, and denoted $P_j^B$. These enforce parts of the trace to take particular values. It is helpful, for example, to verify the initial states. +These are called _transition constraints_ and check the trace's local properties, where local means relative to specific rows. There is another type of constraint, called _boundary constraint_, and denoted $P_j^B$. These enforce parts of the trace to take particular values. It is helpful, for example, to verify the initial states. -So far, these constraints can only express the local properties of the trace. There are situations where the global properties of the trace need to be checked for consistency. For example, a column may need to take all values in a range but not in any predefined way. Several methods exist to express these global properties as local by adding redundant columns. Usually, they need to involve randomness from the verifier to make sense, and they turn into an interactive protocol called *Randomized AIR with Preprocessing*. +So far, these constraints can only express the local properties of the trace. There are situations where the global properties of the trace need to be checked for consistency. For example, a column may need to take all values in a range but not in any predefined way. Several methods exist to express these global properties as local by adding redundant columns. Usually, they need to involve randomness from the verifier to make sense, and they turn into an interactive protocol called _Randomized AIR with Preprocessing_. # Polynomial commitment scheme + To make interactions possible, a crucial cryptographic primitive is the Polynomial Commitment Scheme. This prevents the prover from changing the polynomials to adjust them to what the verifier expects. Such a scheme consists of the commit and the open protocols. STARK uses a univariate polynomial commitment scheme that internally combines a vector commitment scheme and a protocol called FRI. Let's begin with these two components and see how they build up the polynomial commitment scheme. ## Vector commitments -Given a vector $Y = (y_0, \dots, y_M)$, commiting to $Y$ means the following. The prover builds a Merkle tree out of it and sends its root to the verifier. The verifier can then ask the prover to reveal, or *open*, the value of the vector $Y$ at some index $i$. The prover won't have any choice except to send the correct value. The verifier will expect the corresponding value $y_i$ and the authentication path to the tree's root to check its authenticity. The authentication path also encodes the vector's position $i$ and its length $M$. -In STARKs, all commited vectors are of the form $Y = (p(d_1), \dots, p(d_M))$ for some polynomial $p$ and some domain fixed domain $D = (d_1, \dots, d_M)$. The domain is always known to the prover and the verifier. +Given a vector $Y = (y_0, \dots, y_M)$, commiting to $Y$ means the following. The prover builds a Merkle tree out of it and sends its root to the verifier. The verifier can then ask the prover to reveal, or _open_, the value of the vector $Y$ at some index $i$. The prover won't have any choice except to send the correct value. The verifier will expect the corresponding value $y_i$ and the authentication path to the tree's root to check its authenticity. The authentication path also encodes the vector's position $i$ and its length $M$. + +The root of the Merkle tree is said to be the **commitment** of $Y$, and we denote it here by $[Y]$. ## FRI -The FRI protocol is a potent tool to prove that the commitment of a vector $(p(d_1), \dots, p(d_M))$ corresponds to the evaluations of a polynomial $p$ of a certain degree. The degree is a configuration of the protocol. + +In STARKs, all commited vectors are of the form $Y = (p(d_1), \dots, p(d_M))$ for some polynomial $p$ and some fixed domain $D = (d_1, \dots, d_M)$. The domain is always known to the prover and the verifier. It can be proved, as long as $M$ is less than the total number of field elements, that every vector $(y_0, \dots, y_M)$ is equal to $(p(d_1), \dots, p(d_M))$ for a unique polynomial $p$ of degree at most $M-1$. This is called the Lagrange interpolation theorem. It means, there is a unique polynomial of degree at most $M-1$ such that $p(d_i) = y_i$ for all $i$. And $M-1$ is an upper bound to the degree of $p$. It could be less. For example, the vector of all ones $Y = (1,1,\dots,1)$ is the evaluation of the constant polynomial $p = 1$, which has degree $0$. + +Suppose the vector $Y=(y_1, \dots, y_M)$ is the vector of evaluations of a polynomial $p$ of degree strictly less than $M-1$. Suppose one party holds the vector $Y$ and another party holds only the commitment $[Y]$ of it. The FRI protocol is an efficient interactive protocol with which the former can convince the latter that the commitment they hold corresponds to the vector of evaluations of a polynomial $p$ of degree strictly less than $M$. + +More precisely, the protocol depends on the following parameters + +- Powers of two $N = 2^n$ and $M = 2^m$ with $n < m$. +- A vector $D=(d_1,\dots,d_M)$, with $d_i = h\omega^i$, with $h$ a nonzero value in $\mathbb{F}$ and $\omega$ a primitive $M$-root of unity + +A prover holds a vector $Y=(y_1,\dots,y_M)$, and the verifier holds the commitment $[Y]$ of it. The result of the FRI protocol will be _Accept_ if the unique polynomial $p$ of degree less than $M-1$ such that $Y=(p(d_1),\dots,p(d_M))$ has degree less than $N-1$. Even more precisely, the protocol proves that $Y$ is very close to a vector $(p(d_1),\dots,p(d_M))$ with $p$ of degree less than $N-1$, but it may differ in negligible proportion of the coordinates. + +The number $b = M/N = 2^{m-n}$ is called the **blowup factor** and the security of the protocol depends in part on this parameter. The specific shape of the domain set $D$ has some symmetric properties important for the inner workings of FRI, such as $-d_i \in D$ for all $i$. +### Variant useful for STARKs + +FRI is usually described as above. In STARK, FRI is used as a building block for the polynomial commitment scheme of the next section. For that, a small variant of FRI is needed. + +Suppose the prover holds a vector $Y = (y_1, \dots, y_M)$ and the verifier holds its commitment $[Y]$ as before. Suppose further that both parties know a function $F$ that takes two field elements and outputs another field element. For example $F$ could be the function $F(a,b) = a + b^{-1}$. More precisely, the kind of functions we need are $F: \mathbb{F} \times D \to \mathbb{F}$. + +The protocol can be used to prove that the transformed vector $(F(y_1, d_1), \dots, F(y_M, d_M))$ is the vector of evaluations of a polynomial $q$ of degree at most $N-1$. Note that in this variant, the verifier holds originally the commitment of the vector $Y$ and not the commitment of the transformed vector. In the example, the verifier holds the commitment $[Y]$ and FRI will return _Accept_ if $(y_1 + d_1^{-1}, \dots, y_M + d_M^{-1})$ is the vector of evaluations of a polynomial of degree at most $N-1$. ## Polynomial commitments + STARK uses a univariate polynomial commitment scheme. The following is what is expected from the **commit** and **open** protocols: -- *Commit*: given a polynomial $p$, the prover produces a sort of hash of it. We denote it here by $[p]$, called the *commitment* of $p$. This hash is unique to $p$. The prover usually sends $[p]$ to the verifier. -- *Open*: this is an interactive protocol between the prover and the verifier. The prover holds the polynomial $p$. The verifier only has the commitment $[p]$. The verifier sends a value $z$ to the prover at which he wants to know the value $y=p(z)$. The prover sends a value $y$ to the verifier, and then they engage in the *Open* protocol. As a result, the verifier gets convinced that the polynomial corresponding to the hash $[p]$ evaluates to $y$ at $z$. -Let's see how both of these protocols work in detail. Some configuration parameters are needed: +- _Commit_: given a polynomial $p$, the prover produces a sort of hash of it. We denote it here by $[p]$, called the _commitment_ of $p$. This hash is unique to $p$. The prover usually sends $[p]$ to the verifier. +- _Open_: this is an interactive protocol between the prover and the verifier. The prover holds the polynomial $p$. The verifier only has the commitment $[p]$. The verifier sends a value $z$ to the prover at which he wants to know the value $y=p(z)$. The prover sends a value $y$ to the verifier, and then they engage in the _Open_ protocol. As a result, the verifier gets convinced that the polynomial corresponding to the hash $[p]$ evaluates to $y$ at $z$. + +Let's see how both of these protocols work in detail. The same configuration parameters of FRI are needed: + - Powers of two $N = 2^n$ and $M = 2^m$ with $n < m$. -- A vector $D=(d_1,\dots,d_M)$, with $d_i$ in $\mathbb{F}$ for all $i$ and $d_i\neq d_j$ for all $i\neq j$. -- An integer $k > 0$. +- A vector $D=(d_1,\dots,d_M)$, with $d_i = h\omega^i$, with $h$ a nonzero value in $\mathbb{F}$ and $\omega$ a primitive $M$-root of unity -The commitment scheme will only work for polynomials of degree at most $N$. This means anyone can commit to any polynomial, but the Open protocol will pass only for polynomials satisfying that degree bound. +The commitment scheme will only work for polynomials of degree at most $N$ (polynomials of degree $N$ are allowed). This means: anyone can commit to any polynomial, but the Open protocol will pass only for polynomials satisfying that degree bound. ### Commit + Given a polynomial $p$, the commitment $[p]$ is just the commitment of the vector $(p(d_1), \dots, p(d_M))$. That is, $[p]$ is the root of the Merkle tree of the vector of evaluations of $p$ at $D$. ### Open + It is an interactive protocol. So assume there is a prover and a verifier. We describe the process considering an honest prover. In the next section, we analyze what happens for malicious provers. -The prover holds the polynomial $p$, and the verifier only the commitment $[p]$ of it. There is also an element $z$. The prover evaluates $p(z)$ and sends the result to the verifier. As we mentioned, the goal is to generate proof of the validity of the evaluation. Let us denote $y := p(z)$. This is a value now both the prover and verifier have. +The prover holds the polynomial $p$, and the verifier only the commitment $[p]$ of it. There is also an element $z$ chosen by the verifier. The prover evaluates $p(z)$ and sends the result back. As we mentioned, the goal is to generate proof of the validity of the evaluation. Let us denote $y$ the value received by the verifier. -The protocol has three steps: +Now they engage in the variant of the FRI protocol for the function $F(a,b) = (a - y) / (b - z)$. The verifier accepts the value $y$ if and only if the result of FRI is _Accept_. -- **FRI on $p$**: First, the prover and the verifier engage in a FRI protocol for polynomials of degree at most $N$ to check that $[p]$ is the commitment of such a polynomial. We will assume from now on that the degree of $p$ is at most $N$. There is an optimization that completely removes this step—more on that [later](#optimize-the-open-protocol-reusing-fri-internal-challenges). +Let's see why this makes sense. -- **FRI on $(p-y)/(x-z)$**: Since $p(z) = y$, the polynomial $p$ can be written as $p = y + (x - z) q$ for some polynomial $q$. The prover computes the commitment $[q]$ and sends it to the verifier. Now they engage in a FRI protocol for polynomials of degree at most $N-1$, which convinces the verifier that $[q]$ is the commitment of a polynomial of degree at most $N-1$. +### Completeness -- **Point checks**: From the point of view of the verifier the commitments $[p]$ and $[q]$ are still potentially unrelated. Therefore, there is a check to ensure that $q$ was computed properly from $p$ following the formula $q = (p-y)/(x-z)$ and that $[q]$ is its commitment. To do this, the verifier challenges the prover to open $[p]$ and $[q]$ as vectors. They use the open protocol of the vector commitment scheme to reveal the values $p(d_i)$ and $q(d_i)$ for some random point $d_i \in D$ chosen by the verifier. Next he checks that $p(d_i) = y + (d_i - z) q(d_i) $. They repeat this last part $k$ times and, as we'll analyze in the next section, this whole thing will convince the verifier that $p = y + (x -z) q$ as polynomials with overwhelming probability (about $(N/M)^k$). Finally, the verifier deduces that $p(z) = y$ from this equality. +If the prover is honest, $p$ is of degree at most $N$ and $y$ equals $p(z)$. That means that +$$p - y = (X - z) q$$ +for some polynomial $q$. Since $p$ is of degree at most $N$, then $q$ is of degree at most $N-1$. The vector $(q(d_1), \dots, q(d_M))$ is then a vector of evaluations of a polynomial of degree at most $N-1$. And it is equal to $(F(p(d_1), d_1), \dots, F(p(d_M), d_M))$. So the FRI protocol will succeed. ### Soundness -Let's analyze why the open protocol is sound. Keep in mind that the prover always has to provide a commitment $[q]$ of a polynomial of degree at most $N-1$ that satisfies $p(d_i) = y + (d_i - z) q(d_i)$ for the chosen values of the verifier. That's the goal. An essential but subtle part of the soundness is that $D$ is a vector of length $M>N$. To understand why let's see what would happen if $N = M$. -#### Case $M=N$ -Suppose the prover tries to cheat the verifier by sending a value $y$ different from $p(z)$. This means that $p - y$ is not divisible by $X-z$ as polynomials. But as long as $z$ is not in $D$ the prover can interpolate the values $$\frac{p(d_1) - y}{d_1 - z}, \dots, \frac{p(d_N) - y}{d_N - z},$$ -at $D$ and obtain some polynomial $q$ of degree at most $N-1$. This polynomial satisfies $p(d_i) = y + (d_i - z) q(d_i)$ for all $d_i \in D$, but the equality of polynomials $p = y + (x - z) q$ does not hold. Mainly because that would imply that $p(z) = y$, and we are assuming the opposite case. Nevertheless, if they continue with the open protocol, FRI will pass since $q$ is a polynomial of degree at most $N-1$. All subsequent point checks will pass since $q$ was crafted especially for that. +Let's sketch an idea of the soundness. Note that the value $z$ is chosen by the verifier after receiving the commitment $[p]$ of $p$. So the prover does not know in advance, at the moment of sending $[p]$, what $z$ will be. -So how come $p$ is different from $y + (x - z) q$ but has the property that $p(d_i) = y + (d_i - z) q(d_i)$ for all $d_i \in D$? FRI guarantees $q$ is of degree at most $N-1$, which implies that $y + (x - z) q$ is of degree at most $N$. Then, since $p$ is of degree at most $N$, the difference $$f := y + (X-z) q - p$$ is again of degree at most $N$. A polynomial $f$ of degree at most $N$ is zero if and only if we can prove that $f(d) = 0$ for $N+1$ distinct points. But the construction of $q$ only shows that $f(d_i) = 0$ for all $d_i \in D$, a set of $N$ points. It's not enough. One extra point and the equality $f=0$ would hold. Of course, that point doesn't exist, again, because that would imply that $p(z) = y$, which we assume is not true. - -#### Case $M=2N$ -Let's see one more example where we double the de size of $D$ but keep the same bound for the degree of the polynomials. So polynomials are of degree at most $N$, and the length of $D$ is $M = 2N$. - -Again let's assume the prover wants to cheat the verifier similarly and claims that $y$ is $p(z)$ but, in reality, $y \neq p(z)$. Inevitably, for the Open protocol to succeed, the prover has to send a commitment $[q]$ of some polynomial of degree at most $N-1$. The strategy of the prover is pretty much constrained to be the same. But now he can't interpolate $q$ as before in every element of $D$ because that would produce a polynomial of degree at most $2N-1$. This most likely won't pass the FRI check. - -Alternatively, he can choose some subset of $D' \subset D$ of size $N$ and interpolate only there to get a polynomial $q$ of degree at most $N-1$. That will succeed in the FRI phase, but what happens with the point checks? The check will pass if the verifier chooses a challenge $d_i$ that belongs to $D'$. If $d_i \notin D'$, then the check will inevitably fail. This is because, as we saw in the previous case, one extra successful point to the already $N$ points where $q$ was interpolated was enough to prove that $p(z) = y$, which we assume now not to hold. This means, if $d_i \notin D'$, then $p(d_i)$ does not coincide with $y + (d_i - z) q(d_i)$ and the point check fails. So if the verifier chooses the challenges following a uniform distribution, the chances of the prover being caught cheating are $1/2$ for only one challenge. If the verifier performs $k$ challenges, then the chance of the prover not getting caught is $1/2^{k}$. - -#### General case: the blowup factor -If the ratio between $M$ and $N$ is $2$, then $k$ challenges give $1/2^{k}$ of probability for a malicious prover to succeed. If the ratio is $4$, that is, if $M = 4N$, then that probability is $1/4^k$ for the same number of point checks. This provides another way of improving the soundness error. The larger the ratio, the more complex it is to cheat in the open protocol. This ratio is what's called *the blowup factor*. It is a security parameter, and finding a balance between the number of challenges $k$ and the size of $D$ is part of the configuration of the protocol. Increasing the size of $D$ makes committing an expensive operation since it involves building a Merkle tree with a vector of the size of $D$. But increasing the number of challenges makes the size of the final proof larger. - -We denote the blowup factor by $b$, and it's always assumed to be a power of $2$. +Suppose the prover is trying to cheat and sends the commitment $[Y]$ of a vector $Y=(y_1,\dots,y_M)$ that's not the vector of evaluations of a polynomial of degree at most $N$. Then the coordinates of the transformed vector are $(y_i - y) / (d_i - z)$. Since $z$ was chosen by the verifier, dividing by $d_i - z$ shuffles all the elements in a very unpredictable way for the prover. So it is extremely unlikely that the cheating prover can craft an invalid vector $Y$ such that the transformed vector turns out to be of degree at most $N-1$. The expected degree of the polynomial associated with a random vector is $M-1$. ### Batch -During proof generation, polynomials are committed and opened several times. Computing these for each polynomial independently is costly. In this section, we'll see how batching polynomials can reduce the amount of computation. Let $P=\{p_1, \dots, p_L\}$ be a set of polynomials. We will commit and open $P$ as a whole. We note this batch commitment as $[P]$. - -We need the same configuration parameters as before: $N=2^n$, $M=2^m$ with $N0$. -As described earlier, to commit to a single polynomial $p$, a Merkle tree is built over the vector $(p(d_1), \dots, p(d_m))$. When committing to a batch of polynomials $P=\{p_1, \dots, p_n\}$, the leaves of the Merkle tree are instead the concatenation of the polynomial evaluations. That is, in the batch setting, the Merkle tree is built for the vector $(p_1(d_1)||\dots||p_L(d_1), \dots, p_L(d_m)||\dots||p_n(d_m))$. The commitment $[P]$ is the root of this Merkle tree. This reduces the proof size: we only need one Merkle tree for $L$ polynomials. The verifier can then only ask for values in batches. When the verifier chooses an index $i$, the prover sends $p_1 (d_i) , \dots , p_L (d_i)$ along with one authentication path. The verifier on his side computes the concatenation $p_1(d_i)||\dots||p_L(d_i)$ and validates it with the authentication path and $[P]$. This also reduces the computational time. By traversing the Merkle tree one time, it can reveal several components simultaneously. +During proof generation, polynomials are committed and opened several times. Computing these for each polynomial independently is costly. In this section, we'll see how batching polynomials can reduce the amount of computation. Let $P=\{p_1, \dots, p_L\}$ be a set of polynomials. We will commit and open $P$ as a whole. We note this batch commitment as $[P]$. -The batch open protocol proceeds similarly to the case of a single polynomial. The prover will try to convince the verifier that the committed polynomials $P$ at $z$ evaluate to some values $y_i = p_i(z)$. In the batch case, the prover will construct the following polynomial: +We need the same configuration parameters as before: $N=2^n$, $M=2^m$ with $N