diff --git a/docs/basics/cmdline.md b/docs/basics/cmdline.md index b278317b2..79da0a2b5 100644 --- a/docs/basics/cmdline.md +++ b/docs/basics/cmdline.md @@ -374,7 +374,7 @@ Timeout in seconds for client queries, i.e. maximum time a client call will exec `-u 1` disables - system commands from a remote (signals `'access`), including exit via `"\\"` -- access to files outside the current directory for any handle context ([`.z.w`](../ref/dotz.md#zw-handle)) other than 0 +- access to files outside the current directory for any handle context ([`.z.w`](../ref/dotz.md#zw-handle)) other than 0. Segmented database partitions using directories outside the current working directory can be enabled using the method described [here](../database/segment.md#using-symlinks). - hopen on a fifo (since 4.1t 2021.10.13, 4.0 2023.08.11) - hopen of a file (since 4.1t 2021.10.13, 4.0 2023.08.11) - the [`exit`](../ref/exit.md) keyword (since 4.1t 2021.07.12) diff --git a/docs/basics/funsql.md b/docs/basics/funsql.md index ab2072a2e..ff850c11c 100644 --- a/docs/basics/funsql.md +++ b/docs/basics/funsql.md @@ -2,6 +2,7 @@ title: Functional qSQL | Basics | kdb+ and q documentation description: The functional forms of delete, exec, select and update are particularly useful for programmatically-generated queries, such as when column names are dynamically produced. keywords: delete, exec, functional, kdb+, q, select, sql, update +author: [Peter Storeng, Stephen Taylor, Simon Shanks] --- # Functional qSQL @@ -20,6 +21,8 @@ q)?[t;();0b;(enlist `n)!enlist `n] The q interpreter parses `delete`, `exec`, `select`, and `update` into their equivalent functional forms, so there is no performance difference. + + The functional forms are ```syntax @@ -34,46 +37,29 @@ The functional forms are where: -`t` -: is a table, or the name of a table as a symbol atom. - -`c` -: is the [Where phrase](qsql.md#where-phrase), a list of constraints. -: Every constraint in `c` is a [parse tree](parsetrees.md) representing an expression to be evaluated; the result of each being a boolean vector. The parse tree consists of a function followed by a list of its arguments, each an expression containing column names and other variables. (Represented by symbols; distinguish actual symbol constants by enlisting them.) The function is applied to the arguments, producing a boolean vector that selects the rows. The selection is performed in the order of the items in `c`, from left to right: only rows selected by one constraint are evaluated by the next. +* `t` is a table, or the name of a table as a symbol atom. +* `c` is the [Where phrase](qsql.md#where-phrase), a list of constraints. +Every constraint in `c` is a [parse tree](parsetrees.md) representing an expression to be evaluated; the result of each being a boolean vector. The parse tree consists of a function followed by a list of its arguments, each an expression containing column names and other variables. Represented by symbols, it distinguishes actual symbol constants by enlisting them. The function is applied to the arguments, producing a boolean vector that selects the rows. The selection is performed in the order of the items in `c`, from left to right: only rows selected by one constraint are evaluated by the next. - -`b` -: is the [By phrase](qsql.md#by-phrase), one of: - +* `b` is the [By phrase](qsql.md#by-phrase). +The domain of dictionary `b` is a list of symbols that are the key names for the grouping. Its range is a list of column expressions (parse trees) whose results are used to construct the groups. The grouping is ordered by the domain items, from major to minor. +`b` is one of: - the general empty list `()` - boolean atom: `0b` for no grouping; `1b` for distinct - a symbol atom or list naming table column/s - - a dictionary of group-by specifications - -: The domain of dictionary `b` is a list of symbols that are the key names for the grouping. Its range is a list of column expressions (parse trees) whose results are used to construct the groups. The grouping is ordered by the domain items, from major to minor. - -`a` -: is the [Select phrase](../ref/select.md#select-phrase), one of: - + - a dictionary of group-by specifications +* `a` is the [Select phrase](../ref/select.md#select-phrase). +The domain of dictionary `a` is a list of symbols containing the names of the produced columns. [QSQL query templates](qsql.md) assign default column names in the result, but here each result column must be named explicitly. +Each item of its range is an evaluation list consisting of a function and its argument(s), each of which is a column name or another such result list. For each evaluation list, the function is applied to the specified value(s) for each row and the result is returned. The evaluation lists are resolved recursively when operations are nested. +`a` is one of - the general empty list `()` - a symbol atom: the name of a table column - a parse tree - a dictionary of select specifications (aggregations) - -: The domain of dictionary `a` is a list of symbols containing the names of the produced columns. ([QSQL query templates](qsql.md) assign default column names in the result, but here each result column must be named explicitly.) -: Each item of its range is an evaluation list consisting of a function and its argument/s, each of which is a column name or another such result list. For each evaluation list, the function is applied to the specified value/s for each row and the result is returned. The evaluation lists are resolved recursively when operations are nested. - -`i` -: is a list of indexes - -`p` -: is a [parse tree](parsetrees.md) - -`n` -: is a non-negative integer or infinity, indicating the maximum number of records to be returned - -`g` -: is a unary grade function +* `i` is a list of indexes +* `p` is a [parse tree](parsetrees.md) +* `n` is a non-negative integer or infinity, indicating the maximum number of records to be returned +* `g` is a unary grade function ## Call by name @@ -607,6 +593,392 @@ _Q for Mortals_ [§9.12.4 Functional delete](https://code.kx.com/q4m3/9_Queries_q-sql/#9124-functional-delete) +## Conversion using parse + +Applying [parse](parsetrees.md) to a qSQL statement written as a string will return the internal representation of the functional form. +With some manipulation this can then be used to piece together the functional form in q. +This generally becomes more difficult as the query becomes more complex and requires a deep understanding of what kdb+ is doing when it parses qSQL form. + +An example of using parse to convert qSQL to its corresponding functional form is as follows: +```q +q)t:([]c1:`a`b`c; c2:10 20 30) +q)parse "select c2:2*c2 from t where c1=`c" +? +`t +,,(=;`c1;,`c) +0b +(,`c2)!,(*;2;`c2) + +q)?[`t; enlist (=;`c1;enlist `c); 0b; (enlist `c2)!enlist (*;2;`c2)] +c2 +-- +60 +``` + +### Issues converting to functional form + +To convert a `select` query to a functional form one may attempt to +apply the `parse` function to the query string: + +```q +q)parse "select sym,price,size from trade where price>50" +? +`trade +,,(>;`price;50) +0b +`sym`price`size!`sym`price`size +``` + +As we know, `parse` produces a parse tree and since some of the elements may themselves be parse trees we can’t immediately take the output of parse and plug it into the form `?[t;c;b;a]`. After a little playing around with the result of `parse` you might eventually figure out that the correct functional form is as follows. + +```q +q)funcQry:?[`trade;enlist(>;`price;50);0b;`sym`price`size! `sym`price`size] + +q)strQry:select sym,price,size from trade where price>50 q) +q)funcQry~strQry +1b +``` + +This, however, becomes more difficult as the query statements become more complex: + +```q +q)parse "select count i from trade where 140>(count;i) fby sym" +? +`trade +,,(>;140;(k){@[(#y)#x[0]0#x +1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) +0b +(,`x)!,(#:;`i) +``` + +In this case, it is not obvious what the functional form of the above query should be, even after applying `parse`. + +There are three issues with this parse-and-“by eye” method to convert to the equivalent functional form. We will cover these in the next three subsections. + + +#### Parse trees and eval + +The first issue with passing a `select` query to `parse` is that each returned item is in unevaluated form. As [discussed here](parsetrees.md#eval-and-value), simply applying `value` to a parse tree does not work. However, if we evaluate each one of the arguments fully, then there would be no nested parse trees. We could then apply `value` to the result: + +```q +q)eval each parse "select count i from trade where 140>(count;i) fby sym" +? ++`sym`time`price`size!(`VOD`IBM`BP`VOD`IBM`IBM`HSBC`VOD`MS.. +,(>;140;(k){@[(#y)#x[0]0#x +1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) +0b +(,`x)!,(#:;`i) +``` + +The equivalence below holds for a general qSQL query provided as a string: + +```q +q)value[str]~value eval each parse str +1b +``` + +In particular: + +```q +q)str:"select count i from trade where 140>(count;i) fby sym" + +q)value[str]~value eval each parse str +1b +``` + +In fact, since within the functional form we can refer to the table by name we can make this even clearer. Also, the first item in the result of `parse` applied to a `select` query will always be `?` (or `!` for a `delete`or `update` query) which cannot be evaluated any further. So we don’t need to apply `eval` to it. + +```q +q)pTree:parse str:"select count i from trade where 140>(count;i) fby sym" +q)@[pTree;2 3 4;eval] +? +`trade +,(>;140;(k){@[(#y)#x[0]0#x +1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) +0b +(,`x)!,(#:;`i) +q)value[str] ~ value @[pTree;2 3 4;eval] +1b +``` + + +#### Variable representation in parse trees + +Recall that in a parse tree a variable is represented by a symbol containing its name. So to represent a symbol or a list of symbols, you must use `enlist` on that expression. In k, `enlist` is the unary form of the comma operator in k: + +```q +q)parse"3#`a`b`c`d`e`f" +# +3 +,`a`b`c`d`e`f +q)(#;3;enlist `a`b`c`d`e`f)~parse"3#`a`b`c`d`e`f" +1b +``` + +This causes a difficulty. As [discussed above](#variadic-operators), q has no unary syntax for operators. + +Which means the following isn’t a valid q expression and so returns an error. + +```q +q)(#;3;,`a`b`c`d`e`f) +', +``` + +In the parse tree we receive we need to somehow distinguish between k’s unary `,` (which we want to replace with `enlist`) and the binary Join operator, which we want to leave as it is. + + +#### Explicit definitions in `.q` are shown in full + +The `fby` in the `select` query above is represented by its full k +definition. + +```q +q)parse "fby" +k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} +``` + +While using the k form isn’t generally a problem from a functionality perspective, it does however make the resulting functional statement difficult to read. + + +### The solution + +We will write a function to automate the process of converting a `select` query into its equivalent functional form. + +This function, `buildQuery`, will return the functional form as a string. + +```q +q)buildQuery "select count i from trade where 140>(count;i) fby sym" +"?[trade;enlist(>;140;(fby;(enlist;count;`i);`sym));0b; + (enlist`x)! enlist (count;`i)]" +``` + +When executed it will always return the same result as the `select` query from which it is derived: + +```q +q)str:"select count i from trade where 140>(count;i) fby sym" +q)value[str]~value buildQuery str +1b +``` + +And since the same logic applies to `exec`, `update` and `delete` it will be able to convert to their corresponding functional forms also. + +To write this function we will solve the three issues outlined above: + +1. parse-tree items may be parse trees +2. parse trees use k’s unary syntax for operators +3. q keywords from `.q.` are replaced by their k definitions + +The first issue, where some items returned by `parse` may themselves be parse trees is easily resolved by applying `eval` to the individual items. + +The second issue is with k’s unary syntax for `,`. We want to replace it with the q keyword `enlist`. To do this we define a function that traverses the parse tree and detects if any element is an enlisted list of symbols or an enlisted single symbol. If it finds one we replace it with a string representation of `enlist` instead of `,`. + +```q +ereptest:{ //returns a boolean + (1=count x) and ((0=type x) and 11=type first x) or 11=type x} +ereplace:{"enlist",.Q.s1 first x} +funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} +``` + +Before we replace the item we first need to check it has the +correct form. We need to test if it is one of: + +- An enlisted list of syms. It will have type `0h`, count 1 and the type of its first item will be `11h` if and only if it is an enlisted list of syms. +- An enlisted single sym. It will have type `11h` and count 1 if and only if it is an enlisted single symbol. + +The `ereptest` function above performs this check, with `ereplace` performing the replacement. + +!!! tip "Console size" + + `.Q.s1` is dependent on the size of the console so make it larger if necessary. + +Since we are going to be checking a parse tree which may contain parse trees nested to arbitrary depth, we need a way to check all the elements down to the base level. + +We observe that a parse tree is a general list, and therefore of type `0h`. This knowledge combined with the use of `.z.s` allows us to scan a parse tree recursively. The logic goes: if what you have passed into `funcEn` is a parse tree then reapply the function to each element. + +To illustrate we examine the following `select` query. + +```q +q)show pTree:parse "select from trade where sym like \"F*\",not sym=`FD" +? +`trade +,((like;`sym;"F*");(~:;(=;`sym;,`FD))) 0b +() + +q)x:eval pTree 2 //apply eval to Where clause +``` + +Consider the Where clause in isolation. + +```q +q)x //a 2-list of Where clauses +(like;`sym;"F*") +(~:;(=;`sym;,`FD)) + +q)funcEn x +(like;`sym;"F*") +(~:;(=;`sym;"enlist`FD")) +``` + +Similarly we create a function which will replace k functions with +their q equivalents in string form, thus addressing the third issue above. + +```q +q)kreplace:{[x] $[`=qval:.q?x;x;string qval]} +q)funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} +``` + +Running these functions against our Where clause, we see the k +representations being converted to q. + +```q +q)x +(like;`sym;"F*") +(~:;(=;`sym;,`FD)) + +q)funcK x //replaces ~: with “not” +(like;`sym;"F*") +("not";(=;`sym;,`FD)) +``` + +Next, we make a slight change to `kreplace` and `ereplace` and combine them. + +```q +kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} +ereplace:{"~~enlist",(.Q.s1 first x),"~~"} +q)funcEn funcK x +(like;`sym;"F*") ("~~not~~";(=;`sym;"~~enlist`FD~~")) +``` + +The double tilde here is going to act as a tag to allow us to differentiate from actual string elements in the parse tree. This allows us to drop the embedded quotation marks at a later stage inside the `buildQuery` function: + +```q +q)ssr/[;("\"~~";"~~\"");("";"")] .Q.s1 funcEn funcK x +"((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)))" +``` + +thus giving us the correct format for the Where clause in a functional select. By applying the same logic to the rest of the parse tree we can write the `buildQuery` function. + +```q +q)buildQuery "select from trade where sym like \"F*\",not sym=`FD" +"?[trade;((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)));0b;()]" +``` + +One thing to take note of is that since we use reverse lookup on the `.q` namespace and only want one result we occasionally get the wrong keyword back. + +```q +q)buildQuery "update tstamp:ltime tstamp from z" +"![z;();0b;(enlist`tstamp)!enlist (reciprocal;`tstamp)]" + +q).q`ltime +%: +q).q`reciprocal +%: +``` + +These instances are rare and a developer should be able to spot when they occur. Of course, the functional form will still work as expected but could confuse readers of the code. + + +#### Fifth and sixth arguments + +Functional select also has ranks 5 and 6; i.e. fifth and sixth arguments. + +:fontawesome-regular-hand-point-right: +_Q for Mortals_: [§9.12.1 Functional queries](/q4m3/9_Queries_q-sql/#9121-functional-select) + +We also cover these with the `buildQuery` function. + +```q +q)buildQuery "select[10 20] from trade" +"?[trade;();0b;();10 20]" +q)//5th parameter included +``` + +The 6th argument is a column and a direction to order the results by. Use `<` for ascending and `>` for descending. + +```q +q)parse"select[10;:) +`hopen`hclose + +q)qfind each ("<:";">:") //qfind defined above +hopen +hclose +``` + +We see that the k function for the 6th argument of the functional form is `<:` (ascending) or `>:` (descending). At first glance this appears to be `hopen` or `hclose`. In fact in earlier versions of q, `iasc` and `hopen` were equivalent (as were `idesc` and `hclose`). The definitions of `iasc` and `idesc` were later altered to signal a rank error if not applied to a list. + +```q +q)iasc +k){$[0h>@x;'`rank;@x;'`rank;>x]} + +q)iasc 7 +'rank +``` + +Since the columns of a table are lists, it is irrelevant whether the functional form uses the old or new version of `iasc` or `idesc`. + +The `buildQuery` function handles the 6th argument as a special case so will produce `iasc` or `idesc` as appropriate. + +```q +q)buildQuery "select[10 20;>price] from trade" +"?[trade;();0b;();10 20;(idesc;`price)]" +``` + +The full `buildQuery` function code is as follows: + +```q +\c 30 200 +tidy:{ssr/[;("\"~~";"~~\"");("";"")] $[","=first x;1_x;x]} +strBrk:{y,(";" sv x),z} + +//replace k representation with equivalent q keyword +kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} +funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} + +//replace eg ,`FD`ABC`DEF with "enlist`FD`ABC`DEF" +ereplace:{"~~enlist",(.Q.s1 first x),"~~"} +ereptest:{(1=count x) and ((0=type x) and 11=type first x) or 11=type x} +funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} + +basic:{tidy .Q.s1 funcK funcEn x} + +addbraks:{"(",x,")"} + +//Where clause needs to be a list of Where clauses, +//so if only one Where clause, need to enlist. +stringify:{$[(0=type x) and 1=count x;"enlist ";""],basic x} + +//if a dictionary, apply to both keys and values +ab:{ + $[(0=count x) or -1=type x; .Q.s1 x; + 99=type x; (addbraks stringify key x ),"!",stringify value x; + stringify x] } + +inner:{[x] + idxs:2 3 4 5 6 inter ainds:til count x; + x:@[x;idxs;'[ab;eval]]; + if[6 in idxs;x[6]:ssr/[;("hopen";"hclose");("iasc";"idesc")] x[6]]; + //for select statements within select statements + x[1]:$[-11=type x 1;x 1;[idxs,:1;.z.s x 1]]; + x:@[x;ainds except idxs;string]; + x[0],strBrk[1_x;"[";"]"] } + +buildQuery:{inner parse x} +``` + + + ---- :fontawesome-solid-book-open: [qSQL](qsql.md) diff --git a/docs/basics/internal.md b/docs/basics/internal.md index 723ff9d38..9402ca5d3 100644 --- a/docs/basics/internal.md +++ b/docs/basics/internal.md @@ -466,6 +466,13 @@ where `x` is a list of socket handles, returns a table with columns Since v4.0 2020.06.01. ```q +q)h:hopen 5000 +q)-38!h +p| "q" +f| "t" +z| 0b +n| 0 +m| 0 q){([]h)!-38!h:.z.H}[] h| p f z n m -| --------- @@ -473,8 +480,9 @@ h| p f z n m 9| q t 0 0 0 ``` -:fontawesome-solid-book: -[`.z.H` active sockets](../ref/dotz.md#zh-active-sockets) +:fontawesome-solid-hand-point-right: +[`.z.H` active sockets](../ref/dotz.md#zh-active-sockets), [`.z.W` handles](../ref/dotz.md#zw-handles), [`.z.w` handle](../ref/dotz.md#zw-handle) + ## `-120!x` (memory domain) ```syntax diff --git a/docs/basics/ipc.md b/docs/basics/ipc.md index 603a4db08..2a9d80986 100644 --- a/docs/basics/ipc.md +++ b/docs/basics/ipc.md @@ -54,6 +54,8 @@ Sync messages can also be sent without a pre-existing connection using [one-shot The maximum number of connections is defined by the system limit for protocol (operating system configurable). Prior to 4.1t 2023.09.15, the limit was hardcoded to 1022. After the limit is reached, you see the error `'conn` on the server process. All successfully opened connections remain open. +!!! note "It is important to use [`hclose`](../ref/hopen.md#hclose) once finished with any connection. Connections do not automatically close if their associated handle is deleted." + ## Closing connections Client or server connections can be closed using [`hclose`](../ref/hopen.md#hclose). @@ -154,22 +156,18 @@ q)h(`add;2;3) / execute the 'add' function as defined on the server, pas #### One-shot message -A sync message can also be sent on a short-lived connection. +A sync message can also be sent on a short-lived connection (called a [one-shot](../ref/hopen.md#one-shot-request)). When sending multiple messages, this is less efficient than [using a pre-existing connection](#sync-request-get) due to the effort of repeated connections/disconnections. A useful shorthand for a one-shot get is: ```q -q)`::5001 "1+1" +q)`::5001 "1+1" 2 ``` - -Since V4.0 2020.03.09, a one-shot query can be run with a timeout (in milliseconds), as in the second example below: - +which is equivalent to ```q -q)`::4567"2+2" -4 -q)`::[(`::4567;100);"1+1"] +q)`:localhost:5001 "1+1" 2 ``` @@ -217,7 +215,7 @@ You may consider increasing the size of TCP send/receive buffers on your system Messages can be queued for sending to a remote process through using async messaging. kdb+ queues the serialized message in user space, later writing it to the socket as the remote end drains the message queue. -You can see how many messages are queued on a handle and their sizes as a dictionary using the command variable [`.z.W`](../ref/dotz.md#zw-handles "handles"). +You can see the queue size using [-38!](internal.md#-38x-socket-table), or [`.z.W`](../ref/dotz.md#zw-handles "handles") for all handles. Sometimes it is useful to send a large number of aysnc messages, but then to block until they have all been sent. This can be achieved through using async flush – invoked as `neg[h][]` or `neg[h](::)`. diff --git a/docs/basics/parsetrees.md b/docs/basics/parsetrees.md index 01cd366ed..ea02d6019 100644 --- a/docs/basics/parsetrees.md +++ b/docs/basics/parsetrees.md @@ -1,13 +1,19 @@ --- title: Parse trees – Basics – kdb+ and q documentation description: A parse tree represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are eval, which evaluates a parse tree, and parse, which returns one from a string containing a valid q expression. -keywords: kdb+, parse, parse tree, q +keywords: kdb+, parse, parse tree, q, k4, k +author: [Peter Storeng, Stephen Taylor, Simon Shanks] --- # Parse trees +## Overview +[`parse`](../ref/parse.md) is a useful tool for seeing how a statement in q is evaluated. Pass the `parse` keyword a q statement as a string and it returns the parse tree of that expression. -A _parse tree_ represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are [`eval`](../ref/eval.md), which evaluates a parse tree, and [`parse`](../ref/parse.md), which returns one from a string containing a valid q expression. +A _parse tree_ represents an expression, not immediately evaluated. Its virtue is that the expression can be evaluated whenever and in whatever context it is needed. The two main functions dealing with parse trees are: + +1. [`eval`](../ref/eval.md), which evaluates a parse tree. +2. [`parse`](../ref/parse.md), which returns one from a string containing a valid q expression. Parse trees may be the result of applying `parse`, or constructed explicitly. The simplest parse tree is a single constant expression. Note that, in a parse tree, a variable is represented by a symbol containing its name. To represent a symbol or a list of symbols, you will need to use [`enlist`](../ref/enlist.md) on that expression. @@ -32,34 +38,283 @@ q)eval ((/;+);(til;(+;2;2))) 6 ``` +## k4, q and `q.k` -## Functional form of a qSQL query +kdb+ is a database management system which ships with the general-purpose and database language q. Q is an embedded domain-specific language implemented in the k programming language, sometimes known as k4. The q interpreter can switch between q and k modes and evaluate expressions written in k as well as q. + +The `parse` keyword can expose the underlying implementation in `k`. + +The k language is for KX implementors. +It is not documented or supported for use outside KX. +All the same functionality is available in the much more readable q language. However in certain cases, such as debugging, a basic understanding of some k syntax can be useful. + +The `q.k` file is part of the standard installation of q and loads into each q session on startup. It defines many of the q keywords in terms of k. To see how a q keyword is defined in terms of k we could check the `q.k` file or simply enter it into the q prompt: + +```q +q)type +@: +``` + +The `parse` keyword on an operation involving the example above exposes the `k` code. Using the underlying code, it can be run using kdb+ in-build k interpreter to show that it produces the same result: +```q +q)type 6 +-7h +q)parse "type 6" +@: +6 +q)k)@6 +-7h +``` + +A few q keywords are defined natively from C and do not have a k representation: + +```q +q)like +like +``` + +## Parse trees + +A parse tree is a q construct which represents an expression but which is not immediately evaluated. It takes the form of a list where the first item is a function and the remaining items are the arguments. Any of the items of the list can be parse trees themselves. + +Note that, in a parse tree, a variable is represented by a symbol containing its name. Thus, to distinguish a symbol or a list of symbols from a variable, it is necessary to enlist that expression. When we apply the `parse` function to create a parse tree, explicit definitions in `.q` are shown in their full k form. In particular, an enlisted element is represented by a preceding comma. + +```q +q)parse"5 6 7 8 + 1 2 3 4" ++ //the function/operator +5 6 7 8 //first argument +1 2 3 4 //second argument +``` +```q +q)parse"2+4*7" ++ //the function/operator +2 //first argument +(*;4;7) //second argument, itself a parse tree +``` +```q +q)v:`e`f +q)`a`b`c,`d,v +`a`b`c`d`e`f +q)parse"`a`b`c,`d,v" +, // join operator +,`a`b`c //actual symbols/lists of symbols are enlisted +(,;,`d;`v) //v a variable represented as a symbol +``` + +We can also manually construct a parse tree: + +```q +q)show pTree:parse "(aggr;data) fby grp" +k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} //fby in k form +(enlist;`aggr;`data) +`grp + +q)pTree~(fby;(enlist;`aggr;`data);`grp) //manually constructed +1b //parse tree +``` + +As asserted previously every statement in q parses into the form: + +```txt +(function; arg 1; …; arg n) +``` + +where every item could itself be a parse tree. In this way we see that every action in q is essentially a function evaluation. + + +## `eval` and `value` + +[`eval`](../ref/eval.md) can be thought of as the dual to `parse`. The following holds for all valid q statements (without side effects) put into a string. (Recall that `value` executes the command inside a string.) + +```q +//a tautology (for all valid q expressions str) +q)value[str]~eval parse str +1b +q)value["2+4*7"]~eval parse"2+4*7" //simple example +1b +``` + +When passed a list, `value` applies the first item (which contains a function) to the rest of the list (the arguments). + +```q +q)function[arg 1;..;arg n] ~ value(function;arg 1;..;arg n) +1b +``` + +When `eval` and `value` operate on a parse tree with no nested parse trees, they return the same result. However it is not true that `eval` and `value` are equivalent in general. `eval` operates on parse trees, evaluating any nested parse trees, whereas `value` operates on the literals. + +```q +q)value(+;7;3) //parse tree, with no nested trees +10 +q)eval(+;7;3) +10 +q)eval(+;7;(+;2;1)) //parse tree with nested trees +10 +q)value(+;7;(+;2;1)) +'type +``` +```q +q)value(,;`a;`b) +`a`b +q)eval(,;`a;`b) //no variable b defined +'b +q)eval(,;enlist `a;enlist `b) +`a`b +``` + + +## Variadic operators -Sometimes you need to translate a [qSQL query](qsql.md) into its [functional form](funsql.md), for example, so you can pass column names as arguments. -Translation can be non-trivial. +Many operators and some keywords in k and q are [variadic](glossary.md#variadic). That means they are overloaded so that the behavior of the operator changes depending on the number and type of arguments. In q (not k), the unary form of operators such as (`+`, `$`, `.`, `&` etc.) is disabled, and keywords are provided instead. -!!! tip "Use `parse` to reveal the functional form of a qSQL query" +For example, in k the unary form of the `$` operator equates to the `string` +keyword in q. -> The result will often include [k code](exposed-infrastructure.md) but it is usually recognizable and you can use it in functional form. — _Q for Mortals_ §A.67 +```q +q)k)$42 +"42" +q)$42 //$ unary form disabled in q +'$ +q)string 42 +"42" +``` + +!!! info "A parenthesized variadic function applied prefix is parsed as its unary form." + +```q +q)($)42 +"42" +``` + +A familiar example of a variadic function is the Add Over function `+/` derived by applying the Over iterator to the Add operator. -Remove one level for the functional form. +```q +q)+/[1000;2 3 4] // +/ applied binary +1009 +q)+/[2 3 4] // +/ applied unary +9 +q)(+/)2 3 4 // +/ applied unary +9 +``` + +In k, the unary form of an operator can also be specified explicitly by suffixing it with a colon. ```q -q)t:([]c1:`a`b`c; c2:10 20 30) -q)parse "select c2:2*c2 from t where c1=`c" -? -`t -,,(=;`c1;,`c) -0b -(,`c2)!,(*;2;`c2) +q)k)$:42 +"42" +``` -q)?[`t; enlist (=;`c1;enlist `c); 0b; (enlist `c2)!enlist (*;2;`c2)] -c2 --- -60 +`+:` is a unary operator; the unary form of `+`. We can see this in the parse tree: + +```q +q)parse"6(+)4" +6 +(+:;4) ``` ----- -:fontawesome-regular-map: -[Parse trees and functional forms](../wp/parse-trees.md) +The items of a `parse` result use k syntax. Since (most of) the q keywords are defined in the `.q` namespace, you can use dictionary reverse lookup to find the meaning. + +```q +q).q?(+:) +`flip +``` + +So we can see that in k, the unary form of `+` corresponds to `flip` in q. + +```q +q)d:`c1`c2`c3!(1 2;3 4;5 6) +q)d +c1| 1 2 +c2| 3 4 +c3| 5 6 +q)k)+d +c1 c2 c3 +-------- +1 3 5 +2 4 6 +q)k)+:d +c1 c2 c3 +-------- +1 3 5 +2 4 6 +``` + +!!! warning "Exposed infrastructure" + + The unary forms of operators are [exposed infrastructure](). + Their use in q expressions is **strongly discouraged**. + Use the corresponding q keywords instead. + + For example, write `flip d` rather than `(+:)d`. + + The unary forms are reviewed here to enable an understanding of parse trees, in which k syntax is visible. + + + +When using reverse lookup on the `.q` context we are slightly hampered by the fact that it is not an injective mapping. The Find `?` operator returns only the first q keyword matching the k expression. In some cases there is more than one. Instead use the following function: + +```q +q)qfind:{key[.q]where x~/:string value .q} + +q)qfind"k){x*y div x:$[16h=abs[@x];\"j\"$x;x]}" +,`xbar +q)qfind"~:" +`not`hdel +``` + +We see `not` and `hdel` are equivalent. Writing the following could be confusing: + +```q +q)hdel 01001b +10110b +``` + +So q provides two different names for clarity. + + +## Iterators as higher-order functions + +An iterator applies to a value (function, list, or dictionary) to produce a related function. This is again easy to see by inspecting the parse tree: + +```q +q)+/[1 2 3 4] +10 +q)parse "+/[1 2 3 4]" +(/;+) +1 2 3 4 +``` + +The first item of the parse tree is `(/;+)`, which is itself a parse +tree. We know the first item of a parse tree is to be applied to the +remaining items. Here `/` (the Over iterator) is applied to `+` to +produce a new function which sums the items of a list. + +:fontawesome-regular-hand-point-right: +[Iterators](../wp/iterators/index.md) + + +## Functional form of a qSQL query + +Sometimes you need to translate a [qSQL query](qsql.md) into its [functional form](funsql.md). For example, so you can pass column names as arguments. +Details are provided [here](funsql.md#conversion-using-parse). diff --git a/docs/database/segment.md b/docs/database/segment.md index 84d977cd7..6971a1217 100644 --- a/docs/database/segment.md +++ b/docs/database/segment.md @@ -49,7 +49,28 @@ db db db `/0/db` is good, but `/0/db/` can be bad, depending on the filesystem. +### Using symlinks +Security related options such as [`reval`](../ref/eval.md#reval), or the command line option [`-u 1`](../basics/cmdline.md#-u-usr-pwd), restrict access to within the current directory. +This can prevent users from accessing a segmented database when `par.txt` contains references to partitions that are situated outside the current directory. + +In order to provide access, symlinks can be used. An example of using symlinks is as follows: + +```bash +$ ln -s /db1 db1$ +$ ln -s /db2 db2$ +$ ls -l +total 16 +lrwxr-xr-x 1 user kx 6 18 Sep 12:30 db1$ -> /db1 +lrwxr-xr-x 1 user kx 6 18 Sep 12:30 db2$ -> /db2 +-rw-r--r-- 1 user kx 10 18 Sep 12:30 par.txt +-rw-r--r-- 1 user kx 48 18 Sep 12:30 sym +$ cat par.txt +db1$ +db2$ +``` + +Using a trailing `$` in the directory name ensures the originating symlinks are not picked up by kdb+, instead using the directory referenced. ## Multithreading diff --git a/docs/github.md b/docs/github.md index 5de692442..7af49604e 100644 --- a/docs/github.md +++ b/docs/github.md @@ -808,7 +808,7 @@ GitHub topic queries:   :fontawesome-brands-github: [timeseries/twitter-kdb](https://github.com/timeseries/twitter-kdb) -[Wind资讯](https://www.wind.com.cn/en/):fontawesome-brands-github: [FlyingOE/q_Wind](https://github.com/FlyingOE/q_Wind) +[Wind资讯](https://www.wind.com.cn/):fontawesome-brands-github: [FlyingOE/q_Wind](https://github.com/FlyingOE/q_Wind) :fontawesome-brands-yahoo: Yahoo!:fontawesome-brands-github: [fdeleze/tickYahoo](https://github.com/fdeleze/tickYahoo) diff --git a/docs/kb/lp.md b/docs/kb/lp.md index 41e5c1c8a..4f424a643 100644 --- a/docs/kb/lp.md +++ b/docs/kb/lp.md @@ -13,7 +13,7 @@ _Linear Programming is a large topic, of which this article reviews just a few a [![Ken Iverson](../img/kei01.jpg)](https://en.wikipedia.org/wiki/Kenneth_E._Iverson "Wikipedia: Kenneth E. Iverson") Q is a descendant of the notation devised at Harvard by the [Turing Award](https://en.wikipedia.org/wiki/Turing_Award) winner, mathematician [Ken Iverson](https://en.wikipedia.org/wiki/Kenneth_E._Iverson), when he worked with [Howard Aiken](https://en.wikipedia.org/wiki/Howard_H._Aiken) and Nobel Prize winner [Wassily Leontief](https://en.wikipedia.org/wiki/Wassily_Leontief) on the computation of economic input-output tables. At Harvard, Ken Iverson and fellow Turing Award winner [Fred Brooks](https://en.wikipedia.org/wiki/Fred_Brooks) gave the world’s first course in what was then called ‘data processing’. - Like other descendants of Iverson Notation (e.g. [A+](http://www.aplusdev.org/index.html), [APL](https://en.wikipedia.org/wiki/APL_(programming_language)), [J](https://en.wikipedia.org/wiki/J_(programming_language))), q inherits compact and powerful expression of linear algebra. + Like other descendants of Iverson Notation (e.g. [A+](https://en.wikipedia.org/wiki/A%2B_(programming_language)), [APL](https://en.wikipedia.org/wiki/APL_(programming_language)), [J](https://en.wikipedia.org/wiki/J_(programming_language))), q inherits compact and powerful expression of linear algebra. Q Math Library: :fontawesome-brands-github: [zholos/qml](https://github.com/zholos/qml) diff --git a/docs/ref/csv.md b/docs/ref/csv.md index f81c7687d..d4d264b2f 100644 --- a/docs/ref/csv.md +++ b/docs/ref/csv.md @@ -18,8 +18,8 @@ A synonym for `","` for use in preparing text for CSV files, or reading them. :fontawesome-solid-book: [Prepare Text](file-text.md#prepare-text), -[`.h.cd`](doth.md#hcd-csv-from-data), -[`.h.td`](doth.md#htd-tsv) +[`.h.cd`](doth.md#hcd-csv-from-data) (csv from data), +[`.h.td`](doth.md#htd-tsv) (tsv from data)
:fontawesome-solid-book-open: [File system](../basics/files.md) diff --git a/docs/ref/doth.md b/docs/ref/doth.md index 53a0c84d0..895dc15db 100644 --- a/docs/ref/doth.md +++ b/docs/ref/doth.md @@ -12,32 +12,33 @@ _Markup tools_
-[`.h.br`](#hbr-linebreak) linebreak [`.h.cd`](#hcd-csv-from-data) CSV from data -[`.h.code`](#hcode-code-after-tab) code after Tab [`.h.d`](#hd-delimiter) delimiter -[`.h.fram`](#hfram-frame) frame [`.h.ed`](#hed-excel-from-data) Excel from data -[`.h.ha`](#hha-anchor) anchor [`.h.edsn`](#hedsn-excel-from-tables) Excel from tables -[`.h.hb`](#hhb-anchor-target) anchor target [`.h.hc`](#hhc-escape-lt) escape lt -[`.h.ht`](#hht-marqdown-to-html) Marqdown to HTML [`.h.hr`](#hhr-horizontal-rule) horizontal rule -[`.h.hta`](#hhta-start-tag) start tag [`.h.iso8601`](#hiso8601-iso-timestamp) ISO timestamp -[`.h.htac`](#hhtac-element) element [`.h.jx`](#hjx-table) table -[`.h.htc`](#hhtc-element) element [`.h.td`](#htd-tsv-from-data) TSV from data -[`.h.html`](#hhtml-document) document [`.h.tx`](#htx-filetypes) filetypes -[`.h.http`](#hhttp-hyperlinks) hyperlinks [`.h.xd`](#hxd-xml) XML -[`.h.nbr`](#hnbr-no-break) no break [`.h.xs`](#hxs-xml-escape) XML escape -[`.h.pre`](#hpre-pre) pre [`.h.xt`](#hxt-json) JSON -[`.h.text`](#htext-paragraphs) paragraphs -[`.h.xmp`](#hxmp-xmp) XMP - -[`.h.he`](#hhe-http-400) HTTP 400 [`.h.c0`](#hc0-web-color) web color -[`.h.hn`](#hhn-http-response) HTTP response [`.h.c1`](#hc1-web-color) web color -[`.h.hp`](#hhp-http-response-pre) HTTP response pre [`.h.HOME`](#hhome-webserver-root) webserver root -[`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.logo`](#hlogo-kx-logo) KX logo -[`.h.ka`](#hka-http-keepalive) HTTP keep-alive [`.h.sa`](#hsa-anchor-style) anchor style - [`.h.sb`](#hsb-body-style) body style -[`.h.hu`](#hhu-uri-escape) URI escape [`.h.ty`](#hty-mime-types) MIME types -[`.h.hug`](#hhug-uri-map) URI map [`.h.val`](#hval-value) value -[`.h.sc`](#hsc-uri-safe) URI-safe -[`.h.uh`](#huh-uri-unescape) URI unescape +Markup (HTML and XML) Data Serialization + [`.h.br`](#hbr-linebreak) linebreak [`.h.cd`](#hcd-csv-from-data) CSV from data + [`.h.code`](#hcode-code-after-tab) code after Tab [`.h.d`](#hd-delimiter) delimiter + [`.h.fram`](#hfram-frame) frame [`.h.ed`](#hed-excel-from-data) Excel from data + [`.h.ha`](#hha-anchor) anchor [`.h.edsn`](#hedsn-excel-from-tables) Excel from tables + [`.h.hb`](#hhb-anchor-target) anchor target [`.h.ht`](#hht-marqdown-to-html) Marqdown to HTML + [`.h.hc`](#hhc-escape-lt) escape lt [`.h.iso8601`](#hiso8601-iso-timestamp) ISO timestamp + [`.h.hr`](#hhr-horizontal-rule) horizontal rule [`.h.jx`](#hjx-table) table + [`.h.hta`](#hhta-start-tag) start tag [`.h.td`](#htd-tsv-from-data) TSV from data + [`.h.htac`](#hhtac-element) element [`.h.tx`](#htx-filetypes) filetypes + [`.h.htc`](#hhtc-element) element [`.h.xd`](#hxd-xml) XML from data + [`.h.html`](#hhtml-document) document [`.h.xt`](#hxt-json) JSON + [`.h.http`](#hhttp-hyperlinks) hyperlinks + [`.h.logo`](#hlogo-kx-logo) KX logo Web Console + [`.h.nbr`](#hnbr-no-break) no break [`.h.c0`](#hc0-web-color) web color + [`.h.pre`](#hpre-pre) pre [`.h.c1`](#hc1-web-color) web color + [`.h.text`](#htext-paragraphs) paragraphs [`.h.HOME`](#hhome-webserver-root) webserver root + [`.h.xmp`](#hxmp-xmp) XMP [`.h.sa`](#hsa-anchor-style) anchor style + [`.h.xs`](#hxs-xml-escape) XML escape [`.h.sb`](#hsb-body-style) body style + [`.h.val`](#hval-value) value +HTTP + [`.h.he`](#hhe-http-400) HTTP 400 URI formatting + [`.h.hn`](#hhn-http-response) HTTP response [`.h.hu`](#hhu-uri-escape) URI escape + [`.h.hp`](#hhp-http-response-pre) HTTP response pre [`.h.hug`](#hhug-uri-map) URI map + [`.h.hy`](#hhy-http-response-content) HTTP response content [`.h.sc`](#hsc-uri-safe) URI-safe + [`.h.ka`](#hka-http-keepalive) HTTP keep-alive [`.h.uh`](#huh-uri-unescape) URI unescape + [`.h.ty`](#hty-mime-types) MIME types
The `.h` [namespace](../basics/namespaces.md) contains objects for @@ -92,6 +93,8 @@ q).h.cd (`a`b`c;1 2 3;"xyz") Columns can be nested vectors, in which case [`.h.d`](#hd-delimiter) is used to separate subitems. (Since V4.0 2020.03.17.) +:fontawesome-regular-hand-point-right: +[0: load csv](file-text.md#load-csv) ## `.h.code` (code after Tab) @@ -861,3 +864,5 @@ q)first .h.xt[`json;("{\"foo\":\"bar\"}";"{\"this\":\"that\"}")] foo| "bar" ``` +:fontawesome-regular-hand-point-right: +[`.j` namespace](dotj.md) (JSON de/serialization) diff --git a/docs/ref/dotq.md b/docs/ref/dotq.md index 3570a49fb..600e84328 100644 --- a/docs/ref/dotq.md +++ b/docs/ref/dotq.md @@ -11,53 +11,54 @@ _Tools_
**General** **Datatype** [addmonths](#addmonths) [btoa b64 encode](#btoa-b64-encode) - [bt backtrace](#bt-backtrace) [j10 encode binhex](#j10-encode-binhex) - [dd join symbols](#dd-join-symbols) [j12 encode base 36](#j12-encode-base-36) - [def parse options](#def-parse-options) [ty type](#ty-type) - [f format](#f-format) [x10 decode binhex](#x10-decode-binhex) - [fc parallel on cut](#fc-parallel-on-cut) [x12 decode base 36](#x12-decode-base-36) - [ff append columns](#ff-append-columns) - [fmt format](#fmt-format) **Database** - [ft apply simple](#ft-apply-simple) [chk fill HDB](#chk-fill-hdb) - [fu apply unique](#fu-apply-unique) [dpft dpfts save table](#dpft-save-table) - [gc garbage collect](#gc-garbage-collect) [dpt dpts save table unsorted](#dpt-save-table-unsorted) - [gz GZip](#gz-gzip) [dsftg load process save](#dsftg-load-process-save) - [id sanitize](#id-sanitize) [en enumerate varchar cols](#en-enumerate-varchar-cols) - [prf0 code profiler](#prf0-code-profiler) [ens enumerate against domain](#ens-enumerate-against-domain) - [qt is table](#qt-is-table) [fk foreign key](#fk-foreign-key) - [res keywords](#res-keywords) [hdpf save tables](#hdpf-save-tables) - [s plain text](#s-plain-text) [l load](#l-load) - [s1 string representation](#s1-string-representation) [ld load and group](#ld-load-and-group) - [sbt string backtrace](#sbt-string-backtrace) [lo load without](#lo-load-without) - [sha1 SHA-1 encode](#sha1-sha-1-encode) [M chunk size](#m-chunk-size) - [trp extend trap at](#trp-extend-trap-at) [qp is partitioned](#qp-is-partitioned) - [trpd extend trap](#trpd-extend-trap) [qt is table](#qt-is-table) - [ts time and space](#ts-time-and-space) - [u date based](#u-date-based) **Partitioned database state** - [V table to dict](#v-table-to-dict) [bv build vp](#bv-build-vp) - [v value](#v-value) [cn count partitioned table](#cn-count-partitioned-table) - [view subview](#view-subview) [D partitions](#d-partitions) - [ind partitioned index](#ind-partitioned-index) -**Constants** [MAP maps partitions](#map-maps-partitions) - [A a an alphabets](#a-upper-case-alphabet) [par locate partition](#par-locate-partition) - [b6 bicameral alphanums](#b6-bicameral-alphanums) [PD partition locations](#pd-partition-locations) - [n nA nums & alphanums](#n-nums) [pd modified partition locns](#pd-modified-partition-locations) - [pf partition field](#pf-partition-field) -**Environment** [pn partition counts](#pn-partition-counts) - [K k version](#k-version-date) [pt partitioned tables](#pt-partitioned-tables) - [opt command parameters](#opt-command-parameters) [PV partition values](#pv-partition-values) - [w memory stats](#w-memory-stats) [pv modified partition values](#pv-modified-partition-values) - [x non-command parameters](#x-non-command-parameters) [qp is partitioned](#qp-is-partitioned) - [vp missing partitions](#vp-missing-partitions) -**IPC** - [addr IP address](#addr-ip-address) **Segmented database state** - [fps fpn streaming algorithm](#fpn-streaming-algorithm) [P segments](#p-segments) - [fs fsn streaming algorithm](#fs-streaming-algorithm) [u date based](#u-date-based) - [hg HTTP get](#hg-http-get) - [host hostname](#host-hostname) **File I/O** - [hp HTTP post](#hp-http-post) [Cf create empty nested char file](#cf-create-empty-nested-char-file) - [Xf create file](#xf-create-file) - + [dd join symbols](#dd-join-symbols) [j10 encode binhex](#j10-encode-binhex) + [f format](#f-format) [j12 encode base 36](#j12-encode-base-36) + [fc parallel on cut](#fc-parallel-on-cut) [ty type](#ty-type) + [ff append columns](#ff-append-columns) [x10 decode binhex](#x10-decode-binhex) + [fmt format](#fmt-format) [x12 decode base 36](#x12-decode-base-36) + [ft apply simple](#ft-apply-simple) + [fu apply unique](#fu-apply-unique) **Database** + [gc garbage collect](#gc-garbage-collect) [chk fill HDB](#chk-fill-hdb) + [gz GZip](#gz-gzip) [dpft dpfts save table](#dpft-save-table) + [id sanitize](#id-sanitize) [dpt dpts save table unsorted](#dpt-save-table-unsorted) + [qt is table](#qt-is-table) [dsftg load process save](#dsftg-load-process-save) + [res keywords](#res-keywords) [en enumerate varchar cols](#en-enumerate-varchar-cols) + [s plain text](#s-plain-text) [ens enumerate against domain](#ens-enumerate-against-domain) + [s1 string representation](#s1-string-representation) [fk foreign key](#fk-foreign-key) + [sha1 SHA-1 encode](#sha1-sha-1-encode) [hdpf save tables](#hdpf-save-tables) + [V table to dict](#v-table-to-dict) [l load](#l-load) + [v value](#v-value) [ld load and group](#ld-load-and-group) + [view subview](#view-subview) [li load partitions](#li-load-partitions) + [lo load without](#lo-load-without) + **Constants** [M chunk size](#m-chunk-size) + [A a an alphabets](#a-upper-case-alphabet) [qp is partitioned](#qp-is-partitioned) + [b6 bicameral alphanums](#b6-bicameral-alphanums) [qt is table](#qt-is-table) + [n nA nums & alphanums](#n-nums) + **Partitioned database state** + **Debug/Profile** [bv build vp](#bv-build-vp) + [bt backtrace](#bt-backtrace) [bvi build incremental vp](#bvi-build-incremental-vp) + [prf0 code profiler](#prf0-code-profiler) [cn count partitioned table](#cn-count-partitioned-table) + [sbt string backtrace](#sbt-string-backtrace) [D partitions](#d-partitions) + [trp extend trap at](#trp-extend-trap-at) [ind partitioned index](#ind-partitioned-index) + [trpd extend trap](#trpd-extend-trap) [MAP maps partitions](#map-maps-partitions) + [ts time and space](#ts-time-and-space) [par locate partition](#par-locate-partition) + [PD partition locations](#pd-partition-locations) + **Environment** [pd modified partition locns](#pd-modified-partition-locations) + [K k version](#k-version-date) [pf partition field](#pf-partition-field) + [w memory stats](#w-memory-stats) [pn partition counts](#pn-partition-counts) + [pt partitioned tables](#pt-partitioned-tables) + **Environment (Command-line)** [PV partition values](#pv-partition-values) + [def command defaults](#def-command-defaults) [pv modified partition values](#pv-modified-partition-values) + [opt command parameters](#opt-command-parameters) [qp is partitioned](#qp-is-partitioned) + [x non-command parameters](#x-non-command-parameters) [vp missing partitions](#vp-missing-partitions) + + **IPC** **Segmented database state** + [addr IP address](#addr-ip-address) [P segments](#p-segments) + [fps fpn streaming algorithm](#fpn-streaming-algorithm) [u date based](#u-date-based) + [fs fsn streaming algorithm](#fs-streaming-algorithm) + [hg HTTP get](#hg-http-get) **File I/O** + [host hostname](#host-hostname) [Cf create empty nested char file](#cf-create-empty-nested-char-file) + [hp HTTP post](#hp-http-post) [Xf create file](#xf-create-file)
@@ -218,12 +219,16 @@ Since V3.6 2018.05.18. .Q.bv[`] ``` -In partitioned DBs, construct the dictionary `.Q.vp` of table schemas for tables with missing partitions. Optionally allow tables to be missing from partitions, by scanning partitions for missing tables and taking the tables’ prototypes from the last partition. After loading/re-loading from the filesystem, invoke `.Q.bv[]` to (re)populate `.Q.vt`/`.Q.vp`, which are used inside `.Q.p1` during the partitioned select `.Q.ps`. +In partitioned DBs, construct the dictionary [`.Q.vp`](#vp-missing-partitions) of table schemas for tables with missing partitions. Optionally allow tables to be missing from partitions, by scanning partitions for missing tables and taking the tables’ prototypes from the last partition. + +After loading/re-loading from the filesystem, invoke `.Q.bv[]` to (re)populate `.Q.vt`/`.Q.vp`, which are used inside `.Q.p1` during the partitioned select `.Q.ps`. (Since V2.8 2012.01.20, modified V3.0 2012.01.26) -If your table exists at least in the latest partition (so there is a prototype for the schema), you could use `.Q.bv[]` to create empty tables on the fly at run-time without having to create those empties on disk. ``.Q.bv[`]`` (with argument) will use prototype from first partition instead of last. (Since V3.2 2014.08.22.) +If your table exists at least in the latest partition (so there is a prototype for the schema), you could use `.Q.bv[]` to create empty tables on the fly at run-time without having to create those empties on disk. -Some admins prefer to see errors instead of auto-manufactured empties for missing data, which is why `.Q.bv` is not the default behavior. +``.Q.bv[`]`` (with argument) will use prototype from first partition instead of last. (Since V3.2 2014.08.22.) + +!!! note "Some admins prefer to see errors instead of auto-manufactured empties for missing data, which is why `.Q.bv` is not the default behavior." ```q q)n:100 @@ -245,6 +250,11 @@ q)@[get;"select from tt";-2@]; / no error ``` +## `bvi` (build incremental vp) + +It offers the same functionality as [`.Q.bv`](#bv-build-vp), but scans only new partitions loaded in the hdb since the last time `.Q.bv` or `.Q.bvi` was run. Since v4.1 2024.09.13. + + ## `Cf` (create empty nested char file) !!! warning "Deprecated" @@ -255,7 +265,7 @@ q)@[get;"select from tt";-2@]; / no error .Q.Cf x ``` -A projection of `.Q.Xf`: i.e. ``.Q.Xf[`char;]`` +A projection of [`.Q.Xf`](#xf-create-file): i.e. ``.Q.Xf[`char;]`` ## `chk` (fill HDB) @@ -295,7 +305,7 @@ _Q for Mortals_ .Q.cn x ``` -Where `x` is a partitioned table, passed by value, returns its count. Populates `.Q.pn` cache. +Where `x` is a partitioned table, passed by value, returns its count. Populates [`.Q.pn`](#pn-partition-counts) cache. ## `D` (partitions) @@ -304,7 +314,7 @@ Where `x` is a partitioned table, passed by value, returns its count. Populates .Q.D ``` -In segmented DBs, contains a list of the partitions – conformant to `.Q.P` – that are present in each segment. +In segmented DBs, contains a list of the partitions – conformant to [`.Q.P`](#p-segments) – that are present in each segment. `.Q.P!.Q.D` can be used to create a dictionary of partition-to-segment information. @@ -346,19 +356,43 @@ AAPL O AAPL.O IBM N IBM.N ``` +[](){#def-parse-options} +## `def` (command defaults) -## `def` (parse options) - -_Default values for command-line arguments_ +_Default values and type checks for command-line arguments parsed with [`.Q.opt`](#opt-command-parameters)_ ```syntax .Q.def[x;y] ``` -Provides defaults and types for command-line arguments parsed with [``.Q.opt``](#opt-command-parameters). +Where `x` is a dictionary of default parameter names and values, and `y` is the output of `.Q.opt`. -:fontawesome-solid-book: -[`.z.x`](dotz.md#zx-argv) +Types are inferred from the default values provided, which must be an atom type. + +```bash +$ q -abc 123 -xyz 321 +``` +```q +q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x +abc| 123 +xyz| 321f +efg| `a +``` + +If a command-line value cannot be [converted to the data type](tok.md) of the default value, a [null](../basics/datatypes.md) is produced + +```bash +$ q -param1 11 -param2 2000.01.01 -param3 wrong +``` +```q +q).Q.def[`param1`param2`param3!(1;1999.01.01;23.1)].Q.opt .z.x +param1| 11 +param2| 2000.01.01 +param3| 0n +``` + +:fontawesome-solid-hand-point-right: +[`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.opt`](#opt-command-parameters) (command parameters), [`.Q.x`](#x-non-command-parameters) (non-command parameters) ## `dpft` (save table) @@ -554,7 +588,7 @@ Where returns `y` as a string formatted as a float to `x` decimal places. -Because of the limits of precision in a double, for `y` above `1e13` or the limit set by `\P`, formats in scientific notation. +Because of the limits of precision in a double, for `y` above `1e13` or the limit set by [`\P`](../basics/syscmds.md#p-precision), formats in scientific notation. ```q q)\P 0 @@ -599,7 +633,7 @@ q)\t .Q.fc[f]vec 6 ``` -In this case the overhead of creating threads in `peach` significantly outweighs the computational benefit of parallel execution. +In this case the overhead of creating threads in [`peach`](each.md) significantly outweighs the computational benefit of parallel execution. ```q q)\t f peach vec @@ -1107,7 +1141,7 @@ Where `x` is `ab`ab`twowords`a2drifters`a22 ``` -- a **table**, returns `x` with column names sanitized by removing characters that interfere with `select/exec/update` and adding `"1"` to column names which clash with commands in the `.q` namespace. (Updated in V3.2 to include `.Q.res` for checking collisions.) +- a **table**, returns `x` with column names sanitized by removing characters that interfere with `select/exec/update` and adding `"1"` to column names which clash with commands in the `.q` namespace. Updated in V3.2 to include [`.Q.res`](#res-keywords) for checking collisions. ```q q).Q.id flip (5#.Q.res)!(5#()) @@ -1118,6 +1152,17 @@ Where `x` is ---- ``` +- a **dictionary** (since v4.1 2024.09.13), supports the same rules as `table` above + + ```q + q).Q.id (5#.Q.res)!(5#()) + abs1 | + acos1| + asin1| + atan1| + avg1 | + ``` + Since 4.1t 2022.03.25,4.0 2022.10.26 produces a symbol `a` when the input contains a single character that is not in [.Q.an](#all-alphanumerics) (it previously produced an empty sym) e.g. ```q @@ -1275,6 +1320,32 @@ q).Q.ld read0`:funcs.q "/ multi line func" "f:{\n x+y\n }" "/ single line func" "g:{x*y}" ``` +## `li` (load partitions) + +```syntax +.Q.li[partitions] +``` + +In the current hdb, adds any partition(s) which are both in the list supplied and on disk. Partitions can be a list or atomic variable. For example: + +```q +q)`:/tmp/db/2001.01.01/t/ set tt:.Q.en[`:/tmp/db]([]sym:10?`A`B`C;time:10?.z.T;price:10?10f) +q)\l /tmp/db +q)`:2001.01.02/t/`:2001.01.03/t/ set\:tt +q)date +,2001.01.01 +q).Q.li[2001.01.02];date +2001.01.01 2001.01.02 +q).Q.li[2001.01.02 2001.01.03];select count i by date from t +date | x +----------| -- +2001.01.01| 10 +2001.01.02| 10 +2001.01.03| 10 +``` + +Since v4.1 2024.09.20. + ## `lo` (load without) @@ -1333,7 +1404,7 @@ q)0W~.Q.M / defaults to long infinity Keeps partitions mapped to avoid the overhead of repeated file system calls during a `select`. (Since V3.1.) -For use with partitioned HDBS, used in tandem with `\l dir` +For use with partitioned HDBS, used in tandem with [`\l dir`](../basics/syscmds.md#l-load-file-or-directory) ```q q)\l . @@ -1386,10 +1457,32 @@ q).Q.nA .Q.opt .z.x ``` -Returns a dictionary, so you can easily see if a key was defined (flag set or not) or, if a value is passed, to refer to it by its key. +Presents command-line arguments as a dictionary, using the output of [`.z.x`](dotz.md#zx-argv). Defaults can be added using [`.Q.def`](#def-command-defaults). -:fontawesome-solid-book: -[`.z.x`](dotz.md#zx-argv) +```bash +$ q -param1 val1 -param2 val2 +``` +```q +q)params:.Q.opt .z.x +q)show params +param1| "val1" +param2| "val2" +q)params`param1 +"val1" +``` +Example of a command-line parameter with no value and a parameter with multiple values: +```bash +$ q -param1 -param2 as asd -param3 +``` +```q +q).Q.opt .z.x +param1| () +param2| ("as";"asd") +param3| () +``` + +:fontawesome-solid-hand-point-right: +[`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.def`](#def-command-defaults) (command defaults), [`.Q.x`](#x-non-command-parameters) (non-command parameters) ## `P` (segments) @@ -1441,7 +1534,7 @@ q)all{`p=attr .Q.par[`:.;x;`quote]`sym}each date .Q.PD ``` -In partitioned DBs, a list of partition locations – conformant to `.Q.PV` – which represents the partition location for each partition. +In partitioned DBs, a list of partition locations – conformant to [`.Q.PV`](#pv-partition-values) – which represents the partition location for each partition. (In non-segmented DBs, this will be simply ``count[.Q.PV]#`:.``.) `.Q.PV!.Q.PD` can be used to create a dictionary of partition-to-location information. @@ -1467,7 +1560,7 @@ q).Q.PV!.Q.PD .Q.pd ``` -In partitioned DBs, `.Q.PD` as modified by `.Q.view`. +In partitioned DBs, [`.Q.PD`](#pd-partition-locations) as modified by [`.Q.view`](#view-subview). ## `pf` (partition field) @@ -1486,8 +1579,9 @@ Possible values are `` `date`month`year`int``. .Q.pn ``` -In partitioned DBs, returns a dictionary of cached partition counts – conformant to `.Q.pt`, each conformant to `.Q.pv` – as populated by `.Q.cn`. -Cleared by `.Q.view`. +In partitioned DBs, returns a dictionary of cached partition counts – conformant to [`.Q.pt`](#pt-partitioned-tables), each conformant to [`.Q.pv`](#pv-modified-partition-values) – as populated by [`.Q.cn`](#cn-count-partitioned-table). + +Cleared by [`.Q.view`](#view-subview). `.Q.pv!flip .Q.pn` can be used to create a crosstab of table-to-partition-counts once `.Q.pn` is fully populated. @@ -1571,7 +1665,7 @@ Returns a list of partitioned tables. A list of the values of the partition domain: the values corresponding to the slice directories actually found in the root. -In partitioned DBs, `.Q.PV` as modified by `.Q.view`. +In partitioned DBs, [`.Q.PV`](#pv-partition-values) as modified by [`.Q.view`](#view-subview). :fontawesome-solid-street-view: _Q for Mortals_ @@ -1584,8 +1678,8 @@ _Q for Mortals_ .Q.PV ``` -In partitioned DBs, returns a list of partition values – conformant to `.Q.PD` – which represents the partition value for each partition. -(In a date-partitioned DB, unless the date has been modified by `.Q.view`, this will be simply date.) +In partitioned DBs, returns a list of partition values – conformant to [`.Q.PD`](#pd-partition-locations) – which represents the partition value for each partition. +(In a date-partitioned DB, unless the date has been modified by [`.Q.view`](#view-subview), this is simply date.) ```q q).Q.PD @@ -1749,7 +1843,7 @@ extends [Trap At](apply.md#trap-at) (`@[f;x;g]`) to collect backtrace: `g` gets 1. the error string 2. the backtrace object -You can format the backtrace object with `.Q.sbt`. +You can format the backtrace object with [`.Q.sbt`](#sbt-string-backtrace). ```q q)f:{`hello+x} @@ -1820,7 +1914,7 @@ extends [Trap](apply.md#trap) (`.[f;x;g]`) to collect backtrace: `g` is called w 1. the error string 2. the backtrace object -You can format the backtrace object with `.Q.sbt`. +You can format the backtrace object with [`.Q.sbt`](#sbt-string-backtrace). ```q q).Q.trpd[{x+y};(1;2);{2"error: ",x,"\nbacktrace:\n",.Q.sbt y;-1}] @@ -1972,7 +2066,7 @@ _Q for Mortals_ .Q.vp ``` -In partitioned DBs, returns a dictionary of table schemas for tables with missing partitions, as populated by `.Q.bv`. +In partitioned DBs, returns a dictionary of table schemas for tables with missing partitions, as populated by [`.Q.bv`](#bv-build-vp). (Since V3.0 2012.01.26.) ```q @@ -2052,12 +2146,11 @@ q)type get`:emptyNestedCharVector .Q.x ``` -Set by `.Q.opt`: a list of _non-command_ parameters from the command line, where _command parameters_ are prefixed by `-`. +Set by [`.Q.opt`](#opt-command-parameters): a list of _non-command_ parameters from the command line, where _command parameters_ are prefixed by `-`. ```bash -~$ q taq.k path/to/source path/to/destn +$ q taq.k path/to/source path/to/destn ``` - ```q q)cla:.Q.opt .z.X /command-line arguments q).Q.x @@ -2066,8 +2159,7 @@ q).Q.x "path/to/destn" ``` -:fontawesome-solid-book: -[`.z.x`](dotz.md#zx-argv), -[`.z.X`](dotz.md#zx-raw-command-line) +:fontawesome-solid-hand-point-right: +[`.z.x`](dotz.md#zx-argv) (argv), [`.z.X`](dotz.md#zx-raw-command-line) (raw command line), [`.z.f`](dotz.md#zf-file) (file), [`.z.q`](dotz.md#zq-quiet-mode) (quiet mode), [`.Q.opt`](#opt-command-parameters) (command parameters), [`.Q.def`](#def-command-defaults) (command defaults) diff --git a/docs/ref/dotz.md b/docs/ref/dotz.md index 4935a7f8b..5e6b71c7e 100644 --- a/docs/ref/dotz.md +++ b/docs/ref/dotz.md @@ -11,32 +11,36 @@ keywords: callbacks, environment, kdb+, q _Environment and callbacks_
-Environment Callbacks - [.z.a IP address](#za-ip-address) [.z.ac HTTP auth](#zac-http-auth) - [.z.b dependencies](#zb-dependencies) [.z.bm msg validator](#zbm-msg-validator) - [.z.c cores](#zc-cores) [.z.exit action on exit](#zexit-action-on-exit) - [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.pc close](#zpc-close) - [.z.e TLS connection status](#ze-tls-connection-status) [.z.pd peach handles](#zpd-peach-handles) - [.z.ex failed primitive](#zex-failed-primitive) [.z.pg get](#zpg-get) - [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.ph HTTP get](#zph-http-get) - [.z.f file](#zf-file) [.z.pi input](#zpi-input) - [.z.H active sockets](#zh-active-sockets) [.z.pm HTTP methods](#zpm-http-methods) - [.z.h host](#zh-host) [.z.po open](#zpo-open) - [.z.i PID](#zi-pid) [.z.pp HTTP post](#zpp-http-post) - [.z.K version](#zk-version) [.z.pq qcon](#zpq-qcon) - [.z.k release date](#zk-release-date) [.z.r blocked](#zr-blocked) - [.z.l license](#zl-license) [.z.ps set](#zps-set) - [.z.N/n local/UTC timespan](#zn-local-timespan) [.z.pw validate user](#zpw-validate-user) - [.z.o OS version](#zo-os-version) [.z.ts timer](#zts-timer) - [.z.P/p local/UTC timestamp](#zp-local-timestamp) [.z.vs value set](#zvs-value-set) - [.z.q quiet mode](#zq-quiet-mode) [.z.wc WebSocket close](#zwc-websocket-close) - [.z.s self](#zs-self) [.z.wo WebSocket open](#zwo-websocket-open) - [.z.T/t time shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.ws WebSockets](#zws-websockets) - [.z.u user ID](#zu-user-id) - [.z.W/w handles/handle](#zw-handles) - [.z.X/x raw/parsed command line](#zx-raw-command-line) +Environment Callbacks + [.z.a IP address](#za-ip-address) [.z.bm msg validator](#zbm-msg-validator) + [.z.b view dependencies](#zb-view-dependencies) [.z.exit action on exit](#zexit-action-on-exit) + [.z.c cores](#zc-cores) [.z.pc close](#zpc-close) + [.z.ex failed primitive](#zex-failed-primitive) [.z.pd peach handles](#zpd-peach-handles) + [.z.ey arg to failed primitive](#zey-argument-to-failed-primitive) [.z.pg get](#zpg-get) + [.z.f file](#zf-file) [.z.pi input](#zpi-input) + [.z.h host](#zh-host) [.z.po open](#zpo-open) + [.z.i PID](#zi-pid) [.z.pq qcon](#zpq-qcon) + [.z.K version](#zk-version) [.z.r blocked](#zr-blocked) + [.z.k release date](#zk-release-date) [.z.ps set](#zps-set) + [.z.l license](#zl-license) [.z.pw validate user](#zpw-validate-user) + [.z.o OS version](#zo-os-version) [.z.ts timer](#zts-timer) + [.z.q quiet mode](#zq-quiet-mode) [.z.vs value set](#zvs-value-set) + [.z.s self](#zs-self) + [.z.u user ID](#zu-user-id) Callbacks (HTTP) + [.z.X/x raw/parsed command line](#zx-raw-command-line) [.z.ac HTTP auth](#zac-http-auth) + [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults) [.z.ph HTTP get](#zph-http-get) + [.z.pm HTTP methods](#zpm-http-methods) +Environment (Connections) [.z.pp HTTP post](#zpp-http-post) + [.z.e TLS connection status](#ze-tls-connection-status) + [.z.H active sockets](#zh-active-sockets) Callbacks (WebSockets) + [.z.W/w handles/handle](#zw-handles) [.z.wc WebSocket close](#zwc-websocket-close) + [.z.wo WebSocket open](#zwo-websocket-open) +Environment (Time/Date) [.z.ws WebSockets](#zws-websockets) + [.z.D/d date shortcuts](#zt-zt-zd-zd-timedate-shortcuts) + [.z.N/n local/UTC timespan](#zn-local-timespan) + [.z.P/p local/UTC timestamp](#zp-local-timestamp) + [.z.T/t time shortcuts](#zt-zt-zd-zd-timedate-shortcuts) [.z.Z/z local/UTC datetime](#zz-local-datetime) - [.z.zd compression/encryption defaults](#zzd-compressionencryption-defaults)
The `.z` [namespace](../basics/namespaces.md) contains environment variables and functions, and hooks for callbacks. @@ -130,8 +134,8 @@ If the user is not permitted, the client is sent a default 401 HTTP unauthorized :fontawesome-solid-graduation-cap:[HTTP](../kb/http.md) - -## `.z.b` (dependencies) +[](){#zb-dependencies} +## `.z.b` (view dependencies) The dependency dictionary. @@ -145,6 +149,9 @@ y| ,`a :fontawesome-solid-book-open: [`\b`](../basics/syscmds.md#b-views) +
+:fontawesome-solid-graduation-cap: +[Views](../learn/views.md) ## `.z.bm` (msg validator) @@ -297,9 +304,9 @@ q).z.f ## `.z.H` (active sockets) -Active sockets as a list. (A low-cost method.) +Active sockets as a list (a low-cost method). Since v4.0 2020.06.01. -Since v4.0 2020.06.01. +List has [sorted attribute](set-attribute.md#sorted) applied since v4.1 2024.07.08. ```q q).z.H~key .z.W @@ -307,10 +314,7 @@ q).z.H~key .z.W ``` :fontawesome-solid-hand-point-right: -[`.z.W` handles](#zw-handles), [`.z.w` handle](#zw-handle) -
-:fontawesome-solid-book-open: -[`-38!` socket table](../basics/internal.md#-38x-socket-table) +[`.z.W` handles](#zw-handles), [`.z.w` handle](#zw-handle), [`-38!` socket table](../basics/internal.md#-38x-socket-table) ## `.z.h` (host) @@ -951,11 +955,17 @@ q)neg[h]({};til 1000000); neg[h]({};til 10); sum each .z.W 6| 8000140 ``` +Querying known handles can also be performed using [`-38!`](../basics/internal.md#-38x-socket-table), which can be more performant than using `.z.W` to return the entire dataset of handles. +```q +q)h:hopen 5000 +q)neg[h]"11+1111111";.z.W h +24 +q)neg[h]"11+1111111";(-38!h)`m +24 +``` + :fontawesome-solid-hand-point-right: -[`.z.h` active sockets](#zh-active-sockets), [`.z.w` handle](#zw-handle) -
-:fontawesome-solid-book-open: -[`-38!` socket table](../basics/internal.md#-38x-socket-table) +[`.z.h` active sockets](#zh-active-sockets), [`.z.w` handle](#zw-handle), [`-38!` socket table](../basics/internal.md#-38x-socket-table) ## `.z.w` (handle) @@ -970,10 +980,7 @@ q).z.w !!! warning "Inside a `.z.p`* callback it returns the handle of the client session, not the current session." :fontawesome-solid-hand-point-right: -[`.z.h` active sockets](#zh-active-sockets), [`.z.W` handles](#zw-handles) -
-:fontawesome-solid-book-open: -[`-38!` socket table](../basics/internal.md#-38x-socket-table) +[`.z.h` active sockets](#zh-active-sockets), [`.z.W` handles](#zw-handles), [`-38!` socket table](../basics/internal.md#-38x-socket-table) ## `.z.wc` (websocket close) @@ -1067,7 +1074,7 @@ q).z.X ``` :fontawesome-solid-hand-point-right: -[`.z.x` argv](#zx-argv), [`.z.f` file](#zf-file), [`.z.q` quiet mode](#zq-quiet-mode) +[`.z.x`](#zx-argv) (argv), [`.z.f`](#zf-file) (file), [`.z.q`](#zq-quiet-mode) (quiet mode), [`.Q.opt`](dotq.md#opt-command-parameters) (command parameters), [`.Q.def`](dotq.md#def-command-defaults) (command defaults), [`.Q.x`](dotq.md#x-non-command-parameters) (non-command parameters) ## `.z.x` (argv) @@ -1077,50 +1084,16 @@ Command-line arguments as a list of strings ```q $ q test.q -P 0 -abc 123 q).z.x -("-abc";"123") -``` - -Note that the script name and the single-letter options used by q itself are not included. - -Command-line options can be converted to a dictionary using the convenient `.Q.opt` function. - -```bash -$ q -abc 123 -xyz 321 -``` - -```q -q).Q.opt .z.x -abc| "123" -xyz| "321" -``` - -Defaults and types can be provided with `.Q.def`. - -```bash -$ q -abc 123 -xyz 321 +"-abc" +"123" ``` -```q -q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x -abc| 123 -xyz| 321f -efg| `a -q)\\ -``` - -```bash -$ q -abc 123 -xyz 321 -efg foo -``` +!!! Note "The script name and the single-letter options used by q itself are not included." -```q -q).Q.def[`abc`xyz`efg!(1;2.;`a)].Q.opt .z.x -abc| 123 -xyz| 321f -efg| `foo -``` +Command-line options can be converted to a dictionary using the convenient [`.Q.opt`](dotq.md#opt-command-parameters) function. :fontawesome-solid-hand-point-right: -[`.z.X` raw command line](#zx-raw-command-line), [`.z.f` file](#zf-file), [`.z.q` quiet mode](#zq-quiet-mode) +[`.z.X`](#zx-raw-command-line) (raw command line), [`.z.f`](#zf-file) (file), [`.z.q`](#zq-quiet-mode) (quiet mode), [`.Q.opt`](dotq.md#opt-command-parameters) (command parameters), [`.Q.def`](dotq.md#def-command-defaults) (command defaults), [`.Q.x`](dotq.md#x-non-command-parameters) (non-command parameters) ## `.z.Z` (local datetime) diff --git a/docs/ref/index.md b/docs/ref/index.md index f294ca568..8f336ca73 100644 --- a/docs/ref/index.md +++ b/docs/ref/index.md @@ -257,24 +257,23 @@ milliseconds: time mod 1000 ## Namespaces -### [`.h`](doth.md) +### [`.h`](doth.md) (markup) -Markup output for HTTP +HTTP, markup and data conversion. -### [`.j`](dotj.md) +### [`.j`](dotj.md) (JSON) -De/serialize as JSON +De/serialize as JSON. -### `.m` +### [`.m`](dotm.md) (memory backed files) -:fontawesome-regular-hand-point-right: -[Memory backed by files](dotm.md) +Memory backed by files. -### [`.Q`](dotq.md) +### [`.Q`](dotq.md) (utils) -Utilities: general, environment, IPC, datatype, database, partitioned database state, segmented database state, file I/O +Utilities: general, environment, IPC, datatype, database, partitioned database state, segmented database state, file I/O, debugging, profiling. -### [`.z`](dotz.md) +### [`.z`](dotz.md) (environment, callbacks) -System variables, callbacks +Environment, callbacks diff --git a/docs/ref/parse.md b/docs/ref/parse.md index 8460c46e2..0a975c561 100644 --- a/docs/ref/parse.md +++ b/docs/ref/parse.md @@ -120,4 +120,4 @@ q)views[] [Parse trees](../basics/parsetrees.md)
:fontawesome-regular-map: -[Parse trees and functional forms](../wp/parse-trees.md) +[Convert qSQL to functional forms using parse](../basics/parsetrees.md#conversion-using-parse) diff --git a/docs/releases/ChangesIn3.4.md b/docs/releases/ChangesIn3.4.md index 0418d69ab..301b64a08 100644 --- a/docs/releases/ChangesIn3.4.md +++ b/docs/releases/ChangesIn3.4.md @@ -27,7 +27,7 @@ Below is a summary of changes from V3.3. Commercially licensed users may obtain - can utilize the [snappy](http://google.github.io/snappy/) compression algorithm as algo \#3 for [File Compression](../kb/file-compression.md). - certain vector types can now be [updated efficiently](../ref/amend.md), directly on disk, rather than having to rewrite the whole file on change. - added async broadcast as [`-25!`(handles;msg)](../basics/internal.md#-25x-async-broadcast) which serializes the msg once, queuing it as async msg to each handle. -- [`parse`](../basics/parsetrees.md#parse) can now handle k in addition to q code. +- [`parse`](../basics/parsetrees.md) can now handle k in addition to q code. - `.Q.en` can now handle lists of sym vectors: [Enumerating nested varchar columns](../kb/splayed-tables.md#enumerating-nested-varchar-columns-in-a-table) ## Not upwardly compatible diff --git a/docs/wp/astronomy.md b/docs/wp/astronomy.md index a401efec9..38a8be786 100644 --- a/docs/wp/astronomy.md +++ b/docs/wp/astronomy.md @@ -253,22 +253,12 @@ We previously mentioned how astronomy data can often come with a time domain, wh We could apply an attribute to this data for optimization, such as the sorted attribute to the class column in the example data set. The benefits of this would become more apparent as more files were being loaded in. Ciaran Gorman’s [“Columnar database and query optimization”](columnar-database/index.md) gives an in-depth explanation as to how they can be applied. -## Conclusion - -In this paper, we have taken one sample of astronomy data that came in a format which was initially unfamiliar to us, and subsequently analyzed it in kdb+ with relative ease. We made use of the flexibility that kdb+ has in extending to other languages; in this case C, which effectively only required using the Dynamic Load operator. This approach can be replicated where C utilities have been created for interacting with data sources, or for other reasons. - -We were then able to build functionality to calculate the recessional velocity of different object classes, which took less than a tenth of a second to run on a table of one million rows. In doing this we made use of kdb+’s efficient `fby` syntax to make use of an aggregate function as a filter criteria. It is apparent that kdb+ is extremely performant, both in storing and analyzing this data. The nature and expected volumes of data, and the way it will be used, go hand in hand with kdb+ and we have found it to be an ideal fit. - -We believe this will be a very attractive solution to the field of astronomy going forward, coping with the predicted expansion of data volumes in the field due to the new projects that are scheduled to begin in the near future. - - ## Further reading - _The Atlantic_: [“How big data is changing astronomy (again)”](https://www.theatlantic.com/technology/archive/2012/04/how-big-data-is-changing-astronomy-again/255917/) - [Cloud Computing and the Square Kilometre Array](http://www.skatelescope.org/uploaded/8762_134_Memo_Newman.pdf) - ## Authors **Andrew Magowan** is a kdb+ consultant who has developed data and analytic systems for some of the world's largest financial institutions. Andrew is currently based in New York where he maintains a global tick capture application across a range of asset classes at a major investment bank. diff --git a/docs/wp/blockchain/index.md b/docs/wp/blockchain/index.md index 7718bfbc4..4e78a81da 100644 --- a/docs/wp/blockchain/index.md +++ b/docs/wp/blockchain/index.md @@ -972,22 +972,6 @@ time. ![](img/media/image28.png) -## Conclusion - -This paper described how a q process can be built to interact with a -Bitcoin full node to extract and store blockchain transaction data in -kdb+ format. By making use of appropriate schemas, partitioned database -structures and query performance techniques it was possible to create a -simple blockchain explorer process to look up transaction information and -perform some interesting blockchain analytics. This application of kdb+ -to the area of blockchain explorer technology is a further step toward -better understanding blockchain data, the unique challenges associated -with data retrieval and storage, and potential for application -development using the technology within the domain. - -[:fontawesome-solid-print:](/download/wp/blockchain-a4.pdf) - - ## Authors ![Daniel Irwin](../../img/faces/danielirwin.png) diff --git a/docs/wp/capi/index.md b/docs/wp/capi/index.md index 1982e855f..59d5a180d 100644 --- a/docs/wp/capi/index.md +++ b/docs/wp/capi/index.md @@ -1713,11 +1713,6 @@ int main() { } ``` -## Conclusion - -This document covered multiple aspects of the C API interface for connecting with the kdb+ database. Topics covered included the creation of socket connections, execution of queries, error handling, memory management, and the creation and extraction of data from `K` objects such as lists, dictionaries and tables. Practical examples formed the basis for the construction of a C subscriber process, capable of consuming a kdb+ data feed, and a feedhandler process designed to publish data to a kdb+ tickerplant. Finally, the use of shared C libraries to extend the functionality of kdb+ was also demonstrated. - - ## Author ![Jeremy Lucid](../../img/faces/jeremylucid.jpg) diff --git a/docs/wp/compress/index.md b/docs/wp/compress/index.md index 9410664f3..85e644c16 100644 --- a/docs/wp/compress/index.md +++ b/docs/wp/compress/index.md @@ -16,7 +16,7 @@ overall performance. Early versions of kdb+ achieved compression via file systems with inbuilt compression, such as ZFS. V2.7 introduced built-in OS-agnostic compression, which allowed on-disk data to be converted to compressed format using a range of algorithms and compression levels. This was expanded (V2.8), with the ability to stream in-memory data directly to compressed format on disk. ZFS compression is still useful for some kdb+ applications, as it keeps cached data available for multiple processes. However, this paper will focus on the inbuilt data-compression options provided by kdb+, available on all supported architectures. -Each system will have its own characteristics which determine the appropriate compression configurations to use. This paper will cover an introduction to compression in kdb+, a discussion of contributing factors to compression ratios and performance, and an analysis of how the use of compressed data can affect performance of some sample use cases. +Each system has its own characteristics, which determine the appropriate compression configurations to use. All tests were run using kdb+ version 3.1 (2013.09.05) diff --git a/docs/wp/data-loaders/index.md b/docs/wp/data-loaders/index.md index b9aa16976..2fbd53f7e 100644 --- a/docs/wp/data-loaders/index.md +++ b/docs/wp/data-loaders/index.md @@ -184,7 +184,7 @@ Once the orchestrator recognizes that a batch has arrived (through either file n !!! tip "What functions to apply to specific batches/files can be managed with configuration based on the filename." -### Step 1: Orchestrator distributes read and save task per file +#### Step 1: Orchestrator distributes read and save task per file Upon the orchestrator recognizing that a batch with its required files is available (files 1, 2, and 3), the following is executed within the `.mi.sendToFreeWorker` function. @@ -248,7 +248,7 @@ and updates in memory tables for tracking. ``` -### Step 2: Worker receives task to read and write a file +#### Step 2: Worker receives task to read and write a file The worker receives a run task command `.mi.runTask` from the orchestrator: @@ -332,7 +332,7 @@ Once the read and save task is complete, the tracked information i.e. the name o The column memory statistics are available so memory required for any further jobs on these columns could be estimated as part of distributing tasks to extend memory management. -### Step 3: Orchestrator appends to sym and sends next task +#### Step 3: Orchestrator appends to sym and sends next task After a success message is received from a worker via `.mi.workerResponse`, the orchestrator updates the tasks and workers tables. @@ -449,7 +449,7 @@ In order to maintain sym file integrity the following method is used to ensure a [Working with symfiles](../symfiles.md) -### Step 4: Indexing +#### Step 4: Indexing The orchestrator sends another `.mi.runTask` to index the data by the chosen sorting columns (`` `sym`time`` in this example) to an available worker. @@ -501,7 +501,7 @@ set[` sv mdb,`.d;key x`colSizes] ``` -###Step 5: Merge +#### Step 5: Merge Once the index task is complete, the orchestrator assigns each worker a distinct subset of columns to merge one by one based on the index created in Step 4. @@ -531,14 +531,14 @@ set[toPath;data] ``` -### Step 6: Move table/s +#### Step 6: Move table/s After receiving a callback message from each worker that the merge has been completed, a worker is assigned via `.mi.runTask` to move the table/s to the relevant HDB directory. The merged table is moved using system `mv` (`MOVE` on Windows) command and during the move, the main HDB can be temporarily locked from queries. Once the orchestrator receives a success message for the move task the batch is considered complete and the processing of the next batch can commence. -### Post-batch tasks +#### Post-batch tasks In order to reduce downtime between batches, each worker is killed and restarted by the orchestrator process after the batch. Killing workers and restarting them has been found to free up memory faster rather than each worker running garbage collection, which can be time-consuming. @@ -546,6 +546,7 @@ Once the batch successfully completes, any post-ingestion event-driven tasks can :fontawesome-regular-map: [Surveillance techniques to effectively monitor algo- and high-frequency trading](../surveillance/index.md)
+:fontawesome-regular-map: [Transaction-cost analysis using kdb+](../transaction-cost.md) @@ -582,18 +583,6 @@ Key elements and benefits of the proposed framework are: - Post-ingestion actions can be added – e.g. trigger the running regulatory reports, benchmarks or alerts -## Conclusion - -For use cases where real-time feeds are unfeasible (due to cost, technical limitations or time), this style of batch ingestion framework is a great solution which can handle and scale to ever increasing data volumes. The need to ingest batch data as fast as possible due to reporting, regulatory obligations, post-trade requirements or other business needs means kdb+ is perfectly suited for the task. - -In this paper we discussed batch processing and use cases where it may be most appropriate: high volumes and throughput, large number of files, hardware constraints. We then described a proposed mass ingestion framework using kdb+, discussing the technical aspects of the framework such as the number of workers to utilize, maintaining sym file integrity and task allocation. Finally, we went through an example ingestion scenario and concluded by outlining the benefits of the framework. - -:fontawesome-brands-github: -[kxcontrib/massIngestionDataloader](https://github.com/kxcontrib/massIngestionDataloader) - -The framework outlined in this paper is a simplified version of a framework that has been used in several KX implementations for batch ingestion. It has proven to be fast, efficient and scalable. - - ## Author ![Enda Gildea](../../img/faces/endeagildea.jpg) diff --git a/docs/wp/fix-messaging.md b/docs/wp/fix-messaging.md index 386b04beb..e81bb1ed1 100644 --- a/docs/wp/fix-messaging.md +++ b/docs/wp/fix-messaging.md @@ -571,19 +571,6 @@ OrderID SecurityID Side MsgType OrdStatus OrderQty CumQty AvgPx CommValue "00000005" VOD.L 2 8 2 3130 3130 229.7559 143.8272 ``` - -## Conclusion - -This paper has provided a guide to working with FIX messages in kdb+, focusing primarily on capturing messages from an OMS. We focused on some key FIX fields to provide an understanding of the valid entries and an insight into how they should be handled. - -:fontawesome-solid-globe: -[fixwiki.org/fixwiki](http://fixwiki.org/fixwiki/FIXwiki "fixwiki.org") -for a comprehensive list of all fields for each FIX version - -It is an essential requirement to be able to view the current and final state of each order received from the OMS. We provided an example to show how to generate an order state. This process can be extended to also derive the latest execution state. Additional fields need to be extracted to identity the execution type, amendments, cancellations etc, similar to the order state. An execution state enables complex analytics, including risk benchmarking and transaction cost analysis to be computed efficiently on every execution. - -All tests were run using kdb+ version 3.1 (2013.12.27) - ## Author **Damien Barker** is a financial engineer who has worked as a consultant for some of the world's largest financial institutions. Based in London, Damien is currently working on trading and analytics application at a US investment bank. diff --git a/docs/wp/gui/index.md b/docs/wp/gui/index.md index 0d03dd5ec..bd65059c9 100644 --- a/docs/wp/gui/index.md +++ b/docs/wp/gui/index.md @@ -841,38 +841,6 @@ more effort, nor would giving the user the ability to choose what period time they analyze. Furthermore, WebSockets could be used to deliver streaming data from the kdb+ back end to the C# GUI. - -## Conclusion - -Despite its popularity and the potential opportunities for -development, combining C# and kdb+ remains a relatively unexploited -area. We have demonstrated a few simple processes and applications -which could be useful in developing a trading application using kdb+ -and C#. - -The applications developed for this example are simple in nature and -in implementation but the potential for creating more sophisticated -and powerful tools exists. We have shown very limited analytical and -GUI elements but these could be expanded upon by the use of Cascading -Style Sheets and more extensive configuration. - -The paper has also demonstrated the potential versatility of kdb+, not -just in the banking and financial sectors but in all sectors where C# -is popular and a database is required for back-end data storage and -management. These examples could be pushed out to analytical or -performance-based sectors or markets inexperienced in kdb+ but -requiring tools to help utilize the rapidly growing Big Data -environment. - -All examples of kdb+ were run using version 3.2 (2015.01.14). All -tests of C# were run using .NET version 4.0. The example applications -were built with Visual Studio 2010. - -The C# source code for this paper -can be found on GitHub at :fontawesome-brands-github: -[kxcontrib/csharpgui](https://github.com/kxcontrib/csharpgui). - - ## Author **Michael Reynolds** works as a kdb+ consultant for one of the largest investment banks in the world. As part of his daily job, Michael is responsible for maintaining kdb+ databases as well as a C# APIs and plug-ins. diff --git a/docs/wp/index.md b/docs/wp/index.md index 9c8ed10fc..c59600e9b 100644 --- a/docs/wp/index.md +++ b/docs/wp/index.md @@ -73,7 +73,6 @@ White papers are flagged in the navigation menus. ## :fontawesome-solid-code: Programming in q - [**Iterators**](iterators/index.md)
Conor Slattery & Stephen Taylor, 2019.03 -- [**Parse trees and functional forms**](parse-trees.md)
Peter Storeng & Stephen Taylor, 2019.03 - [**kdb+ query scaling**](query-scaling.md)
Ian Lester, 2014.01 - [**The application of foreign keys and linked columns in kdb+**](foreign-keys.md)
Kevin Smyth, 2013.04 - [**Columnar database and query optimization**](columnar-database/index.md)
Ciáran Gorman, 2012.06 diff --git a/docs/wp/intraday-writedown/index.md b/docs/wp/intraday-writedown/index.md index 5465df2c4..c91030a3c 100644 --- a/docs/wp/intraday-writedown/index.md +++ b/docs/wp/intraday-writedown/index.md @@ -637,18 +637,6 @@ All cols, 3 syms | partWrite.q | ``select from quote where int in sym?`eurusd`e As can be seen in Table 3 and Figure 3, the speed-up in query times is substantial and thus may be worth the more complex storage and querying method. - -## Conclusion - -This paper has outlined two different methods for dealing with insufficient RAM on a server, meaning a full day’s worth of data cannot be held in memory. It looked at `w.q`, available from GitHub, which writes to a splayed table on disk and examined a number of customizations that could be made to it. - -It also detailed an alternative method using a partitioned temporary directory and how to query this data. The paper also compared the query speed and speed of the end-of-day process of the two solutions and discussed the reasons for their relative performance. - -It is important to point out that any of these solutions come with their own drawbacks and as such the best solution for your individual project will depend on the specific requirements. No one solution will provide the performance and ease of use equal to just maintaining a full day’s worth of data in memory. The complexity of how that data is stored and queried will inevitably increase. However, by using some or a combination of the ideas presented in this paper, a workable solution that fits your needs may be possible. - -Tests performed using kdb+ version 3.1 (2014.02.08) - - ## Author **Colm McCarthy** is a senior kdb+ consultant who has worked for leading investment banks across a number of different asset classes. diff --git a/docs/wp/iterators/index.md b/docs/wp/iterators/index.md index 5553a3437..e3a7d394b 100644 --- a/docs/wp/iterators/index.md +++ b/docs/wp/iterators/index.md @@ -1006,25 +1006,6 @@ BB 80.14611 CC 67.48254 ``` - -## Conclusion - -This white paper summarizes the q iterators, showing how they derive new functions from values – functions, file- and process handles, lists and dictionaries. - -It showed with examples how the effect of the iterator is determined sometimes by the rank of the applicable value, and sometimes also by the rank at which the derived function is applied. Even complicated examples, composing multiple iterators, can be analyzed to understand the behavior. - -Certain uses of iterators, such as the creation of recursive functions and applying iterators to functions within select statements, were examined in more detail, as these are often poorly understood, but useful in many situations. -Some common uses were looked at in to demonstrate the ability of iterators to reduce execution times. - -This white paper illustrates how iterators can be used easily to extend the -application of inbuilt and user-defined functions, allowing code to take full advantage of kdb+’s ability to process large volumes of data quickly. - -Using iterators correctly minimizes the amount of data manipulation and produces code that is concise and easier to maintain. - -All tests were run using kdb+ 3.6 (2018.10.23). - - - ## Authors **Conor Slattery** is a financial engineer who has designed kdb+ applications for a range of asset classes. Conor is currently working with a New York-based investment firm, developing kdb+ trading platforms for the US equity markets. diff --git a/docs/wp/lightning-tickerplants/index.md b/docs/wp/lightning-tickerplants/index.md index 8a5f54dd9..ffab4a201 100644 --- a/docs/wp/lightning-tickerplants/index.md +++ b/docs/wp/lightning-tickerplants/index.md @@ -1089,22 +1089,6 @@ invoice settlement messages back to individual devices to release data to subscr In this way, individual devices do not need to run their own node or store Bitcoin private keys. -## Conclusion - -The technology of Bitcoin and layer-two solutions like Lightning open up the possibility for applications -to interact directly with a decentralized peer-to-peer payments layer through the use of simple APIs, where the value -transfer reduces to the exchange of encoded text messages over TCP/IP. - -This ability to easily send and receive payments in a peer-to-peer fashion, especially micropayments, has the potential -to enable the construction of new innovative applications not hindered by third-party friction. - -In the tickerplant example, a simple template was provided to demonstrate how market data, or any other form of streaming data, -could be monetized with the creation of a pay-per-request system utilizing Lightning micropayments. - -While Lightning remains an experimental and rapidly changing technology, with many outstanding challenges, -it is hoped that this paper has at least helped explain some of the key concepts and techniques, and also showcased some synergies between the technology and kdb+ for potential integrations. - - ## Author ![Jeremy Lucid](../../img/faces/jeremylucid.jpg) @@ -1112,11 +1096,6 @@ it is hoped that this paper has at least helped explain some of the key concepts **Jeremy Lucid** is a kdb+ consultant, based in Ireland, who has worked on real-time Best Execution projects for a major multinational banking institution and a _Kx for Surveillance_ implementation at a leading options and futures exchange. - - ## Acknowledgments diff --git a/docs/wp/multi-thread/index.md b/docs/wp/multi-thread/index.md index be2dfba94..64c9e503d 100644 --- a/docs/wp/multi-thread/index.md +++ b/docs/wp/multi-thread/index.md @@ -479,8 +479,6 @@ Used incorrectly, parallel processing can be less efficient than the equivalent The use of multi-threading in kdb+ should therefore be treated on a case-by-case basis. It is a feature which should often be considered by developers, but performance testing is recommended before implementation. -System-management solutions are available to assist with administering multi-process environments, like [KX Control](../../devtools.md#kx-control) which provides a range of tools for visualizing process workflow, task scheduling and system-resource monitoring. These features are especially helpful at managing primary/replica configurations, where client queries or the loading of data is divided between processes in order to increase throughput. - All tests performed using kdb+ version 3.1 (2013.06.25) diff --git a/docs/wp/option-pricing/index.md b/docs/wp/option-pricing/index.md index 642c07c24..e29b4a3cb 100644 --- a/docs/wp/option-pricing/index.md +++ b/docs/wp/option-pricing/index.md @@ -682,9 +682,6 @@ In this paper we demonstrated that it is possible to calculate option prices usi Looking at the results produced, it is clear that both the option price produced and the resulting RMSE/log RMSE converged fastest when compared with the Black-Scholes price for the Quasi-Monte Carlo approach, with Sobol’ sequence number generation and Brownian-bridge construction. -Additionally, by plotting results we have shown that the q implementation replicates the original results produced in C++, presented in the paper by [S. Kucherenko et al. 2007](http://www.broda.co.uk/gsa/wilmott_GSA_SK.pdf "Wilmott"). - - ## Author **Deanna Morgan** joined First Derivatives in June 2018 as a data scientist in the Capital Markets Training Program and currently works as a machine-learning engineer in London. diff --git a/docs/wp/parse-trees.md b/docs/wp/parse-trees.md deleted file mode 100644 index 35b9b2cc3..000000000 --- a/docs/wp/parse-trees.md +++ /dev/null @@ -1,700 +0,0 @@ ---- -title: Parse trees and functional forms | kdb+ and q documentation -description: How to understand parse trees and use functional forms in q queries; how to convert qSQL expressions to functional form. -author: [Peter Storeng, Stephen Taylor] -keywords: functional, kdb+, parse, parse tree, q, qSQL, query, SQL ---- -# Parse trees and functional forms - -by [Peter Storeng & Stephen Taylor](#authors) -{: .wp-author} - - -The importance of understanding and using the functional form of qSQL statements in kdb+ cannot be overstated. The functional form has many advantages over the qSQL approach, including the ability to select columns and build Where clauses dynamically. It is important for any q programmer to understand the functional form fully and how to convert to it from qSQL. - -Applying `parse` to a qSQL statement written as a string will return the internal representation of the functional form. With some manipulation this can then be used to piece together the functional form in q. This generally becomes more difficult as the query becomes more complex and requires a deep understanding of what kdb+ is doing when it parses qSQL form. - -The main goal of this paper is to show in detail how this conversion works, so that it is understood how to build the functional form of qSQL statements correctly. In order to do this, we will need to look at how q and k commands relate to each other, as the parse function often returns the cryptic k code for functions. An understanding of what parse trees are, and how to use them, is also vital in the building of functional queries. - -Finally, this paper will look at creating a function which will automate the process of converting qSQL statements into functional form. This is to be used as a helpful development tool when facing the task of writing a tricky functional statement. - -All tests were run using kdb+ version 3.2 (2015.01.14). - - -## k4, q and `q.k` - -kdb+ is a database management system which ships with the general-purpose and database language q. Q is an embedded domain-specific language implemented in the k programming language, sometimes known as k4. The q interpreter can switch between q and k modes and evaluate expressions written in k as well as q. - -The k language is for KX implementors. -It is not documented or supported for use outside KX. -All the same functionality is available in the much more readable q language. However in certain cases, such as debugging, a basic understanding of some k syntax can be useful. - -The `q.k` file is part of the standard installation of q and loads into each q session on startup. It defines many of the q keywords in terms of k. To see how a q keyword is defined in terms of k we could check the `q.k` file or simply enter it into the q prompt: - -```q -q)key -!: -``` - -A few q keywords are defined natively from C and do not have a k representation: - -```q -q)like -like -``` - - -## The `parse` keyword - -`parse` is a useful tool for seeing how a statement in q is evaluated. Pass the `parse` keyword a q statement as a string and it will return the parse tree of that expression. - - -### Parse trees - -A parse tree is a q construct which represents an expression but which is not immediately evaluated. It takes the form of a list where the first item is a function and the remaining items are the arguments. Any of the items of the list can be parse trees themselves. - -Note that in a parse tree a variable is represented by a symbol containing its name. Thus to distinguish a symbol or a list of symbols from a variable it is necessary to enlist that expression. When we apply the `parse` function to create a parse tree, explicit definitions in `.q` are shown in their full k form. In particular, an enlisted element is represented by a preceding comma. - -```q -q)parse"5 6 7 8 + 1 2 3 4" -+ //the function/operator -5 6 7 8 //first argument -1 2 3 4 //second argument -``` -```q -q)parse"2+4*7" -+ //the function/operator -2 //first argument -(*;4;7) //second argument, itself a parse tree -``` -```q -q)v:`e`f -q)`a`b`c,`d,v -`a`b`c`d`e`f -q)parse"`a`b`c,`d,v" -, // join operator -,`a`b`c //actual symbols/lists of symbols are enlisted -(,;,`d;`v) //v a variable represented as a symbol -``` - -We can also manually construct a parse tree: - -```q -q)show pTree:parse "(aggr;data) fby grp" -k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} //fby in k form -(enlist;`aggr;`data) -`grp - -q)pTree~(fby;(enlist;`aggr;`data);`grp) //manually constructed -1b //parse tree -``` - -As asserted previously every statement in q parses into the form: - -```txt -(function; arg 1; …; arg n) -``` - -where every item could itself be a parse tree. In this way we see that every action in q is essentially a function evaluation. - - -### `eval` and `value` - -`eval` can be thought of as the dual to `parse`. The following holds for all valid q statements (without side effects) put into a string. (Recall that `value` executes the command inside a string.) - -```q -//a tautology (for all valid q expressions str) -q)value[str]~eval parse str -1b -q)value["2+4*7"]~eval parse"2+4*7" //simple example -1b -``` - -When passed a list, `value` applies the first item (which contains a function) to the rest of the list (the arguments). - -```q -q)function[arg 1;..;arg n] ~ value(function;arg 1;..;arg n) -1b -``` - -When `eval` and `value` operate on a parse tree with no nested parse trees they return the same result. However it is not true that `eval` and `value` are equivalent in general. `eval` operates on parse trees, evaluating any nested parse trees, whereas `value` operates on the literals. - -```q -q)value(+;7;3) //parse tree, with no nested trees -10 -q)eval(+;7;3) -10 -q)eval(+;7;(+;2;1)) //parse tree with nested trees -10 -q)value(+;7;(+;2;1)) -'type -``` -```q -q)value(,;`a;`b) -`a`b -q)eval(,;`a;`b) //no variable b defined -'b -q)eval(,;enlist `a;enlist `b) -`a`b -``` - - -### Variadic operators - -Many operators and some keywords in k and q are [variadic](../basics/glossary.md#variadic): they are overloaded so that the behavior of the operator changes depending on the number and type of arguments. In q (not k) the unary form of operators such as (`+`, `$`, `.`, `&` etc.) is disabled: keywords are provided instead. - -For example, in k the unary form of the `$` operator equates to the `string` -keyword in q. - -```q -q)k)$42 -"42" -q)$42 //$ unary form disabled in q -'$ -q)string 42 -"42" -``` - -!!! info "A parenthesized variadic function applied prefix is parsed as its unary form." - -```q -q)($)42 -"42" -``` - -A familiar example of a variadic function is the Add Over function `+/` derived by applying the Over iterator to the Add operator. - -```q -q)+/[1000;2 3 4] // +/ applied binary -1009 -q)+/[2 3 4] // +/ applied unary -9 -q)(+/)2 3 4 // +/ applied unary -9 -``` - -In k, the unary form of an operator can also be specified explicitly by suffixing it with a colon. - -```q -q)k)$:42 -"42" -``` - -`+:` is a unary operator; the unary form of `+`. We can see this in the parse tree: - -```q -q)parse"6(+)4" -6 -(+:;4) -``` - -The items of a `parse` result use k syntax. Since (most of) the q keywords are defined in the `.q` namespace, we can use dictionary reverse lookup to find the meaning. - -```q -q).q?(+:) -`flip -``` - -So we can see that in k, the unary form of `+` corresponds to `flip` in q. - -```q -q)d:`c1`c2`c3!(1 2;3 4;5 6) -q)d -c1| 1 2 -c2| 3 4 -c3| 5 6 -q)k)+d -c1 c2 c3 --------- -1 3 5 -2 4 6 -q)k)+:d -c1 c2 c3 --------- -1 3 5 -2 4 6 -``` - -!!! warning "Exposed infrastructure" - - The unary forms of operators are [exposed infrastructure](). - Their use in q expressions is **strongly discouraged**. - Use the corresponding q keywords instead. - - For example, write `flip d` rather than `(+:)d`. - - The unary forms are reviewed here to enable an understanding of parse trees, in which k syntax is visible. - - - -When using reverse lookup on the `.q` context we are slightly hampered by the fact that it is not an injective mapping. The Find `?` operator returns only the first q keyword matching the k expression. In some cases there is more than one. Instead use the following function: - -```q -q)qfind:{key[.q]where x~/:string value .q} - -q)qfind"k){x*y div x:$[16h=abs[@x];\"j\"$x;x]}" -,`xbar -q)qfind"~:" -`not`hdel -``` - -We see `not` and `hdel` are equivalent. Writing the following could be confusing: - -```q -q)hdel 01001b -10110b -``` - -So q provides two different names for clarity. - - -### Iterators as higher-order functions - -An iterator applies to a value (function, list, or dictionary) to produce a related function. This is again easy to see by inspecting the parse tree: - -```q -q)+/[1 2 3 4] -10 -q)parse "+/[1 2 3 4]" -(/;+) -1 2 3 4 -``` - -The first item of the parse tree is `(/;+)`, which is itself a parse -tree. We know the first item of a parse tree is to be applied to the -remaining items. Here `/` (the Over iterator) is applied to `+` to -produce a new function which sums the items of a list. - -:fontawesome-regular-hand-point-right: -[Iterators](iterators/index.md) - - -## Functional queries - -Alongside each qSQL query we also have the equivalent functional -forms. These are especially useful for programmatically-generated -queries, such as when column names are dynamically queried. - -```q -?[t;c;b;a] // select and exec -![t;c;b;a] // update and delete -``` - -Here - -- `t` is a table -- `c` is a list of constraints in the portable parse tree format -- `b` is a dictionary of group-bys -- `a` is a dictionary of aggregates - -:fontawesome-regular-hand-point-right: -_Q for Mortals_: [§9.12 Functional forms of queries](/q4m3/9_Queries_q-sql/#912-functional-forms) - -The q interpreter parses the syntactic forms of `select`, `exec`, `update` -and `delete` into their equivalent functional forms. Therefore there is -no performance difference between a qSQL query and a functional -one. - - -### Issues converting to functional form - -To convert a `select` query to a functional form one may attempt to -apply the `parse` function to the query string: - -```q -q)parse "select sym,price,size from trade where price>50" -? -`trade -,,(>;`price;50) -0b -`sym`price`size!`sym`price`size -``` - -As we know, `parse` produces a parse tree and since some of the elements may themselves be parse trees we can’t immediately take the output of parse and plug it into the form `?[t;c;b;a]`. After a little playing around with the result of `parse` you might eventually figure out that the correct functional form is as follows. - -```q -q)funcQry:?[`trade;enlist(>;`price;50);0b;`sym`price`size! `sym`price`size] - -q)strQry:select sym,price,size from trade where price>50 q) -q)funcQry~strQry -1b -``` - -This, however, becomes more difficult as the query statements become more complex: - -```q -q)parse "select count i from trade where 140>(count;i) fby sym" -? -`trade -,,(>;140;(k){@[(#y)#x[0]0#x -1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) -0b -(,`x)!,(#:;`i) -``` - -In this case, it is not obvious what the functional form of the above query should be, even after applying `parse`. - -There are three issues with this parse-and-“by eye” method to convert to the equivalent functional form. We will cover these in the next three subsections. - - -### Parse trees and eval - -The first issue with passing a `select` query to `parse` is that each returned item is in unevaluated form. As [discussed above](#eval-and-value), simply applying `value` to a parse tree does not work. However, if we evaluate each one of the arguments fully, then there would be no nested parse trees. We could then apply `value` to the result: - -```q -q)eval each parse "select count i from trade where 140>(count;i) fby sym" -? -+`sym`time`price`size!(`VOD`IBM`BP`VOD`IBM`IBM`HSBC`VOD`MS.. -,(>;140;(k){@[(#y)#x[0]0#x -1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) -0b -(,`x)!,(#:;`i) -``` - -The equivalence below holds for a general qSQL query provided as a string: - -```q -q)value[str]~value eval each parse str -1b -``` - -In particular: - -```q -q)str:"select count i from trade where 140>(count;i) fby sym" - -q)value[str]~value eval each parse str -1b -``` - -In fact, since within the functional form we can refer to the table by name we can make this even clearer. Also, the first item in the result of `parse` applied to a `select` query will always be `?` (or `!` for a `delete`or `update` query) which cannot be evaluated any further. So we don’t need to apply `eval` to it. - -```q -q)pTree:parse str:"select count i from trade where 140>(count;i) fby sym" -q)@[pTree;2 3 4;eval] -? -`trade -,(>;140;(k){@[(#y)#x[0]0#x -1;g;:;x[0]'x[1]g:.=y]};(enlist;#:;`i);`sym)) -0b -(,`x)!,(#:;`i) -q)value[str] ~ value @[pTree;2 3 4;eval] -1b -``` - - -### Variable representation in parse trees - -Recall that in a parse tree a variable is represented by a symbol containing its name. So to represent a symbol or a list of symbols, you must use `enlist` on that expression. In k, `enlist` is the unary form of the comma operator in k: - -```q -q)parse"3#`a`b`c`d`e`f" -# -3 -,`a`b`c`d`e`f -q)(#;3;enlist `a`b`c`d`e`f)~parse"3#`a`b`c`d`e`f" -1b -``` - -This causes a difficulty. As [discussed above](#variadic-operators), q has no unary syntax for operators. - -Which means the following isn’t a valid q expression and so returns an error. - -```q -q)(#;3;,`a`b`c`d`e`f) -', -``` - -In the parse tree we receive we need to somehow distinguish between k’s unary `,` (which we want to replace with `enlist`) and the binary Join operator, which we want to leave as it is. - - -### Explicit definitions in `.q` are shown in full - -The `fby` in the `select` query above is represented by its full k -definition. - -```q -q)parse "fby" -k){@[(#y)#x[0]0#x 1;g;:;x[0]'x[1]g:.=y]} -``` - -While using the k form isn’t generally a problem from a functionality perspective, it does however make the resulting functional statement difficult to read. - - -## The solution - -We will write a function to automate the process of converting a `select` query into its equivalent functional form. - -This function, `buildQuery`, will return the functional form as a string. - -```q -q)buildQuery "select count i from trade where 140>(count;i) fby sym" -"?[trade;enlist(>;140;(fby;(enlist;count;`i);`sym));0b; - (enlist`x)! enlist (count;`i)]" -``` - -When executed it will always return the same result as the `select` query from which it is derived: - -```q -q)str:"select count i from trade where 140>(count;i) fby sym" -q)value[str]~value buildQuery str -1b -``` - -And since the same logic applies to `exec`, `update` and `delete` it will be able to convert to their corresponding functional forms also. - -To write this function we will solve the three issues outlined above: - -1. parse-tree items may be parse trees -2. parse trees use k’s unary syntax for operators -3. q keywords from `.q.` are replaced by their k definitions - -The first issue, where some items returned by `parse` may themselves be parse trees is easily resolved by applying `eval` to the individual items. - -The second issue is with k’s unary syntax for `,`. We want to replace it with the q keyword `enlist`. To do this we define a function that traverses the parse tree and detects if any element is an enlisted list of symbols or an enlisted single symbol. If it finds one we replace it with a string representation of `enlist` instead of `,`. - -```q -ereptest:{ //returns a boolean - (1=count x) and ((0=type x) and 11=type first x) or 11=type x} -ereplace:{"enlist",.Q.s1 first x} -funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} -``` - -Before we replace the item we first need to check it has the -correct form. We need to test if it is one of: - -- An enlisted list of syms. It will have type `0h`, count 1 and the type of its first item will be `11h` if and only if it is an enlisted list of syms. -- An enlisted single sym. It will have type `11h` and count 1 if and only if it is an enlisted single symbol. - -The `ereptest` function above performs this check, with `ereplace` performing the replacement. - -!!! tip "Console size" - - `.Q.s1` is dependent on the size of the console so make it larger if necessary. - -Since we are going to be checking a parse tree which may contain parse trees nested to arbitrary depth, we need a way to check all the elements down to the base level. - -We observe that a parse tree is a general list, and therefore of type `0h`. This knowledge combined with the use of `.z.s` allows us to scan a parse tree recursively. The logic goes: if what you have passed into `funcEn` is a parse tree then reapply the function to each element. - -To illustrate we examine the following `select` query. - -```q -q)show pTree:parse "select from trade where sym like \"F*\",not sym=`FD" -? -`trade -,((like;`sym;"F*");(~:;(=;`sym;,`FD))) 0b -() - -q)x:eval pTree 2 //apply eval to Where clause -``` - -Consider the Where clause in isolation. - -```q -q)x //a 2-list of Where clauses -(like;`sym;"F*") -(~:;(=;`sym;,`FD)) - -q)funcEn x -(like;`sym;"F*") -(~:;(=;`sym;"enlist`FD")) -``` - -Similarly we create a function which will replace k functions with -their q equivalents in string form, thus addressing the third issue above. - -```q -q)kreplace:{[x] $[`=qval:.q?x;x;string qval]} -q)funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} -``` - -Running these functions against our Where clause, we see the k -representations being converted to q. - -```q -q)x -(like;`sym;"F*") -(~:;(=;`sym;,`FD)) - -q)funcK x //replaces ~: with “not” -(like;`sym;"F*") -("not";(=;`sym;,`FD)) -``` - -Next, we make a slight change to `kreplace` and `ereplace` and combine them. - -```q -kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} -ereplace:{"~~enlist",(.Q.s1 first x),"~~"} -q)funcEn funcK x -(like;`sym;"F*") ("~~not~~";(=;`sym;"~~enlist`FD~~")) -``` - -The double tilde here is going to act as a tag to allow us to differentiate from actual string elements in the parse tree. This allows us to drop the embedded quotation marks at a later stage inside the `buildQuery` function: - -```q -q)ssr/[;("\"~~";"~~\"");("";"")] .Q.s1 funcEn funcK x -"((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)))" -``` - -thus giving us the correct format for the Where clause in a functional select. By applying the same logic to the rest of the parse tree we can write the `buildQuery` function. - -```q -q)buildQuery "select from trade where sym like \"F*\",not sym=`FD" -"?[trade;((like;`sym;\"F*\");(not;(=;`sym;enlist`FD)));0b;()]" -``` - -One thing to take note of is that since we use reverse lookup on the `.q` namespace and only want one result we occasionally get the wrong keyword back. - -```q -q)buildQuery "update tstamp:ltime tstamp from z" -"![z;();0b;(enlist`tstamp)!enlist (reciprocal;`tstamp)]" - -q).q`ltime -%: -q).q`reciprocal -%: -``` - -These instances are rare and a developer should be able to spot when they occur. Of course, the functional form will still work as expected but could confuse readers of the code. - - -### Fifth and sixth arguments - -Functional select also has ranks 5 and 6; i.e. fifth and sixth arguments. - -:fontawesome-regular-hand-point-right: -_Q for Mortals_: [§9.12.1 Functional queries](/q4m3/9_Queries_q-sql/#9121-functional-select) - -We also cover these with the `buildQuery` function. - -```q -q)buildQuery "select[10 20] from trade" -"?[trade;();0b;();10 20]" -q)//5th parameter included -``` - -The 6th argument is a column and a direction to order the results by. Use `<` for ascending and `>` for descending. - -```q -q)parse"select[10;:) -`hopen`hclose - -q)qfind each ("<:";">:") //qfind defined above -hopen -hclose -``` - -We see that the k function for the 6th argument of the functional form is `<:` (ascending) or `>:` (descending). At first glance this appears to be `hopen` or `hclose`. In fact in earlier versions of q, `iasc` and `hopen` were equivalent (as were `idesc` and `hclose`). The definitions of `iasc` and `idesc` were later altered to signal a rank error if not applied to a list. - -```q -q)iasc -k){$[0h>@x;'`rank;@x;'`rank;>x]} - -q)iasc 7 -'rank -``` - -Since the columns of a table are lists, it is irrelevant whether the functional form uses the old or new version of `iasc` or `idesc`. - -The `buildQuery` function handles the 6th argument as a special case so will produce `iasc` or `idesc` as appropriate. - -```q -q)buildQuery "select[10 20;>price] from trade" -"?[trade;();0b;();10 20;(idesc;`price)]" -``` - -The full `buildQuery` function code is provided in the Appendix. - - -## Conclusion - -This paper has investigated how statements in q are evaluated by the parser. To do this we examined the relationship between q and k, and explained why parse trees generated by the parse function contain k code. We have then used this understanding to explain how qSQL statements can be converted into their equivalent and more powerful functional form. - -To further help those learning how to construct functional statements, and also as a useful development tool, we have written a function which will correctly convert a qSQL query into the q functional form. This would be useful for checking that a functional statement is formed correctly or to help with a particularly tricky query. - -All tests were run using kdb+ version 3.2 (2015.01.14). - - -## Authors - -**Peter Storeng** is a mathematician and software developer who has been using kdb+ since 2012. - -![Stephen Taylor](../img/faces/stephentaylor.png) -{: .small-face} - -**Stephen Taylor** FRSA has followed the evolution of the Iversonian languages through APL, J, k, and q, and is a former editor of [_Vector_](https://vector.org.uk), the journal of the British APL Association. - -## Appendix - -```q -\c 30 200 -tidy:{ssr/[;("\"~~";"~~\"");("";"")] $[","=first x;1_x;x]} -strBrk:{y,(";" sv x),z} - -//replace k representation with equivalent q keyword -kreplace:{[x] $[`=qval:.q?x;x;"~~",string[qval],"~~"]} -funcK:{$[0=t:type x;.z.s each x;t<100h;x;kreplace x]} - -//replace eg ,`FD`ABC`DEF with "enlist`FD`ABC`DEF" -ereplace:{"~~enlist",(.Q.s1 first x),"~~"} -ereptest:{(1=count x) and ((0=type x) and 11=type first x) or 11=type x} -funcEn:{$[ereptest x;ereplace x;0=type x;.z.s each x;x]} - -basic:{tidy .Q.s1 funcK funcEn x} - -addbraks:{"(",x,")"} - -//Where clause needs to be a list of Where clauses, -//so if only one Where clause, need to enlist. -stringify:{$[(0=type x) and 1=count x;"enlist ";""],basic x} - -//if a dictionary, apply to both keys and values -ab:{ - $[(0=count x) or -1=type x; .Q.s1 x; - 99=type x; (addbraks stringify key x ),"!",stringify value x; - stringify x] } - -inner:{[x] - idxs:2 3 4 5 6 inter ainds:til count x; - x:@[x;idxs;'[ab;eval]]; - if[6 in idxs;x[6]:ssr/[;("hopen";"hclose");("iasc";"idesc")] x[6]]; - //for select statements within select statements - x[1]:$[-11=type x 1;x 1;[idxs,:1;.z.s x 1]]; - x:@[x;ainds except idxs;string]; - x[0],strBrk[1_x;"[";"]"] } - -buildQuery:{inner parse x} -``` diff --git a/docs/wp/permissions/index.md b/docs/wp/permissions/index.md index c934cd7da..072a6cb73 100644 --- a/docs/wp/permissions/index.md +++ b/docs/wp/permissions/index.md @@ -11,7 +11,9 @@ by [Tom Martin](#author) {: .wp-author} -Due to its efficiency in storing and retrieving large volumes of data, kdb+ is the data-storage technology of choice for many financial institutions. kdb+ processes thus often contain sensitive, proprietary information in the form of data or proprietary code and so it is important to restrict who can and cannot access this information. kdb+ offers a number of in-built access functions, though in a default kdb+ instance these are not activated. This paper discusses various methods in which a permissioning and entitlements system can be implemented in kdb+ by extending these in-built functions, allowing access to sensitive information to be controlled and restricted, exposing data to some clients but not to others. +kdb+ processes often contain sensitive, proprietary information in the form of data or proprietary code. Thus, it is important to restrict the access to this information. + +kdb+ offers a number of in-built access functions. This paper discusses various methods in which a permissioning and entitlements system can be implemented in kdb+ by extending these in-built functions, allowing access to sensitive information to be controlled and restricted, exposing data to some clients but not to others. !!! tip "Commercial-grade products" @@ -1092,7 +1094,9 @@ poweruser1 "select from quote" 1 "" ## Conclusion -This paper was an introduction to permissioning in kdb+ without using LDAP or any other external entitlements system. As kdb+ is the market leader in the storage and retrieval of big data, it is the database technology of choice for the world’s top financial institutions and as such it is used to store sensitive, proprietary information. In order to pass a security audit, access to this data should be controlled and logged to ensure that only those who are entitled to view the information are able to do so. +This paper was an introduction to permissioning in kdb+ without using LDAP or any other external entitlements system. + +In order to pass a security audit, access to this data should be controlled and logged to ensure that only those who are entitled to view the information are able to do so. We have described a number of methods of securing a kdb+ process. We examined the concept of splitting clients into separate groups or classes, each with different permission levels. We examined how to block write access on a kdb+ process, and how to restrict certain users from viewing proprietary code. While the system described in the paper offers a broad coverage, including blocking some forms of code injection, it is not intended to be complete. diff --git a/docs/wp/socket-sharding/index.md b/docs/wp/socket-sharding/index.md index 390781647..f2a92b2fe 100644 --- a/docs/wp/socket-sharding/index.md +++ b/docs/wp/socket-sharding/index.md @@ -329,9 +329,7 @@ The reconnect attempt immediately connected resulting in minimal downtime. ## Conclusion -In this white paper, we demonstrated the set-up and use of socket sharding in kdb+ as well as investigating several example scenarios. - -From results generated throughout this paper, it is evident that there are both pros and cons of using socket sharding in kdb+. If processes listening on the sharded socket are all running with the same performance, socket sharding can reduce the response time for requests to clients and processing load on a listener process. +It is evident that there are both pros and cons of using socket sharding in kdb+. If processes listening on the sharded socket are all running with the same performance, socket sharding can reduce the response time for requests to clients and processing load on a listener process. As the processes are all listening on the same port, clients do not need configuration changes in order for connection requests to be assigned to new servers. diff --git a/docs/wp/surveillance-latency/index.md b/docs/wp/surveillance-latency/index.md index 68a7a667b..ed5ad0c31 100644 --- a/docs/wp/surveillance-latency/index.md +++ b/docs/wp/surveillance-latency/index.md @@ -330,7 +330,7 @@ As time progresses up to 12 hours, we can see that runtime memory usage fluctuat These results show that the use of event ID windows instead of timestamp windows causes a marginal but acceptable hit on the performance of the window-join approach. -## Conclusions +## Conclusion There are other factors that we have not experimented with – the distribution of the twenty runs of the alert analytics out to different secondary threads, implementing intraday memory management techniques in the alert engines or varying the hard-coded lookback threshold of five minutes on top of varying the intraday execution point frequency. diff --git a/docs/wp/symfiles.md b/docs/wp/symfiles.md index b214f87da..a6c5fd1c9 100644 --- a/docs/wp/symfiles.md +++ b/docs/wp/symfiles.md @@ -742,28 +742,6 @@ system"mv sym zym" /make backup of sym file } peach symFiles ``` - -## Conclusion - -While the q language provides tools to easily create and update HDBs -through functions in the `.Q` namespace, understanding how the underlying -code and concepts work is integral to creating and maintaining a -functioning database. - -In this paper, we demonstrated, through examples, enumeration in kdb+ -and various methods of using symbols within an HDB; using multiple sym -files and moving data between HDBs. In situations where you need new -columns `dbmaint.q` provides utility functions to manage this with ease. - -We discussed datatype considerations in order to utilize the performance -benefits of enumerations while maintaining a sym universe containing -only necessary symbols. However, if there is still a need to perform -maintenance on the sym file because of bloat we walked through two -methods to reduce the size of the sym file. - -All tests performed using kdb+ version 3.6 (2018.05.17). - - ## Author ![Paula Clarke](../img/faces/paulaclarke.jpg) @@ -771,7 +749,6 @@ All tests performed using kdb+ version 3.6 (2018.05.17). **Paula Clarke** is a senior kdb+ consultant for KX who has worked for some of the world’s largest financial institutions designing, developing and maintaining global tick-capture kdb+ systems. - ## Related articles :fontawesome-regular-map: diff --git a/mkdocs.yml b/mkdocs.yml index b3c7a9f5b..6bb8fc03e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -419,7 +419,6 @@ nav: - Metadata: basics/metadata.md - Namespaces: basics/namespaces.md - Parse trees: basics/parsetrees.md - - Parse trees, functional SQL: wp/parse-trees.md - qSQL: - qSQL queries: basics/qsql.md - Functional qSQL: basics/funsql.md