Merge pull request #325 from sshanks-kx/refactor

refactor

docs/architecture/index.md

@@ -29,14 +29,16 @@ KX’s [Fusion interfaces](../interfaces/index.md#fusion-interfaces) connect kdb
 
 ### Tickerplant (TP)
 
-A kdb+ processing acting as a TP (tickerplant) captures the initial data feed, writes it to the log file and [publishes](../kb/publish-subscribe.md) these messages to any registered subscribers.
+A kdb+ processing acting as a TP (tickerplant) captures the initial data feed, writes it to the log file and publishes these messages to any registered subscribers.
 Aims for zero-latency.
 Includes ingesting data in batch mode.
 
 Manages subscriptions: adds and removes subscribers, and sends subscriber table definitions.
 
 Handles end-of-day (EOD) processing.
 
+[`tick.q`](tickq.md) represents a tickerplant and is provided as a starting point for most environments.
+
 !!! tip "Best practices for tickerplants"
 
     Tickerplants should be lightweight, not capturing data and using very little memory. 
@@ -67,6 +69,8 @@ At startup, the RDB sends a message to the tickerplant and receives a reply cont
 
 At end of day usually writes intraday data to the Historical Database, and sends it a new EOD message.
 
+[`r.q`](rq.md) represents a tickerplant and is provided as a starting point for most environments.
+
 !!! tip "Best practices for real-time databases"
 
     RDBs queried intraday should exploit attributes in their tables. For example, a trade table might be marked as sorted by time (`` `s#time``) and grouped by sym (`` `g#sym``).
@@ -157,26 +161,3 @@ Can connect both the real-time and historical data to allow users to query acros
     [Query Routing: A kdb+ framework for a scalable, load balanced system](../wp/query-routing/index.md)
 
 
-## :fontawesome-solid-hand-point-right: What next?
-
-:fontawesome-regular-map:
-[Building real-time tick subscribers](../wp/rt-tick/index.md)
-<br>
-:fontawesome-regular-map:
-[Data recovery for kdb+ tick](../wp/data-recovery.md)
-<br>
-:fontawesome-regular-map:
-[Disaster-recovery planning for kdb+ tick systems](../wp/disaster-recovery/index.md)
-<br>
-:fontawesome-regular-map:
-[Intraday writedown solutions](../wp/intraday-writedown/index.md)
-<br>
-:fontawesome-regular-map:
-[Query routing: a kdb+ framework for a scalable load-balanced system](../wp/query-routing/index.md)
-<br>
-:fontawesome-regular-map:
-[Order book: a kdb+ intraday storage and access methodology](../wp/order-book.md)
-<br>
-:fontawesome-regular-map:
-[kdb+tick profiling for throughput optimization](../wp/tick-profiling.md)
-

docs/architecture/rq.md

@@ -0,0 +1,126 @@
+---
+title: RBB (r.q) | Documentation for q and kdb+
+description: How to construct an RDB
+keywords: kdb+, q, rdb, streaming
+---
+# Real-time Database (RDB) using r.q
+
+`r.q` is available from :fontawesome-brands-github:[KxSystems/kdb-tick](https://github.com/KxSystems/kdb-tick)
+
+## Overview
+
+A kdb+ process acting as an RDB stores a current day’s data in-memory for client queries.
+It can write its contents to disk at end-of-day, clearing out it in-memory data to prepare for the next day.
+After writing data to disk, it instructs a HDB to load the written data.
+
+### Customization
+
+`r.q` provides a starting point to most environments. The source code is freely avaialble and can be tailered to individual needs. For example:
+
+#### Memory use
+
+The default RDB stores all of a days data in memory before end-of-day writing to disk. The host machines should be configured that all required resources 
+can handle the demands that may be made of them (both for today and the future).
+Depending on when there may be periods of low/no activity, [garbage collection](../ref/dotq.md#gc-garbage-collect) could be deployed after clearing tables at end-of-day, or a system for intra-day writedowns.
+
+#### User queries
+
+A gateway process should control user queries and authorization/authentication, using RDBs/RTEs/HDBs to retrieve the required information.
+If known/common queries can be designed, the RDB can load additional scripts to pre-define functions a gateway can call.
+
+### End-of-day
+
+The end-of-day event is governed by the tickerplant process. The tickerplant calls the RDB [`.u.end`](#uend) function when this event occurs.
+The main end-of-day event for an RDB is to save todays data from memory to disk, clear its tables and cause the HDB to be aware of a new days dataset for it to access.
+
+!!! Note "[.u.rep](#urep) sets the HDB directory be the same as the tickerplant log file directory. This can be edited to use a different directory if required"
+
+### Recovery
+
+Using IPC, the RDB process can retrieve the current tickerplant log location and use via the [variables](tickq.md#variables) the tickerplant maintains.
+The function [`.u.rep`](#urep) is then used to populate any tables from the log.
+
+!!! Note "The RDB should be able to access the tickerplant log from a directory on the same machine. The RDB/tickerplant can be changed to reside on different hosts but this increases the resources needed to transmit the log file contents over the network."
+
+## Usage
+
+```bash
+q tick/r.q [host1]:port1[:usr:pwd] [host2]:port2[:usr:pwd] [-p 5020]
+```
+
+| Parameter Name | Description | Default |
+| ---- | ---- | --- |
+| host1 | host running kdb+ instance that the RDB will subscribe to e.g. tickerplant host | localhost |
+| port1 | port of kdb+ instance that the RDB will subscribe to  e.g. tickerplant port | 5010 |
+| host2 | host of kdb+ instance to inform at end-of-day, after data saved to disk  e.g. HBD host | localhost |
+| port2 | port of kdb+ instance to inform at end-of-day, after data saved to disk  e.g. HBD port | 5012 |
+| usr   | username | <none> |
+| pwd   | password | <none> |
+| -p    | [listening port](../basics/cmdline.md#-p-listening-port) for client communications | <none> |
+
+!!! Note "Standard kdb+ [command line options](../basics/cmdline.md) may also be passed"
+
+## Variables
+
+| Name | Description |
+| ---- | ---- |
+| .u.x | Connection list. First element populated by [`host1`](#usage) (tickerplant), and second element populated by [`host2`](#usage) (HDB) |
+
+## Functions
+
+Functions are open source & open to customisation.
+
+### upd
+
+Called by external process to update table data. Defaults to [`insert`](../ref/insert.md) to insert/append data to a table.
+
+```q
+upd[x;y]
+```
+Where
+
+* `x` is a symbol atom naming a table
+* `y` is table data to add to table `x`, which can contain one or more rows.
+
+### .u.end
+
+Perform end-of-day actions of saving tables to disk, clearing tables and running reload on HDB instance to make it aware of new day of data.
+
+```q
+.u.end[x]
+```
+Where x is the date that has ended.
+
+Actions performed:
+
+* finds all tables with the group attribute on the sym column
+* calls [`.Q.dpft`](../ref/dotq.md#hdpf-save-tables), with params: 
+    * HDB connection from [`.u.x`](#variables) (second element)
+    * current directory (note: directory was changed in [`.u.rep`](#urep))
+    * date passed to this function (`x`) i.e. day that is ending
+    * `` `sym `` column
+* re-apply group attribute to sym column for those tables found in first steps (as clearing the table removed grouped attribute)
+
+### .u.rep
+
+Initialise RDB by creating tables, which is then populated with any existing tickerplant log. Will set the HDB directory to use at end-of-day.
+
+```q
+.u.rep[x;y]
+```
+Where
+
+* `x` is a list of table details, each element a two item list
+    * symbol for table name
+    * schema table
+* `y` is the tickerplant log details.log comprising of a two item list:
+    * a long for the log count (null represents no log)
+    * a file symbol for the location of the current tickerplant log (null represents no log)
+
+Actions performed:
+
+* tables are created using `x`
+* if a tickerplant log file has been provided
+    * log file is replayed using [`-11!`](../basics/internal.md#-11-streaming-execute) to populate tables
+    * set the HDB directory by changing the working directory to the same directory as used by the log file (see [`tick.q`](tickq.md#usage) for details on how to alter the log file directory).
+