diff --git a/doc/benchmark.md b/doc/benchmark.md index 887e4508..34311427 100644 --- a/doc/benchmark.md +++ b/doc/benchmark.md @@ -4,6 +4,15 @@ To test performance of KVDK, you can run our benchmark tool "bench", the tool is You can manually run individual benchmark follow the examples as shown bellow, or simply run our basic benchmark script "scripts/run_benchmark.py" to test all the basic read/write performance. +To run the script, you shoulf first build kvdk, then run: + +``` +scripts/run_benchmark.py [data_type] [key distribution] +``` + +data_type: Which data type to benchmark, it can be string/sorted/hash/list/blackhole/all + +key distribution: Distribution of key of the benchmark workloads, it can be random/zipf/all ## Fill data to new instance To test performance, we need to first fill key-value pairs to the KVDK instance. Since KVDK did not support cross-socket access yet, we need to bind bench program to a numa node: @@ -20,7 +29,7 @@ Explanation of arguments: -space: PMem space that allocate to the KVDK instance. - -max_access_threads: Max concurrent access threads of the KVDK instance, set it to the number of the hyper-threads for performance consideration. + -max_access_threads: Max concurrent access threads in the KVDK instance, set it to the number of the hyper-threads for performance consideration. You can call KVDK API with any number of threads, but if your parallel threads more than max_access_threads, the performance will be degraded due to synchronization cost -type: Type of key-value pairs to benchmark, it can be "string", "hash" or "sorted". diff --git a/doc/user_doc.md b/doc/user_doc.md index 809a8e32..c982c499 100644 --- a/doc/user_doc.md +++ b/doc/user_doc.md @@ -1,9 +1,9 @@ KVDK ======= -KVDK(Key-Value Development Kit) is a Key-Value store for Persistent memory(PMem). +KVDK(Key-Value Development Kit) is a Key-Value store for Persistent Memory (PMem). -KVDK supports both sorted and unsorted KV-Pairs. +KVDK supports basic read and write operations on both sorted and unsorted KV-Pairs, it also support some advanced features, such as **backup**, **checkpoint**, **expire key**, **atomic batch write** and **transactions**. Code snippets in this user documents are from `./examples/tutorial/cpp_api_tutorial.cpp`, which is built as `./build/examples/tutorial/cpp_api_tutorial`. @@ -70,7 +70,7 @@ int main() `kvdk::Status` indicates status of KVDK function calls. Functions return `kvdk::Status::Ok` if such a function call is a success. If exceptions are raised during function calls, other `kvdk::Status` is returned, -such as `kvdk::Status::MemoryOverflow`. +such as `kvdk::Status::MemoryOverflow` while no enough memory to allocate. ## Close a KVDK instance @@ -97,26 +97,45 @@ int main() ``` ## Data types -KVDK currently supports string type for both keys and values. -### Strings -All keys and values in a KVDK instance are strings. +KVDK currently supports raw string, sorted collection, hash collection and list data type. + +### Raw String + +All keys and values in a KVDK instance are strings. You can directly store or read key-value pairs in global namespace, which is accessible via Get, Put, Delete and Modify operations, we call them string type data in kvdk. Keys are limited to have a maximum size of 64KB. -A string value can be at max 64MB in length by default. The maximum length can be configured when initializing a KVDK instance. +A value can be at max 64MB in length by default. The maximum length can be configured when initializing a KVDK instance. + +### Collections + +Instead of raw string, you can organize key-value pairs to a collection, each collection has its own namespace. + +Currently we have three types of collection: + +#### Sorted Collection + +KV pairs are stored with some kind of order (lexicographical order by default) in Sorted Collection, they can be iterated forward or backward starting from an arbitrary point(at a key or between two keys) by an iterator. They can also be directly accessed via SortedGet, SortedPut, SortedDelete operations. + +#### Hash Collection + +Hash Collection is like Raw String with a name space, you can access KV pairs via HashGet, HashPut, HashDelete and HashModify operations. + +In current version, performance of operations on hash collection is similar to sorted collection, which much slower than raw-string, so we recomend use raw-string or sorted collection as high priority. + +#### List -## Collections -All Key-Value pairs(KV-Pairs) are organized into collections. +List is a list of string elements, you can access elems at the front or back via ListPushFront, ListPushBack, ListPopFron, ListPopBack, or operation elems with index via ListInsertAt, ListInsertBefore, ListInsertAfter and ListErase. Notice that operation with index take O(n) time, while operation on front and back only takes O(1). -There is an anonymous global collection with KV-Pairs directly accessible via Get, Put, Delete operations. The anonymous global collection is unsorted. +### Namespace -Users can also create named collections. +Each collection has its own namespace, so you can store same key in every collection. Howevery, collection name and raw string key are in a same namespace, so you can't assign same name for a collection and a string key, otherwise a error status (Status::WrongType) will be returned. -KVDK currently supports sorted named collections. Users can iterate forward or backward starting from an arbitrary point(at a key or between two keys) by an iterator. Elements can also be directly accessed via SortedGet, SortedPut, SortedDelete operations. +## API Examples -## Reads and Writes in Anonymous Global Collection +### Reads and Writes with String type -A KVDK instance provides Get, Put, Delete methods to query/modify/delete entries. +A KVDK instance provides Get, Put, Delete methods to query/modify/delete raw string kvs. The following code performs a series of Get, Put and Delete operations. @@ -125,7 +144,7 @@ int main() { ... Open a KVDK instance as described in "Open a KVDK instance" ... - // Reads and Writes on Anonymous Global Collection + // Reads and Writes String KV { std::string key1{"key1"}; std::string key2{"key2"}; @@ -173,11 +192,11 @@ int main() } ``` -## Reads and Writes in a Named Collection +### Reads and Writes in a Sorted Collection A KVDK instance provides SortedGet, SortedPut, SortedDelete methods to query/modify/delete sorted entries. -The following code performs a series of SortedGet, SortedPut and SortedDelete operations, which also initialize a named collection implicitly. +The following code performs a series of SortedGet, SortedPut and SortedDelete operations on a sorted collection. ```c++ int main() @@ -194,9 +213,13 @@ int main() std::string value2{"value2"}; std::string v; + // You must create sorted collections before you do any operations on them + status = engine->SortedCreate(collection1); + assert(status == kvdk::Status::Ok); + status = engine->SortedCreate(collection2); + assert(status == kvdk::Status::Ok); + // Insert key1-value1 into "my_collection_1". - // Implicitly create a collection named "my_collection_1" in which - // key1-value1 is stored. status = engine->SortedPut(collection1, key1, value1); assert(status == kvdk::Status::Ok); @@ -206,8 +229,6 @@ int main() assert(v == value1); // Insert key1-value2 into "my_collection_2". - // Implicitly create a collection named "my_collection_2" in which - // key1-value2 is stored. status = engine->SortedPut(collection2, key1, value2); assert(status == kvdk::Status::Ok); @@ -236,8 +257,13 @@ int main() status = engine->SortedDelete(collection1, key1); assert(status == kvdk::Status::Ok); - printf("Successfully performed SortedGet, SortedPut, SortedDelete operations on named " - "collections.\n"); + // Destroy sorted collections + status = engine->SortedDestroy(collection1); + assert(status == kvdk::Status::Ok); + status = engine->SrotedDestroy(collection2); + assert(status == kvdk::Status::Ok); + + printf("Successfully performed SortedGet, SortedPut, SortedDelete operations.\n"); } ... Do something else with KVDK instance ... @@ -246,17 +272,18 @@ int main() } ``` -## Iterating a Named Collection -The following example demonstrates how to iterate through a named collection. It also demonstrates how to iterate through a range defined by Key. +### Iterating a Sorted Collection +The following example demonstrates how to iterate through a sorted collection at a consistent view of data. It also demonstrates how to iterate through a range defined by Key. ```c++ int main() { ... Open a KVDK instance as described in "Open a KVDK instance" ... - // Iterating a Sorted Named Collection + // Iterating a Sorted Sorted Collection { std::string sorted_collection{"my_sorted_collection"}; + engine->SortedCreate(sorted_collection); // Create toy keys and values. std::vector> kv_pairs; for (int i = 0; i < 10; ++i) { @@ -282,7 +309,9 @@ int main() // Sort kv_pairs for checking the order of "my_sorted_collection". std::sort(kv_pairs.begin(), kv_pairs.end()); - // Iterate through collection "my_sorted_collection" + // Iterate through collection "my_sorted_collection", the iter is + // created on a consistent view while you create it, e.g. all + // modifications after you create the iter won't be observed auto iter = engine->SortedIteratorCreate(sorted_collection); iter->SeekToFirst(); { @@ -320,7 +349,7 @@ int main() } } - printf("Successfully iterated through a sorted named collections.\n"); + printf("Successfully iterated through a sorted collections.\n"); engine->SortedIteratorRelease(iter); } @@ -330,7 +359,7 @@ int main() } ``` -## Atomic Updates +### Atomic Updates KVDK supports organizing a series of Put, Delete operations into a `kvdk::WriteBatch` object as an atomic operation. If KVDK fail to apply the `kvdk::WriteBatch` object as a whole, i.e. the system shuts down during applying the batch, it will roll back to the status right before applying the `kvdk::WriteBatch`. ```c++ @@ -387,7 +416,12 @@ A KVDK instance can be accessed by multiple read and write threads safely. Synch Users can configure KVDK to adapt to their system environment by setting up a `kvdk::Configs` object and passing it to 'kvdk::Engine::Open' when initializing a KVDK instance. ### Max Access Threads -Maximum number of access threads is specified by `kvdk::Configs::max_access_threads`. Defaulted to 48. It's recommended to set this number to the number of threads provided by CPU. +Maximum number of internal access threads in kvdk is specified by `kvdk::Configs::max_access_threads`. Defaulted to 64. It's recommended to set this number to the number of threads provided by CPU. + +You can call KVDK API with any number of threads, but if your parallel threads more than max_access_threads, the performance will be degraded due to synchronization cost + +### Clean Threads +KVDK reclaim space of updated/deleted data in background with dynamic number of clean threads, you can specify max clean thread number with `kvdk::Configs::clean_threads`. Defaulted to 8, you can config more clean threads in delete intensive workloads to avoid space be exhausted. ### PMem File Size `kvdk::Configs::pmem_file_size` specifies the space allocated to a KVDK instance. Defaulted to 2^38Bytes = 256GB. @@ -418,3 +452,7 @@ Specified by `kvdk::Configs::hash_bucket_num`. Greater number will improve perfo ### Buckets per Slot Specified by `kvdk::Configs::num_buckets_per_slot`. Smaller number will improve performance by reducing lock contentions and improving caching at the cost of greater DRAM space. Please read Architecture Documentation for details before tuning this parameter. + +## Advanced features and more API + +Please read examples/tutorial for more API and advanced features in KVDK. diff --git a/engine/hash_collection/hash_list.hpp b/engine/hash_collection/hash_list.hpp index f035ccae..033bf691 100644 --- a/engine/hash_collection/hash_list.hpp +++ b/engine/hash_collection/hash_list.hpp @@ -82,6 +82,17 @@ class HashList : public Collection { // Notice: the deleting key should already been locked by engine WriteResult Delete(const StringView& key, TimestampType timestamp); + // Modify value of "key" in the hash list + // + // Args: + // * modify_func: customized function to modify existing value of key. See + // definition of ModifyFunc (types.hpp) for more details. + // * modify_args: customized arguments of modify_func. + // + // Return: + // Status::Ok if modify success. + // Status::Abort if modify function abort modifying. + // Return other non-Ok status on any error. WriteResult Modify(const StringView key, ModifyFunc modify_func, void* modify_args, TimestampType timestamp); diff --git a/engine/kv_engine.cpp b/engine/kv_engine.cpp index d2220a86..cb7cc794 100644 --- a/engine/kv_engine.cpp +++ b/engine/kv_engine.cpp @@ -1226,10 +1226,10 @@ Status KVEngine::batchWriteRollbackLogs() { return Status::Ok; } -Status KVEngine::GetTTL(const StringView str, TTLType* ttl_time) { +Status KVEngine::GetTTL(const StringView key, TTLType* ttl_time) { *ttl_time = kInvalidTTL; - auto ul = hash_table_->AcquireLock(str); - auto res = lookupKey(str, ExpirableRecordType); + auto ul = hash_table_->AcquireLock(key); + auto res = lookupKey(key, ExpirableRecordType); if (res.s == Status::Ok) { ExpireTimeType expire_time; @@ -1266,7 +1266,7 @@ Status KVEngine::TypeOf(StringView key, ValueType* type) { if (res.s == Status::Ok) { switch (res.entry_ptr->GetIndexType()) { case PointerType::Skiplist: { - *type = ValueType::SortedSet; + *type = ValueType::SortedCollection; break; } case PointerType::List: { @@ -1274,7 +1274,7 @@ Status KVEngine::TypeOf(StringView key, ValueType* type) { break; } case PointerType::HashList: { - *type = ValueType::HashSet; + *type = ValueType::HashCollection; break; } case PointerType::StringRecord: { @@ -1289,7 +1289,7 @@ Status KVEngine::TypeOf(StringView key, ValueType* type) { return res.s == Status::Outdated ? Status::NotFound : res.s; } -Status KVEngine::Expire(const StringView str, TTLType ttl_time) { +Status KVEngine::Expire(const StringView key, TTLType ttl_time) { auto thread_holder = AcquireAccessThread(); int64_t base_time = TimeUtils::millisecond_time(); @@ -1298,10 +1298,10 @@ Status KVEngine::Expire(const StringView str, TTLType ttl_time) { } ExpireTimeType expired_time = TimeUtils::TTLToExpireTime(ttl_time, base_time); - auto ul = hash_table_->AcquireLock(str); + auto ul = hash_table_->AcquireLock(key); auto snapshot_holder = version_controller_.GetLocalSnapshotHolder(); // TODO: maybe have a wrapper function(lookupKeyAndMayClean). - auto lookup_result = lookupKey(str, ExpirableRecordType); + auto lookup_result = lookupKey(key, ExpirableRecordType); if (lookup_result.s == Status::Outdated) { return Status::NotFound; } @@ -1313,7 +1313,7 @@ Status KVEngine::Expire(const StringView str, TTLType ttl_time) { ul.unlock(); version_controller_.ReleaseLocalSnapshot(); lookup_result.s = Modify( - str, + key, [](const std::string* old_val, std::string* new_val, void*) { new_val->assign(*old_val); return ModifyOperation::Write; diff --git a/engine/kv_engine.hpp b/engine/kv_engine.hpp index c2d6ad8c..f577e96b 100644 --- a/engine/kv_engine.hpp +++ b/engine/kv_engine.hpp @@ -77,13 +77,13 @@ class KVEngine : public Engine { // 1. Expire assumes that str is not duplicated among all types, which is not // implemented yet // 2. Expire is not compatible with checkpoint for now - Status Expire(const StringView str, TTLType ttl_time) final; + Status Expire(const StringView key, TTLType ttl_time) final; // Get time to expire of str // // Notice: // Expire assumes that str is not duplicated among all types, which is not // implemented yet - Status GetTTL(const StringView str, TTLType* ttl_time) final; + Status GetTTL(const StringView key, TTLType* ttl_time) final; Status TypeOf(StringView key, ValueType* type) final; diff --git a/include/kvdk/configs.hpp b/include/kvdk/configs.hpp index 3d010591..0dca292c 100644 --- a/include/kvdk/configs.hpp +++ b/include/kvdk/configs.hpp @@ -19,9 +19,6 @@ enum class LogLevel : uint8_t { None, }; -// A snapshot indicates a immutable view of a KVDK engine at a certain time -struct Snapshot {}; - // Configs of created sorted collection // For correctness of encoding, please add new config field in the end of the // existing fields @@ -31,12 +28,14 @@ struct SortedCollectionConfigs { }; struct Configs { + // TODO: rename to concurrent internal threads + // // Max number of concurrent threads read/write the kvdk instance internally. - // Set it >= your CPU core number to get best performance + // Set it to the number of the hyper-threads to get best performance // // Notice: you can call KVDK API with any number of threads, but if your // parallel threads more than max_access_threads, the performance will be - // damaged due to synchronization cost + // degraded due to synchronization cost uint64_t max_access_threads = 64; // Size of PMem space to store KV data, this is not scalable in current diff --git a/include/kvdk/engine.hpp b/include/kvdk/engine.hpp index d408191b..8bc0c319 100644 --- a/include/kvdk/engine.hpp +++ b/include/kvdk/engine.hpp @@ -10,6 +10,7 @@ #include "comparator.hpp" #include "configs.hpp" #include "iterator.hpp" +#include "snapshot.hpp" #include "transaction.hpp" #include "types.hpp" #include "write_batch.hpp" @@ -22,7 +23,7 @@ class Engine { public: // Open a new KVDK instance or restore a existing KVDK instance // - // Para: + // Args: // * engine_path: indicates the dir path that persist the instance // * engine_ptr: store the pointer to restored instance // * configs: engine configs4 @@ -37,7 +38,7 @@ class Engine { // Restore a KVDK instance from a backup log file. // - // Para: + // Args: // * engine_path: indicates the dir path that persist the instance // * backup_log: the backup log file restored from // * engine_ptr: store the pointer to restored instance @@ -53,16 +54,43 @@ class Engine { const StringView backup_log, Engine** engine_ptr, const Configs& configs, FILE* log_file = stdout); + // Get type of key, it can be String, SortedCollection, HashCollection or List + // + // Return: + // Status::Ok and store type to "*type" on success + // Status::NotFound if key does not exist virtual Status TypeOf(StringView key, ValueType* type) = 0; - // Insert a STRING-type KV to set "key" to hold "value", return Ok on - // successful persistence, return non-Ok on any error. + // Insert a STRING-type KV to set "key" to hold "value". + // + // Args: + // *options: customized write options + // + // Return: + // Status Ok on success . + // Status::WrongType if key exists but is a collection. + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted. virtual Status Put(const StringView key, const StringView value, const WriteOptions& options = WriteOptions()) = 0; + // Search the STRING-type KV of "key" in the kvdk instance. + // + // Return: + // Return Status::Ok and store the corresponding value to *value on success. + // Return Status::NotFound if the "key" does not exist. + virtual Status Get(const StringView key, std::string* value) = 0; + + // Remove STRING-type KV of "key". + // + // Return: + // Status::Ok on success or the "key" did not exist + // Status::WrongType if key exists but is a collection type + // Status::PMemOverflow if PMem exhausted + virtual Status Delete(const StringView key) = 0; + // Modify value of existing key in the engine // - // Para: + // Args: // * modify_func: customized function to modify existing value of key. See // definition of ModifyFunc (types.hpp) for more details. // * modify_args: customized arguments of modify_func. @@ -75,45 +103,77 @@ class Engine { void* modify_args, const WriteOptions& options = WriteOptions()) = 0; + // Atomically do a batch of operations (Put or Delete) to the instance, these + // operations either all succeed, or all fail. The data will be rollbacked if + // the instance crash during a batch write + // + // Return: + // Status::Ok on success + // Status::NotFound if a collection operated by the batch does not exist + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted + // + // Notice: + // BatchWrite has no isolation guaranteed, if you need it, you should use + // Transaction API virtual Status BatchWrite(std::unique_ptr const& batch) = 0; + // Create a write batch for BatchWrite operation virtual std::unique_ptr WriteBatchCreate() = 0; + // Start a transaction on the kvdk instance. + // + // The transaction is implemented with pessimistic locking, so any operation + // may conflict with other access thread and compete locks. A transaction + // operation will return Status::Timeout in this case to avoid dead lock. + // + // Return: + // Return a pointer to transaction struct for doing transactions. + // + // Notice: + // 1. A thread should not call normal write APIs while a transaction of + // it has not been committed or rollbacked, otherwise the thread may deadlock + // as the transaction may holding locks required by normal write operations. + // 2. Once the transaction is committed or rollbacked, you can start next + // transaction on this struct. + // 3. Commit or Rollback the transaction as soon as possible to release locks + // it holds. virtual std::unique_ptr TransactionCreate() = 0; - // Search the STRING-type KV of "key" and store the corresponding value to - // *value on success. If the "key" does not exist, return NotFound. - virtual Status Get(const StringView key, std::string* value) = 0; - - // Search the STRING-type or Collection and store the corresponding expired + // Search the STRING-type or Collection and get the corresponding expired // time to *expired_time on success. - /* - * @param ttl_time. - * If the key is persist, ttl_time is INT64_MAX and Status::Ok; - * If the key is expired or does not exist, ttl_time is 0 and return - * Status::NotFound. - */ - virtual Status GetTTL(const StringView str, int64_t* ttl_time) = 0; - - /* Put the STRING-type or Collection type expired time. - * @param ttl_time is negetive or positive number. - * If ttl_time == INT64_MAX, the key is persistent; - * If ttl_time <=0, the key is expired immediately; - */ - virtual Status Expire(const StringView str, int64_t ttl_time) = 0; + // + // Args: + // * key: STRING-type key or collection name to search. + // * ttl_time: store TTL result. + // + // Return: + // Status::Ok and store ttl time to *ttl_time on success + // Status::NotFound if key is expired or does not exist + virtual Status GetTTL(const StringView key, int64_t* ttl_time) = 0; - // Remove STRING-type KV of "key". - // Return Ok on success or the "key" did not exist, return non-Ok on any - // error. - virtual Status Delete(const StringView key) = 0; + // Set ttl_time for STRING-type or Collection type data + // + // Args: + // * key: STRING-type key or collection name to set ttl_time. + // * ttl_time: ttl time to set. if ttl_time == kPersistTTL, the name will not + // be expired. If ttl_time <=0, the name is expired immediately. + // + // Return: + // Status::Ok on success. + // Status::NotFound if key does not exist. + virtual Status Expire(const StringView key, int64_t ttl_time) = 0; - // Create a sorted collection with configs + // Create a empty sorted collection with configs. You should always create + // collection before you do any operations on it + // + // Args: + // * configs: customized config of creating collection + // // Return: // Status::Ok on success // Status::Existed if sorted collection already existed // Status::WrongType if collection existed but not a sorted collection - // Status::PMemOverflow if PMem exhausted - // Status::MemoryOverflow if DRAM exhausted + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted virtual Status SortedCreate( const StringView collection, const SortedCollectionConfigs& configs = SortedCollectionConfigs()) = 0; @@ -127,26 +187,35 @@ class Engine { // Get number of elements in a sorted collection // - // Return Ok on success, return NotFound if collection not exist + // Return: + // Status::Ok on success + // Status::NotFound if collection not exist virtual Status SortedSize(const StringView collection, size_t* size) = 0; - // Insert a SORTED-type KV to set "key" of sorted collection "collection" - // to hold "value", if "collection" not exist, it will be created, return - // Ok on successful persistence, return non-Ok on any error. + // Insert a KV to set "key" in sorted collection "collection" + // to hold "value" + // Return: + // Status::Ok on success. + // Status::NotFound if collection not exist. + // Status::WrongType if collection exists but is not a sorted collection. + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted. virtual Status SortedPut(const StringView collection, const StringView key, const StringView value) = 0; - // Search the SORTED-type KV of "key" in sorted collection "collection" - // and store the corresponding value to *value on success. If the - // "collection"/"key" did not exist, return NotFound. + // Search the KV of "key" in sorted collection "collection" + // + // Return: + // Status::Ok and store the corresponding value to *value on success. + // Status::NotFound If the "collection" or "key" does not exist. virtual Status SortedGet(const StringView collection, const StringView key, std::string* value) = 0; - // Remove SORTED-type KV of "key" in the sorted collection "collection". + // Remove KV of "key" in the sorted collection "collection". + // // Return: // Status::Ok on success or key not existed in collection // Status::NotFound if collection not exist - // Status::WrongType if collection exists but is not a sorted collection + // Status::WrongType if collection exists but is not a sorted collection. // Status::PMemOverflow if PMem exhausted. virtual Status SortedDelete(const StringView collection, const StringView key) = 0; @@ -155,10 +224,10 @@ class Engine { // Create an empty List. // Return: - // Status::WrongType if list name existed but is not a List. - // Status::Existed if a List named list already exists. - // Status::PMemOverflow if PMem exhausted. - // Status::Ok if successfully created the List. + // Status::WrongType if list name existed but is not a List. + // Status::Existed if a List named list already exists. + // Status::PMemOverflow if PMem exhausted. + // Status::Ok if successfully created the List. virtual Status ListCreate(StringView list) = 0; // Destroy a List associated with key @@ -170,50 +239,50 @@ class Engine { // Total elements in List. // Return: - // Status::InvalidDataSize if list name is too long - // Status::WrongType if list name is not a List. - // Status::NotFound if list does not exist or has expired. - // Status::Ok and length of List if List exists. + // Status::InvalidDataSize if list name is too long + // Status::WrongType if list name is not a List. + // Status::NotFound if list does not exist or has expired. + // Status::Ok and length of List if List exists. virtual Status ListSize(StringView list, size_t* sz) = 0; // Push element as first element of List // Return: - // Status::InvalidDataSize if list name or elem is too long - // Status::WrongType if list name is not a List. - // Status::PMemOverflow if PMem exhausted. - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if list name or elem is too long + // Status::WrongType if list name is not a List. + // Status::PMemOverflow if PMem exhausted. + // Status::Ok if operation succeeded. virtual Status ListPushFront(StringView list, StringView elem) = 0; // Push element as last element of List // Return: - // Status::InvalidDataSize if list name or elem is too long - // Status::WrongType if list name in the instance is not a List. - // Status::PMemOverflow if PMem exhausted. - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if list name or elem is too long + // Status::WrongType if list name in the instance is not a List. + // Status::PMemOverflow if PMem exhausted. + // Status::Ok if operation succeeded. virtual Status ListPushBack(StringView list, StringView elem) = 0; // Pop first element of list // Return: - // Status::InvalidDataSize if list name is too long - // Status::WrongType if list is not a List. - // Status::NotFound if list does not exist or has expired. - // Status::Ok and element if operation succeeded. + // Status::InvalidDataSize if list name is too long + // Status::WrongType if list is not a List. + // Status::NotFound if list does not exist or has expired. + // Status::Ok and element if operation succeeded. virtual Status ListPopFront(StringView list, std::string* elem) = 0; // Pop last element of List // Return: - // Status::InvalidDataSize if list name is too long - // Status::WrongType if list is not a List. - // Status::NotFound if list does not exist or has expired. - // Status::Ok and element if operation succeeded. + // Status::InvalidDataSize if list name is too long + // Status::WrongType if list is not a List. + // Status::NotFound if list does not exist or has expired. + // Status::Ok and element if operation succeeded. virtual Status ListPopBack(StringView list, std::string* elem) = 0; // Push multiple elements to the front of List // Return: - // Status::InvalidDataSize if list name or elem is too long - // Status::WrongType if list is not a List. - // Status::PMemOverflow if PMem exhausted. - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if list name or elem is too long + // Status::WrongType if list is not a List. + // Status::PMemOverflow if PMem exhausted. + // Status::Ok if operation succeeded. virtual Status ListBatchPushFront(StringView list, std::vector const& elems) = 0; virtual Status ListBatchPushFront(StringView list, @@ -221,10 +290,10 @@ class Engine { // Push multiple elements to the back of List // Return: - // Status::InvalidDataSize if list or elem is too long - // Status::WrongType if list name is not a List. - // Status::PMemOverflow if PMem exhausted. - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if list or elem is too long + // Status::WrongType if list name is not a List. + // Status::PMemOverflow if PMem exhausted. + // Status::Ok if operation succeeded. virtual Status ListBatchPushBack(StringView list, std::vector const& elems) = 0; virtual Status ListBatchPushBack(StringView list, @@ -232,19 +301,19 @@ class Engine { // Pop first N element of List // Return: - // Status::InvalidDataSize if list is too long - // Status::WrongType if list is not a List. - // Status::NotFound if list does not exist or has expired. - // Status::Ok and element if operation succeeded. + // Status::InvalidDataSize if list is too long + // Status::WrongType if list is not a List. + // Status::NotFound if list does not exist or has expired. + // Status::Ok and element if operation succeeded. virtual Status ListBatchPopFront(StringView list, size_t n, std::vector* elems) = 0; // Pop last N element of List // Return: - // Status::InvalidDataSize if list is too long - // Status::WrongType if list is not a List. - // Status::NotFound if list does not exist or has expired. - // Status::Ok and element if operation succeeded. + // Status::InvalidDataSize if list is too long + // Status::WrongType if list is not a List. + // Status::NotFound if list does not exist or has expired. + // Status::Ok and element if operation succeeded. virtual Status ListBatchPopBack(StringView list, size_t n, std::vector* elems) = 0; @@ -252,9 +321,9 @@ class Engine { // src_pos and dst_pos can only be 0, indicating List front, // or -1, indicating List back. // Return: - // Status::WrongType if src or dst is not a List. - // Status::NotFound if list or element not exist - // Status::Ok and moved element if operation succeeded. + // Status::WrongType if src or dst is not a List. + // Status::NotFound if list or element not exist + // Status::Ok and moved element if operation succeeded. virtual Status ListMove(StringView src_list, ListPos src_pos, StringView dst_list, ListPos dst_pos, std::string* elem) = 0; @@ -262,49 +331,58 @@ class Engine { // Insert a element to a list at index, the index can be positive or // negative // Return: - // Status::InvalidDataSize if list name is too large - // Status::WrongType if collection is not a list - // Status::NotFound if collection not found or index is beyond list size - // Status::Ok if operation succeeded + // Status::InvalidDataSize if list name is too large + // Status::WrongType if collection is not a list + // Status::NotFound if collection not found or index is beyond list size + // Status::Ok if operation succeeded virtual Status ListInsertAt(StringView list, StringView elem, long index) = 0; // Insert an element before element "pos" in list "collection" // Return: - // Status::InvalidDataSize if elem is too long. - // Status::PMemOverflow if PMem exhausted. - // Status::NotFound if List of the ListIterator has expired or been - // deleted, or "pos" not exist in the list - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if elem is too long. + // Status::PMemOverflow if PMem exhausted. + // Status::NotFound if List of the ListIterator has expired or been + // deleted, or "pos" not exist in the list + // Status::Ok if operation succeeded. virtual Status ListInsertBefore(StringView list, StringView elem, StringView pos) = 0; // Insert an element after element "pos" in list "collection" // Return: - // Status::InvalidDataSize if elem is too long. - // Status::PMemOverflow if PMem exhausted. - // Status::NotFound if List of the ListIterator has expired or been - // deleted, or "pos" not exist in the list - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if elem is too long. + // Status::PMemOverflow if PMem exhausted. + // Status::NotFound if List of the ListIterator has expired or been + // deleted, or "pos" not exist in the list + // Status::Ok if operation succeeded. virtual Status ListInsertAfter(StringView list, StringView elem, StringView pos) = 0; // Remove the element at index // Return: - // Status::NotFound if the index beyond list size. - // Status::Ok if operation succeeded, and store removed elem in "elem" + // Status::NotFound if the index beyond list size. + // Status::Ok if operation succeeded, and store removed elem in "elem" virtual Status ListErase(StringView list, long index, std::string* elem) = 0; // Replace the element at index // Return: - // Status::InvalidDataSize if elem is too long - // Status::NotFound if if the index beyond list size. - // Status::Ok if operation succeeded. + // Status::InvalidDataSize if elem is too long + // Status::NotFound if if the index beyond list size. + // Status::Ok if operation succeeded. virtual Status ListReplace(StringView list, long index, StringView elem) = 0; // Create a KV iterator on list "list", which is able to iterate all elems in - // the list at "snapshot" version, if snapshot is nullptr, then a - // internal snapshot will be created at current version and the iterator will - // be created on it + // the list + // + // Args: + // * snapshot: iterator will iterate all elems a t "snapshot" version, if + // snapshot is nullptr, then a internal snapshot will be created at current + // version and the iterator will be created on it + // * status: store operation status if not null + // + // Return: + // Return A pointer to iterator on success. + // Return nullptr if list not exist or any other errors, and store error + // status to "status" // // Notice: // 1. Iterator will be invalid after the passed snapshot is released @@ -313,20 +391,76 @@ class Engine { virtual ListIterator* ListIteratorCreate(StringView list, Snapshot* snapshot = nullptr, Status* status = nullptr) = 0; + // Release a ListIterator and its holding resources virtual void ListIteratorRelease(ListIterator*) = 0; /// Hash APIs /////////////////////////////////////////////////////////////// + // Create a empty hash collection. You should always create collection before + // you do any operations on it + // + // Return: + // Status::Ok on success + // Status::Existed if hash collection already existed + // Status::WrongType if collection existed but not a hash collection + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted virtual Status HashCreate(StringView collection) = 0; + + // Destroy a hash collection + // Return: + // Status::Ok on success + // Status::WrongType if collection existed but not a hash collection + // Status::PMemOverflow if PMem exhausted virtual Status HashDestroy(StringView collection) = 0; + + // Get number of elements in a hash collection + // + // Return: + // Status::Ok on success + // Status::NotFound if collection not exist virtual Status HashSize(StringView collection, size_t* len) = 0; + + // Search the KV of "key" in hash collection "collection" + // + // Return: + // Status::Ok and store the corresponding value to *value on success. + // Status::NotFound If the "collection" or "key" does not exist. virtual Status HashGet(StringView collection, StringView key, std::string* value) = 0; + + // Insert a KV to set "key" in hash collection "collection" + // to hold "value" + // Return: + // Status::Ok on success. + // Status::NotFound if collection not exist. + // Status::WrongType if collection exists but is not a hash collection. + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted. virtual Status HashPut(StringView collection, StringView key, StringView value) = 0; + + // Remove KV of "key" in the hash collection "collection". + // + // Return: + // Status::Ok on success or key not existed in collection + // Status::NotFound if collection not exist + // Status::WrongType if collection exists but is not a hash collection. + // Status::PMemOverflow if PMem exhausted. virtual Status HashDelete(StringView collection, StringView key) = 0; + + // Modify value of a existing key in a hash collection + // + // Args: + // * modify_func: customized function to modify existing value of key. See + // definition of ModifyFunc (types.hpp) for more details. + // * modify_args: customized arguments of modify_func. + // + // Return: + // Return Status::Ok if modify success. + // Return Status::Abort if modify function abort modifying. + // Return other non-Ok status on any error. virtual Status HashModify(StringView collection, StringView key, ModifyFunc modify_func, void* cb_args) = 0; + // Create a KV iterator on hash collection "collection", which is able to // iterate all elems in the collection at "snapshot" version, if snapshot is // nullptr, then a internal snapshot will be created at current version and @@ -362,9 +496,18 @@ class Engine { virtual void ReleaseSnapshot(const Snapshot*) = 0; // Create a KV iterator on sorted collection "collection", which is able to - // sequentially iterate all KVs in the "collection" at "snapshot" version, if + // sequentially iterate all KVs in the "collection". + // + // Args: + // * snapshot: iterator will iterate all elems a t "snapshot" version, if // snapshot is nullptr, then a internal snapshot will be created at current // version and the iterator will be created on it + // * status: store operation status if not null + // + // Return: + // Return A pointer to iterator on success. + // Return nullptr if collection not exist or any other errors, and store error + // status to "status" // // Notice: // 1. Iterator will be invalid after the passed snapshot is released @@ -374,13 +517,14 @@ class Engine { Snapshot* snapshot = nullptr, Status* s = nullptr) = 0; - // Release a sorted iterator + // Release a sorted iterator and its holding resouces virtual void SortedIteratorRelease(SortedIterator*) = 0; // Register a customized comparator to the engine on runtime // - // Return true on success, return false if a comparator of comparator_name - // already existed + // Return: + // Return true on success + // Return false if a comparator of comparator_name already existed virtual bool registerComparator(const StringView& comparator_name, Comparator) = 0; diff --git a/include/kvdk/snapshot.hpp b/include/kvdk/snapshot.hpp new file mode 100644 index 00000000..91b29622 --- /dev/null +++ b/include/kvdk/snapshot.hpp @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2021 Intel Corporation + */ + +#pragma once + +namespace KVDK_NAMESPACE { + +// A snapshot indicates a immutable view of a KVDK engine at a certain time +struct Snapshot {}; + +} // namespace KVDK_NAMESPACE \ No newline at end of file diff --git a/include/kvdk/transaction.hpp b/include/kvdk/transaction.hpp index 05cbf739..9a02978c 100644 --- a/include/kvdk/transaction.hpp +++ b/include/kvdk/transaction.hpp @@ -9,28 +9,98 @@ #include "types.hpp" namespace KVDK_NAMESPACE { +// This struct is used to do transaction operations. A transaction struct is +// assotiated with a kvdk instance class Transaction { public: - // TODO: use StringView instead of std::string + // Put a STRING-type KV to the transaction + // + // Return: + // Status::Ok on success + // Status::Timeout on conflict and long-time lock contention virtual Status StringPut(const StringView key, const StringView value) = 0; + // Delete a STRING-type key to the transaction + // + // Return: + // Status::Ok on success + // Status::Timeout on conflict and long-time lock contention virtual Status StringDelete(const StringView key) = 0; + // Get value of a STRING-type KV. It will first get from the transaction + // operations (Put/Delete), then the kvdk instance of the transaction + // + // Return: + // Status::Ok on success and store value to "*value" + // Status::NotFound if key not existed or be deleted by this transaction + // Status::Timeout on conflict and long-time lock contention virtual Status StringGet(const StringView key, std::string* value) = 0; + // Put a KV of sorted collection to the transaction + // + // Return: + // Status::Ok on success + // Status::NotFound if collection does not exist + // Status::Timeout on conflict and long-time lock contention virtual Status SortedPut(const StringView collection, const StringView key, const StringView value) = 0; + // Delete a KV from sorted collection to the trnasaction. + // + // Return: + // Status::Ok on success + // Status::NotFound if collection does not exist + // Status::Timeout on conflict and long-time lock contention virtual Status SortedDelete(const StringView collection, const StringView key) = 0; + // Get value of a KV from sorted collection. It will first get from the + // transaction operations (Put/Delete), then the kvdk instance of the + // transaction + // + // Return: + // Status::Ok and store value to "*value" on success + // Status::NotFound if collection or key does not exist + // Status::Timeout on conflict and long-time lock contention virtual Status SortedGet(const StringView collection, const StringView key, std::string* value) = 0; + // Put a KV of hash collection to the transaction + // + // Return: + // Status::Ok on success + // Status::NotFound if collection does not exist + // Status::Timeout on conflict and long-time lock contention virtual Status HashPut(const StringView collection, const StringView key, const StringView value) = 0; + // Delete a KV from hash collection to the transaction + // + // Return: + // Status::Ok on success + // Status::NotFound if collection does not exist + // Status::Timeout on conflict and long-time lock contention virtual Status HashDelete(const StringView collection, const StringView key) = 0; + + // Get value of a KV from hash collection. It will first get from the + // transaction operations (Put/Delete), then the kvdk instance of the + // transaction + // + // Return: + // Status::Ok and store value to "*value" on success + // Status::NotFound if collection or key does not exist + // Status::Timeout on conflict and long-time lock contention virtual Status HashGet(const StringView collection, const StringView key, std::string* value) = 0; + // Commit all operations of the transaction to the kvdk instance, and + // release all locks it holds + // + // Return: + // Status::Ok on success, all operations will be persistent on the instance + // Status::PMemOverflow/Status::MemoryOverflow if PMem/DRAM exhausted virtual Status Commit() = 0; + + // Rollback all operations of the transaction, release locks it holds virtual void Rollback() = 0; + + // Return status of the last transaction operation virtual Status InternalStatus() = 0; + virtual ~Transaction() = default; }; } // namespace KVDK_NAMESPACE \ No newline at end of file diff --git a/include/kvdk/types.h b/include/kvdk/types.h index dfa97b8a..f7a1b637 100644 --- a/include/kvdk/types.h +++ b/include/kvdk/types.h @@ -43,8 +43,8 @@ typedef void (*KVDKFreeFunc)(void*); #define KVDK_TYPES(GEN) \ GEN(String) \ - GEN(SortedSet) \ - GEN(HashSet) \ + GEN(SortedCollection) \ + GEN(HashCollection) \ GEN(List) typedef enum { KVDK_TYPES(GENERATE_ENUM) } KVDKValueType; diff --git a/tests/tests.cpp b/tests/tests.cpp index 2fe32c97..7f25c086 100644 --- a/tests/tests.cpp +++ b/tests/tests.cpp @@ -473,8 +473,8 @@ TEST_F(EngineBasicTest, TypeOfKey) { ASSERT_EQ(Engine::Open(db_path.c_str(), &engine, configs, stdout), Status::Ok); std::unordered_map key_types; - for (auto type : {ValueType::String, ValueType::HashSet, ValueType::List, - ValueType::SortedSet}) { + for (auto type : {ValueType::String, ValueType::HashCollection, + ValueType::List, ValueType::SortedCollection}) { std::string key = KVDKValueTypeString[type]; key_types[key] = type; ValueType type_resp; @@ -483,7 +483,7 @@ TEST_F(EngineBasicTest, TypeOfKey) { ASSERT_EQ(engine->Put(key, ""), Status::Ok); break; } - case ValueType::HashSet: { + case ValueType::HashCollection: { ASSERT_EQ(engine->HashCreate(key), Status::Ok); break; } @@ -491,7 +491,7 @@ TEST_F(EngineBasicTest, TypeOfKey) { ASSERT_EQ(engine->ListCreate(key), Status::Ok); break; } - case ValueType::SortedSet: { + case ValueType::SortedCollection: { ASSERT_EQ(engine->SortedCreate(key), Status::Ok); break; }