CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Types of changes

Added for new features. Changed for changes in existing functionality. Deprecated for soon-to-be removed features. Removed for now removed features. Fixed for any bug fixes. Security in case of vulnerabilities.

staging

please add your unreleased change here.

[v1.11.0.dev20241108] - 2024-11-08

Added

• [component] add new io.data_sink component, it is used to export data to an external data source
• [component] add new psi_tp component, which is three party psi
• [component] add sql_processor component for handling SQL preprocessing
• [component] psi component add report for outputing the row nums

Changed

• [component] sf is split into two parts: sf without FL algorithms and sf_fl. sf-lite release contains sf, sf-full release contains both sf and sf_fl.
• [sgb] The label holder bucket sum now uses HEU calculation, removing the need for the Numba dependency
• [spu] bump spu version to 0.9.3.dev20241101

Fixed

• [sgb] Fix sgb set params non-idempotent issue

[v1.10.0.dev20240906] - 2024-09-06

Added

• [component] IO component supports import and export sgb/glm model
• [component] Switch from ray to a local task scheduler
• [component] Support export SGD/GLM 2-Party HE model package
• [component] component reflect, include all component in the package of stats/io/preprocessing(exclude psi), and update the component version to 1.0.0
• [component] Integrate with DataProxy SDK

Changed

• [data] Change single party r2_score to sklearn function
• [docs] Security warning translation

Fixed

• [sgb] Fix checkpoint prediction initialization

[v1.9.0b0] - 2024-08-28

Added

• [component] support sql null
• [component] io_write_data supports xgb
• [component] Add expr_condition_filter
• [component] PSI supports specifying party

Fix

• [component] fix training error on empty tree

[v1.8.0b0] - 2024-07-17

Added

• [component] support tweedie learning objective in SGB
• [component] update graph builder in model export

[v1.7.0.dev20240605] - 2024-06-05

Added

• [component] Add stats psi component
• [component] Add score card component
• [component] Add data sampling component
• [component] Add type cast component
• [component] Optimization GLM & LR training speed by using beaver cache
• [SLModel] Add attack method BLA
• [SLModel] Add defense method CAE
• [sgb] Support sample weight training

Fix

• [component] The online prediction of the GLM model has a large deviation compared with the offline prediction.

[v1.6.1a0] - 2024-05-24

Fix

• [component] fix sgb export model sigmoid type not consistent with the offline prediction sigmoid type
• [component] fix sgb/xgb export model miss base score

[v1.6.0a0] - 2024-05-21

Added

• [component] pvalue support GLM
• [component] onehot add drop parameter to support first & mode
• [component] XGB/SGB support checkpoint
• [component] LR support report weights
• [component] Add union component
• [FLModel/SLModel] tf backend support custom loss
• [SLModel] Base model support additional loss
• [SLModel] Add attack method FSHA
• [SLModel] Add defense method MID
• [SLModel] Add defense method FedPass
• [Infra] Add arm64 build support

[v1.5.0.dev240315] - 2024-03-15

Added

• SL: custom trainning step supported by lightning style base module.

Removed

• No longer providing x64 macOS binary packages.

[v1.5.0.dev240312] - 2024-03-12

Added

• SL: grad_average is supported on GPU.
• SL: mix_up defence can be called through callback.
• Simulation: add some unpartitioned datasets.

Fixed

• Fix order in componet load table.

Changed

• Serving: Update serving linkfunc type.
• FL: rename get/set_gradients methods
• SGX: improve doc for the accuracy analysis.
• SL: import fia attack.

[v1.5.0.dev240304] - 2024-03-04

Added

• Add gradient_average callback in SL.

Fixed

• Fix tuner default resource allocate.

Changed

• Boost the speed of autoattack benchmark.
• Upgrade to Python 3.10.

[v1.5.0.dev240227] - 2024-02-27

Added

• Component: Suppport Early Stop in SGB via XGBoost Callbacks.
• Component: Add storage interface and oss/s3 storage impl.
• FLModel: Add new sample methods for datasets.
• Device Support cheetah h2a.
• Tuner optimizes resource allocation logic.
• Docs: Add SecretFlow Benchmark Results.
• Docs: Add more docs for SS-GLM.

Changed

• Use ray instead of secretflow-ray and bump to ray-2.9.

[v1.4.0.dev240205] - 2024-02-05

Added

• Add Criteo(100M) Dataset.
• Add split learning nn component.
• Add sgb & ss_xgb model export.
• Split learning support prediction for file input.
• GPU support for split learning attack algorithm.

Changed

• Unify nn Callback naming.

[v1.4.0.dev240129] - 2024-01-29

Added

• Split learning in PyTorch supports GPU in debug mode.

• Split learning in PyTorch supports pipeline strategy.

• Sparse compression supports the compression of multi-dimensional data, such as images.

• SLModel supports input in the form of file names and allows for multiple label inputs.

• Component: prediction operator results can now save feature columns.

• Component: binning operators can now view reports directly.

• Component: binning rule modification is supported.

• Component: support model exports, support both online and offline prediction.

Fixed

• fix: bump rayfed to 0.2.0a18 to fix shutdown lock.
• Component: improve GLM training accuracy.
• Component: reduce GLM training memory and computation cost.

[v1.4.0.dev240122] - 2024-01-22

Added

• Add a new feature of chunked computation to control the memory peak.
• The binning component supports displaying binning rules.

Fixed

• Fix a bug that occurs when saving the SGB model fails when the number of trees in the model is 1.
• Fix bin rule report.

[1.4.0.dev240116] - 2024-01-16

Added

• sgb/ss_xgb/ss_glm/ss_sgd can save specified columns with result now.
• ss_glm: expose two params infeed size and newton.
• SplitLearning
  • Add a defense method against data reconstruction attacks.
  • Adapt replay attack and gradreplace attack for deepfm.
  • Add expoit attack.

Changed

• Component: update fillna descriptions.

Fixed

• Fix SS-GLM toturial.

[1.4.0.dev240103] - 2024-1-3

Added

• Add grad replace attack and replay attack.
• Add autoattack benchmark examples.
• Expose job_name param in sf.init.

Changed

• Bump rayfed version: optimizing Error Propagation and Capture.
• Component: woe_bins requires at least 5 bins to read.
• Component: add barrier_on_shutdown as sf cluster config.

[1.4.0.dev231225] - 2023-12-25

Added

• Add DataProxy binary writer.
• Add PSI v2 APIs.
• Add sl_deepfm_torch, sl_dnn_tf and sl_dnn_torch applications.
• Support sgb conversion into a single model.
• Support debug mode in tuner.
• Docs: Update installation and deployment docs.
• Docs: Add fl gpu docs.
• Docs: Add debug mode docs.

Changed

• Component: onehot_encode component limits size of col_rules.
• Component: sgb_predict component rm batch_size param.

Fixed

• Fix pytorch_audio_classification example and tutorial.
• Fix bin substitution for incomplete rule.
• Fix sgb_predict component when all features come from a party.

[1.3.0.dev231219] - 2023-12-19

Added

• Make barrier_on_shutdown optional.
• Support SGB label holder without features.
• Support SL Model training on file data with mutiple labels.
• Add SL ResNet and VGG application.
• Secretflow ic: Add package interconnection protobuf files.
• Component: Add feature calculate component to generate new features by performing calculations on original features.
• Component: Support SGB prediction on big dataset.

Changed

• SGB optimize memory usage in prediction.
• Component: Bump groupby statistics version.
• Component: Improve translation.

Fixed

• Component: Fix woe io and fillna.

[1.3.0.dev231211] - 2023-12-11

Added

• Add IO component including read, write and identity.
• Change groupby component to by-query style.

[1.3.0.dev231128] - 2023-11-28

Added

• Add secretflow tuner for automl and autoattack.
• Add IO component including read, write and identity.
• Change groupby component to by-query style.

[1.3.0.dev231120] - 2023-11-20

Added

• Support file data input in SLModel.
• Expose copts in SPU devices.
• Component: Add benchmarks.
• Add federated callback framework.
• Add new agg method -- concat,sum.
• Add ic_mode.

Changed

• Component: add upper_bound for max_group_size in groupby_statistics.
• Component: modify test_size and train_size restrictions in train_test_split.
• Clear legacy history class in FL.

Fixed

• Kuscia adapter: check datasource only when author matches.

[1.3.0.dev231109] - 2023-11-9

Added

• Component: Support eq_range binning.
• Component: Support TLS in nsjail.
• Component: Add test framework.
• Component: Adapt to DataProxy.

Changed

• Component: Select features in binning.
• Component: biclassification_eval return nan values if min_item_cnt_per_bucket doesn't match.

Fixed

• Enhance sgb.
• Component: Fix groupby_statistics

[1.2.0] - 2023-11-8

Added

• Add debug mode.
• Support GLM model transition from MPC version to Federated version.
• FLModel supports PFL to allow custom aggregation logic on server.
• Add split learning applications: BST, MMoE.
• Add sparse, quantized and mixed compressor.
• Add polars backend for dataframe and SL model to enhance the data processing performance.

Changed

• Refactor data preprocessing module (VDataFrame, Partition).
• Doc: rearrange docs for split recommendation suite: SplitRec.
• Doc: Update PSI benchmark.
• Component: expose cross_silo_comm_backend option for secretflow init.
• Component: add vert_binning.
• Component: migrate to SecretFlow Open Specification.

[1.2.0.dev231025] - 2023-10-25

Added

• Support naive sl on torch backend.
• Add kwargs for custom strategy in sl.
• Add bst and mmoe in sl applications.

Changed

• Input attributes of some components are modified.
• Convert domain data to individual tables in preprocess_sf_node_eval_param of kuscia adapter.

[1.2.0.dev231016] - 2023-10-16

Added

• Secretflow support debug mode.
• Add vert binning for equal range bining method

Changed

• The data preprocessing module (VDataFrame, Partition) has been refactored, enhancing the data processing performance (primarily targeting the Polars backend).

Fixed

• Fix error when flmodel with tf backend use gpu.
• Fix kuscia adapter

[1.2.0.dev231009] - 2023-10-9

Added

• FLModel supports PFL to allow custom aggregation logic on server.

[1.2.0.dev230926] - 2023-9-26

Changed

• component: expose cross_silo_comm_backend option for secretflow init.

Fixed

• SPU device: rm invalid use_link option.

[1.2.0.dev230925] - 2023-9-25

Changed

• update psi benchmark.

Fixed

• fix tls on kuscia.

[1.2.0.dev230918] - 2023-9-18

Added

• support GLM model transition from MPC version to Federated version.
• add sparse, quantized and mixed compressor.
• add polars backend for dataframe and SL model.
• add DisPFL adn VFGNN example.
• add cross_silo_comm_backend option to SFClusterConfig.RayFedConfig.

Fixed

• fix dataset build in SL.

[1.1.0b0] - 2023-9-6

Added

• SLModel: supports quantization compression algorithm, reducing communication volume by 2-4 times.
• SLModel: supports pipeline strategy, which can accelerate model training by 2-4 times in most scenarios.
• SLModel: PyTorch backend supports GPU.
• SLModel: introduces two attack and defense algorithms, LIA and FIA, for testing model security in the research and development stage.
• SLModel: supports a mode where one party only provides labels without providing features.
• Component: GLM train and predict components.
• Support the usage of brpc link as a backend for cross-silo communication.

Changed

• Component: add more parameters for SGB components.

[1.1.0.dev230825] - 2023-8-25

Added

• Stateful task for teeu
• docs: DeepFM translation

Changed

• Switch to shared workflow

[1.1.0.dev230818] - 2023-8-18

Added

• Add five papers in Vertical Federated Learning
• docs: update references on homomorphic encryption
• Add new quantized compressor method and tutorial

[1.1.0.dev230811] - 2023-8-11

Fixed

• PSI use psi_csv in psi comp.

Added

• GLM train and predict components

[1.1.0b0.dev1] - 2023-8-7

Added

• Support the usage of brpc link as a backend for cross-silo communication.

[1.1.0b0.dev0] - 2023-8-7

Added

• Label inference attack v3

[1.0.0b2] - 2023-7-28

Changed

• SGB upgrade: use SGBFactory to replace SGB, update parameters and tutorials. SGB now supports more functionalities.
• Predict supports callbacks, call the callback function before/after prediction starts and after every step.

[1.0.0a1] - 2023-7-26

Fixed

• SLModel fix bug in handling data with databuilder
• The FLModel solves the problem of the production mode hanging due to a small batch size.

[1.0.0a0] - 2023-7-3

Added

• SLModel supports AggLayer
• SLModel（nn/deepfm）supports one party providing features and the other party providing labels.
• Component Specification and SecretFlow Component List v0.0.1.

Changed

• Bump spu to 0.4.1b0

Fixed

• Fix logic error when sl base model load from none

[0.8.3b0] - 2023-6-14

Added

• Split learning add application of deepfm for recommendation scenarios.
• Split learning add pytorch support.
• Add IO tutorials for federated learning.
• Reorg DP strategies
• SGB and XGB refactor. Add data checks. Improve qcut.
• Add the preview version of components.

Changed

• Bump spu to 0.3.3b2

Fixed

• TEEU function serialization.

[0.8.2b3] - 2023-5-15

Fixed

• Correct TEEU function serialization protocol and docs.

[0.8.2b2] - 2023-5-9

Fixed

• Fix SPU compilation cache bug

Changed

• Bump spu to 0.3.3b2

[0.8.2b1] - 2023-4-27

Fixed

• Add missing init files.

[0.8.2b0] - 2023-4-19

Added

• TEE python Unit(TEEU) is introduced as the TEE cryptographic device and enables authorized computation with authorized data in TEE. TEEU brings more possibilities for hybrid computation.
• An experimental SecretFlow component design.
• SGB feature: add support for pre-pruning and model save & load.

Changed

• Use pytest instead of unittest.
• Bump spu to 0.3.3b0
• Bump heu to 0.4.3b2

Fix

• Fix hess lr auc err with large learning_rate.

[0.8.1b0] - 2023-4-7

Added

• SecureBoost and its benchmark.
• Unbalanced PSI
  • Two sub-protocols for generating cache and transmitting cache.
  • Online with shuffling sub-protocol, supporting big data parties to obtain data.
• Semi-homomorphic encryption protocol - OU

Changed

• Bump spu to 0.3.2b12

Fix

• Fix psi_join_csv output columns error.
• Fix heu object decode.

[0.8.0b1] - 2023-3-31

• Bump spu to 0.3.2b11

[0.8.0b0] - 2023-3-28

Added

• SCQL (Secure Collaborative Query Language).

Changed

• Bump spu to 0.3.2b9.

[0.7.18b5] - 2023-3-22

Changed

• Bump many dependencies for security fix.

[0.7.18b4] - 2023-3-7

Changed

• Bump RayFed to 0.1.1a.

Fix

• give min num_cpus for simulation.

[0.7.18b3] - 2023-2-13

Fix

• add init.py to sl tensorflow strategy folder.

[0.7.18b2] - 2023-2-9

Fix

• add party as resource label in simulation mode.

[0.7.18b1] - 2023-1-30

Changed

• add an option whether exit on cross-silo sending.
• put all requires in one file except dev.

Fix

• Fix psi_join_csv output columns error.
• fix heu object decode.

[0.7.18] - 2023-01-09

Changed

• Bump:
  • rayfed to 0.1.0b0.
  • spu to 0.3.1b9.
  • sf-heu to 0.3.2b1.

[0.7.17] - 2022-12-21

Added

• add export_model api for SLModel
• add get_params api for preprocessing

Changed

• VDataFrame read_csv uses spu.psi_csv instead of spu.psi_df to reduce memory usage.
• Bump rayfed to 0.1.0a10.
• Control the concurrency of ray task in SLModel fit/predict/evaluate.

Fixed

• psi join multi-key sort command parameters

[0.7.16] - 2022-12-16

Changed

• Optimize psi join memory usage.

[0.7.15] - 2022-12-14

Changed

• Bump rayfed to 0.1.0a9.

Fixed

• SS GLM distribution type err.

[0.7.14] - 2022-12-12

Added

• GLM.

Changed

• bump rayfed to 0.1.0a8.

Fixed

• SSXgb for user_specified_num_returns=1.
• SLModel sets steps_per_epoch for worker.
• set num_cpu and resources only when start ray with local mode.

[0.7.13] - 2022-12-08

Added

• Multi controller based on rayfed.

Changed

• Rewrite data builder.
• Bump to yacl.
• utils.testing.cluster_def supports NPC.

Fixed

• FLModel save path.
• SSXgb col sub error.
• HomeXgb callback.
• Compress mask.
• spu.call fails when SPUCompilerNumReturnsPolicy is FROM_USER and user_specified_num_returns is 1.

[0.7.12] - 2022-11-18

Changed

• OneHotEncoder remove PYUObject property.

Fixed

• Add sl_model metrics wrap to fix format error.

[0.7.11] - 2022-11-15

Added

• Add Finetune and FedEval to SFXgboost
• Add SLModel support multi parties(>=2)

Changed

• FLModel supports most metrics of regression and classification

Fixed

• SLModel can be initialized without model.
• PSI doc typos.
• Fix the logic error bug of savemodel and loadmodel path in SLModel and FLModel

[0.7.10] - 2022-10-25

Added

• Add scorecard.
• Add replace/mode function to DataFrame.
• Add round function to VDataFrame.
• Add psi_join_csv and psi_join_df.
• Add preprocessing.LogroundTransformer.
• Add args to preprocessing.OneHotEncoder.

Changed

• Bump dependencies
  • secretflow-ray to 2.0.0.dev2
• Update psi_df doc.
• Optimize sl_model by tf_funciton.
• Add curve parameter for ecdh psi.
• Protect biclassification, psi and pva with pyu object.
• Modify XgbModel predict api.

Fixed

• Raise exception if spu_fe.compile fails.
• Fix quantile security vulnerability.
• Fix woe bin bugs.
• Fix psi_join recv timeout.

[0.7.9] - 2022-10-24

Added

• omp_num_threads param for secretflow init().
• Regression and biclassification evaluation.
• Xgboost evaluation.
• Horizontal fl supports default naive aggreagte for metrics.
• PVA calculation.

Changed

• Remove graph util NodeDataLoader.

Fixed

• VDataFrame docstring.
• Remove dependencies
  • dgl

Changed

• Get rid of import tensorflow/torch when import secretflow.

[0.7.8] - 2022-9-22

Added

• Add license file

Fixed

• Fix sl predict & remove reveal
• Fix typos in function docs.

Changed

• Bump dependencies
  • TensorFlow to 2.10.0
  • Jax to 0.3.17
  • Jaxlib to 0.3.15

[0.7.7] - 2022-9-16

Changed

• Bump dependencies
  • sf-heu to 0.2.0
  • spu to 0.2.5

[0.7.6] - 2022-09-08

Fixed

• Missing requirements in dev-requirements.txt.

[0.7.5] - 2022-09-05

Added

• SPU config param: throttle_window_size

[0.7.4] - 2022-09-01

Added

• PSI param: bucket_size

Changed

• SPU config param http_timeout_ms defaults to 120s.
• Bump dependencies
  • sf-heu to 0.1.3.2

[0.7.3] - 2022-08-29

Added

• Add pytorch backend for fl model for classification
• Add FL Dp strategy

Changed

• Update document
• Shrink docker image size
• Bump dependencies
  • sf-heu to 0.1.3.1

[0.7.2] - 2022-08-25

Changed

• Use secretflow-ray instead of ray.

[0.7.1] - 2022-08-25

Added

• Add steps_per_epoch parameter to callback function of SLBaseTFModel.

Changed

• Bump dependencies
  • sf-heu to 0.1.2
• Remove example of mixlr as mixlr is in official code already.

Fixed

• Fix psi docs.

[0.7.0] - 2022-08-23

Added

• Pytorch backend and FL strategy.
• SS pvalue.
• Horizontal NN global DP with RDP accountant.
• HEU supports encrypt with audit log.

Changed

• HEU uses c++ numpy api.
• Update ant pypi address.
• Complete cluster model deployment doc.
• Use multiprocess.cpu_count instead of multiprocessing.cpu_count for compatibility with macOS.

Fixed

• Fix sl gnn test.

[0.6.17] - 2022-08-11

Fixed

• fix model handle_data parties_length by adding partition_shape to dataframe

[0.6.16] - 2022-08-08

Added

• SS VIF.
• FL strategy: FedProx.
• Split GNN.

Changed

• Remove duplicated shape_spu_to_np & dtype_spu_to_np in spu.py.

[0.6.15] - 2022-08-02

Added

• Development and release docker.
• FL model strategy.
• Sigmoid approximation in python.
• SS LR.
• Verical FL LR.
• Auto ray.get for nested params with pyu objects in proxy decorated cls.
• Link desc in spu construction.

Changed

• Refactor datasets from oss instead of lfs.
• Many doc improvements.

Fixed

• SecureAggregator average when weights are multi-dimensions.

[0.6.14] - 2022-07-07

Added

• Vertical dp.

Changed

• Many docs improvements.

Fixed

• Increase H2A mask bits
• Include c++ lib in setup.

[0.6.13] - 2022-06-30

Added

• simulation.dataset for tutorial
• update tutorial of FL SL & SFXgboost
• add csv stream reader for FL

[0.6.12] - 2022-06-29

[0.6.11] - 2022-06-28

Fixed

• fix sf.init argument.

[0.6.10] - 2022-06-27

Fixed

• typos, grammatical errors, implicit in docs.

[0.6.9] - 2022-06-27

Added

• Secretflow shutdown.

Fixed

• LabelEncoder returns np.int dtype.

[0.6.8] - 2022-06-22

Added

• FlModel supports csv loader.

Changed

• Rename PPU to SPU.

[0.6.7] - 2022-06-16

Added

• MixLR demo.
• DP on split learning DNN.
• XGBoost tutorial.
• DataFrame and FedNdarray support astype method.

Changed

• Use lfs instead of http file.
• FL model requires model define no more when using load_model.
• dataframe.to_csv returns object ref.

Security

• Use more secure random.
• Complete security and not-for-production warning.

[0.6.6] - 2022-06-09

Added

• SplitDNN dp.
• Use lfs.
• XGboost tutorial.

Fixed

• DataFrame.to_csv returns object refs for further wait.

[0.6.5] - 2022-06-06

Added

• Runtime_config as input to utils.testing.cluster_def.
• SFXgboost optimization.
• Horizontal preprocessing
  • StardardScaler
  • KBinsDiscretizater
  • Binning
• Vertical preprocessing: WOE binning and substitution.
• HEU supports int/fxp data type.
• HEU Object supports slice and sum.
• Differential privacy.
• API docstrings.
• Custom pytree node to ppu.
• English docs.

Changed

• Remove default reveal in aggreagtor and compartor.
• Bump dependencies
  • jax to 0.3.7
  • sf-heu to 0.0.5
  • sf-ppu to 0.0.10.4
• FL model: up early stop from step to epoch.
• SecureAggregation uses powers of 2.
• Rename vdf partitions_dimensions to partition_columns.
• Use *args instead of args in aggregation for reducing ray task dependency.

Fixed

• FL model progress bug.
• train_test_split typo.

[0.6.4]

• Fix PPU dtype mismatch caused by JAX 32bit mode.
• Vertical PearsonR.

[0.6.3]

• FlModel/SlModel support model path dict.

[0.6.2]

• Upgrade sf-ppu version to 0.0.7.1

[0.6.1]

• More perfect HEU
• Split learning benchmark model
• SFXgboost for homo xgboost training

[0.6.0]

• FL: different batch size for different clients.
• Wait method for pyu objects.
• FLModel evaluate returns detailed metrics.

[0.0.6] - 2022-04-06

• PPU listen address.