v5.0.0

Major changes that affect users

Number of subbands changed from 128 to 512

To lower the current draw on the FPGA, the number of channels in the polyphase filter bankw was increased from 128 to 512. This means there are 512 subbands instead of 128. This mainly affects find_freq. We now provide start_freq and stop_freq optional arguments which allow then user to define the frequency range to sweep. This is more convenient to use than defining subbands.

Ref phase delay change

The low current firmware has longer processing times. This means the ref_phase_delay is larger than in past releases. The user can remeasure the full delay of the system with estimate_phase_delay. Note that this only changes the value in the current Rogue session. If the Rogue session is killed, it will not be reloaded. The config file can be updated with this new value. This will set the correct value for phase delay on setup(). The user can manually set the processing delay with set_processing_delay().

Tracking setup change lms_gain to feedback_gain

The optional argument lms_gain is now an “expert only” variable. Use feed_back gain instead. See this jupyter notebook and the docstring for more details.

Example script

Mitch's example script that walks users through using the system.
slaclab/pysmurf

Pull Requests Since v4.3.0

Client

Interface-change

1. #621 - Find freq eta scan speedup

Bug

1. #621 - Find freq eta scan speedup
2. #606 - Fixed no_dir error during setup
3. #604 - Adds a few register_file calls for noise plots

Enhancement

1. #621 - Find freq eta scan speedup
2. #616 - take_noise_psd return_noise_params expanding

Pull Request Details

Find freq eta scan speedup

Author:	Edward [email protected]
Date:	Wed Mar 24 14:05:36 2021 -0700
Pull:	#621 (2144 additions, 217 deletions, 12 files changed)
Branch:	slaclab/find-freq-eta-scan-speedup
Labels:	bug, client, enhancement, interface-change

Notes:

Issue

This PR updates the pysmurf software for the low current FW.

Description

[Describe what this PR does here]

Does this PR break any interface?

• Yes
• No

Which interface changed?

Number of subbands changed from 128 to 512

To lower the current draw on the FPGA, the number of channels in the polyphase filter bankw was increased from 128 to 512. This means there are 512 subbands instead of 128. This mainly affects find_freq. We now provide start_freq and stop_freq optional arguments which allow then user to define the frequency range to sweep. This is more convenient to use than defining subbands.

Ref phase delay change

The low current firmware has longer processing times. This means the ref_phase_delay is larger than in past releases. The user can remeasure the full delay of the system with estimate_phase_delay. Note that this only changes the value in the current Rogue session. If the Rogue session is killed, it will not be reloaded. The config file can be updated with this new value. This will set the correct value for phase delay on setup(). The user can manually set the processing delay with set_processing_delay().

Tracking setup change lms_gain to feedback_gain

The optional argument lms_gain is now an “expert only” variable. Use feed_back gain instead. See this jupyter notebook and the docstring for more details.

What was the interface before the change

[Describe here how was the original interface]

What will be the new interface after the change

[Describe here how is the new interface, and what the difference with the original interface]

---

take_noise_psd return_noise_params expanding

Author:	Edward [email protected]
Date:	Fri Mar 19 14:33:52 2021 -0700
Pull:	#616 (21 additions, 11 deletions, 1 files changed)
Branch:	slaclab/issue614
Issues:	#614
Labels:	client, enhancement

Notes:

Issue

This resolves #614

Description

Returns all the fit parameters

Does this PR break any interface?

• Yes
• No

The dimension of the return parameter is now larger.

Which interface changed?

[Indicate here what is the interface that changed, preferable with a link to the code where the interface is defined]

What was the interface before the change

[Describe here how was the original interface]

What will be the new interface after the change

[Describe here how is the new interface, and what the difference with the original interface]

---

Fixed no_dir error during setup

Author:	Edward [email protected]
Date:	Tue Feb 16 15:46:38 2021 -0800
Pull:	#606 (49 additions, 49 deletions, 1 files changed)
Branch:	slaclab/issue600
Issues:	#600
Labels:	bug, client

Notes:

Issue

Resolves #600 and https://github.com/simonsobs/smurf-issues/issues/24

Description

Figured out there was an error during setup in this branch that happened because some code was indented once to many times, which caused some of the setup not to be run if no_dir=True. This PR de-indents it.

Does this PR break any interface?

• Yes
• No

---

Adds a few register_file calls for noise plots

Author:	Edward [email protected]
Date:	Tue Feb 16 15:45:37 2021 -0800
Pull:	#604 (12 additions, 0 deletions, 1 files changed)
Branch:	slaclab/register_noise_plots
Issues:	#603
Labels:	bug, client

Notes:

Issue

Issue #603

Description

Adds a few register_file statements to take_noise_psd and correctly sets the publisher action.

I'm not exactly sure which branch I should merge this into... I think we have some people running on the main branch, but if find-freq-eta-scan-speedup will be merged soon then I think it's ok.

Does this PR break any interface?

• Yes
• No

---

v4.3.0

Pull Requests Since v4.2.1

Client

Bug

1. #595 - Bug in emulator period data type handling
2. #599 - Update set_default_pv to handle the new RuntimeError exception
3. #613 - Change units in read_stream_data

Enhancement

1. #617 - Update tracking_setup, add feedback_gain and documentation
2. #587 - Issue586 - Changing name of NEI_to_NEP to get_NEI_to_NEP_factor
3. #590 - Adding retry attempt to caget
4. #607 - Fixes #605.

Core

Interface-change

1. #553 - Add external trigger (from timing system) to the downsampler
2. #593 - Smurf Server Publisher modifications

Bug

1. #615 - Update smurf-rogue base image to version R2.8.3

Enhancement

1. #553 - Add external trigger (from timing system) to the downsampler
2. #593 - Smurf Server Publisher modifications

Unlabeled

1. #597 - Updating smurf_cmd.py
2. #602 - Adding function for finding probe tone gap

Pull Request Details

Update tracking_setup, add feedback_gain and documentation

Author:	Edward [email protected]
Date:	Tue Mar 2 18:09:31 2021 -0800
Pull:	#617 (49 additions, 4 deletions, 1 files changed)
Branch:	slaclab/feedback-doc
Issues:	#592
Labels:	client, enhancement

Notes:

Issue

This PR resolves #592.

Description

Adds doc strings for LMS gain. Also introduces feedback_gain parameter to tracking

Does this PR break any interface?

• Yes
• No

Which interface changed?

[Indicate here what is the interface that changed, preferable with a link to the code where the interface is defined]

What was the interface before the change

[Describe here how was the original interface]

What will be the new interface after the change

[Describe here how is the new interface, and what the difference with the original interface]

---

Update smurf-rogue base image to version R2.8.3

Author:	Jesus Vasquez [email protected]
Date:	Fri Feb 26 19:38:51 2021 -0800
Pull:	#615 (1 additions, 1 deletions, 1 files changed)
Branch:	slaclab/ESCRYODET-728
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-728
Labels:	bug, core

Notes:

Issue

This PR resolves ESCRYODET-728.

Description

This PR updates Rogue to version v4.11.7. This version of Rogue contains a fix for the unbound memory usage increase described in ESCRYODET-728.

Does this PR break any interface?

• Yes
• No

---

Change units in read_stream_data

Author:	Edward [email protected]
Date:	Fri Feb 26 13:31:01 2021 -0800
Pull:	#613 (1 additions, 1 deletions, 1 files changed)
Branch:	slaclab/issue612
Issues:	#612
Labels:	bug, client

Notes:

Issue

This PR resolves #612

Description

Corrects doc-string for read_stream_data.

Does this PR break any interface?

• Yes
• No

---

Fixes #605.

Author:	Edward [email protected]
Date:	Wed Feb 17 14:44:02 2021 -0800
Pull:	#607 (6 additions, 2 deletions, 1 files changed)
Branch:	slaclab/issue605
Labels:	client, enhancement

Notes:

Issue

This PR resolves #605

Description

Adds an output directory keyword (output_dir) to the analyze_iv_from_file so we can dump the analysis results to another location.

Does this PR break any interface?

• Yes
• No

Which interface changed?

Extra keyword in analyze_iv_from_file

What was the interface before the change

IV analysis outputs could only be saved in the folder structure on the smurf server. Would throw errors if the files had been moved.

What will be the new interface after the change

Extra keyword in analyze_iv_from_file means you can tell it where to save the results.

---

Smurf Server Publisher modifications

Author:	Edward [email protected]
Date:	Tue Feb 16 16:16:38 2021 -0800
Pull:	#593 (36 additions, 4 deletions, 1 files changed)
Branch:	slaclab/issue588
Issues:	#588
Labels:	core, enhancement, interface-change

Notes:

Issue

This PR resolves #588.

Description

This PR adds the following changes to the Publisher class as requested in #588:

• Metadata publishes will now have msgtype set to metadata, instead of general,
• When publishing metadata, the content of the data field will now be key, value pairs, in JSON format, containing the Rogue register path (path), the register value (value), and the value type (type).

I tested this changes, with the default variable group, and modifying the _backend_null method to print the received messages, and obtained this:

{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3430576, "seq_no": 1565, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.ChannelMapper.NumChannels", "value": 1, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3466105, "seq_no": 1566, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FrameRxStats.FrameLossCnt", "value": 123, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3493185, "seq_no": 1567, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.PreDataEmulator.Amplitude", "value": 65535, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3505917, "seq_no": 1568, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FileWriter.TotalSize", "value": 0, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3531475, "seq_no": 1569, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.PostDataEmulator.Amplitude", "value": 4294967295, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.355861, "seq_no": 1570, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FileWriter.CurrentSize", "value": 0, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3562307, "seq_no": 1571, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FrameRxStats.FrameSize", "value": 8320, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3567724, "seq_no": 1572, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.PostDataEmulator.Offset", "value": 0, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3584292, "seq_no": 1573, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FileWriter.FrameCount", "value": 0, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3591156, "seq_no": 1574, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FrameRxStats.BadFrameCnt", "value": 0, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.359428, "seq_no": 1575, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.PreDataEmulator.Offset", "value": 0, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.3594973, "seq_no": 1576, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FrameRxStats.FrameCnt", "value": 791596, "type": "int"}}
{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611797885.36481, "seq_no": 1577, "type": "metadata", "payload": {"path": "AMCc.SmurfProcessor.FrameRxStats.FrameOutOrderCnt", "value": 0, "type": "int"}}

I added a numpy array variable (np.zeros(5)) for testing and got this message:

{"host": "smurf-srv06", "id": "undeclared", "script": "PySmurf_Server", "time": 1611858311.23576, "seq_no": 1239, "type": "metadata", "payload": {"path": "AMCc.SmurfApplication.NPArrayTest", "value": [0.0, 0.0, 0.0, 0.0, 0.0], "type": "ndarray"}}

Additionally, I added try-exception statement when converting the message to JSON to catch exceptions. If an exception occurs, the error message is logged.

Does this PR break any interface?

• Yes
• No

Which interface changed?

The content of the metadata send to the Publisher

What was the interface before the change

• msgtype was general,
• data was a string containing register_path=register_value

What will be the new interface after the change

• msgtype is now metadata,
• data is now formed by the following key, value pairs in JSON format:
  • path : Register path,
  • value : Register value (numpy arrays are converted to lists)
  • type : the value type, as a string.

---

Add external trigger (from timing system) to the downsampler

Author:	Edward [email protected]
Date:	Tue Feb 16 16:01:54 2021 -0800
Pull:	#553 (147 additions, 42 deletions, 4 files changed)
Branch:	slaclab/ESCRYODET-472
Jira:	https://jira.slac.stanford.edu/issues/ESCRYO...

v4.2.1

Pull Requests Since v4.2.0

Client

Bug

1. #585 - Bug fix: During setup(), check if set_defaults_pv succeeded before continue with setup.

Enhancement

1. #581 - pre/post data emulator

Core

Bug

1. #585 - Bug fix: During setup(), check if set_defaults_pv succeeded before continue with setup.
2. #583 - Fix drop of processed frames into data files and custom transmitters, and improve performance

Enhancement

1. #584 - Add SmurfProcessor filter and unwrapper tests to the Github Action pysmurf core CI tests
2. #583 - Fix drop of processed frames into data files and custom transmitters, and improve performance

Pull Request Details

Fix drop of processed frames into data files and custom transmitters, and improve performance

Author:	Jesus Vasquez [email protected]
Date:	Tue Jan 19 14:56:19 2021 -0800
Pull:	#583 (23 additions, 15 deletions, 1 files changed)
Branch:	slaclab/ESCRYODET-796
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-796
Labels:	bug, core, enhancement

Notes:

Issue

This PR resolves ESCRYODET-796.

Description

This PR fixes a bug in the data processor which was causing processed frames to be dropped before going to the FileWriter as well as Custom transmitter clients (like SO streamers for example).

This PR also improved the performance when moving processed data to the thread that passes the frame to the FileWriter and Custom transmitter, so now it can handle frames rates up to ~14-20 frames/s (respect to ~ 1 kframe/sin the previous version).

Details of the test and results can be found here:
https://jira.slac.stanford.edu/browse/ESCRYODET-796

Does this PR break any interface?

• Yes
• No

---

pre/post data emulator

Author:	Shawn Henderson [email protected]
Date:	Tue Jan 19 11:47:04 2021 -0800
Pull:	#581 (210 additions, 0 deletions, 2 files changed)
Branch:	slaclab/issue580
Issues:	#580, #292
Labels:	client, enhancement

Notes:

Issue

This PR resolves #580 and #292 .

Description

Introduces the pre/post data emulator pysmurf hooks.

Does this PR break any interface?

• Yes
• No

Which interface changed?

[Indicate here what is the interface that changed, preferable with a link to the code where the interface is defined]

What was the interface before the change

[Describe here how was the original interface]

What will be the new interface after the change

[Describe here how is the new interface, and what the difference with the original interface]

---

Bug fix: During setup(), check if set_defaults_pv succeeded before continue with setup.

Author:	Jesus Vasquez [email protected]
Date:	Tue Jan 19 09:01:21 2021 -0800
Pull:	#585 (38 additions, 31 deletions, 3 files changed)
Branch:	slaclab/ESCRYODET-791
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-791
Labels:	bug, client, core

Notes:

Issue

This PR resolves ESCRYODET-791.

Description

This PR fixes a bug in setup(), which was continuing with the setup process when set_defaults_pv fails. It also reduces the number of number of times the server retries to load defaults to 4, to be equivalent to the 400s timeout the client uses.

This PR also adds a fix to profile_band.py test, which should check if setup() succeed before continuing with the test.

Does this PR break any interface?

• Yes
• No

---

Add SmurfProcessor filter and unwrapper tests to the Github Action pysmurf core CI tests

Author:	Jesus Vasquez [email protected]
Date:	Fri Jan 15 15:06:58 2021 -0800
Pull:	#584 (65 additions, 7 deletions, 6 files changed)
Branch:	slaclab/ESCRYODET-799
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-799
Labels:	core, enhancement

Notes:

Issue

This PR resolves ESCRYODET-799.

Description

This PR adds the test scripts to validate the SmurfProcessor filter and unwrapper code (tests/core/validate_filter.py and tests/core/validate_unwrapper.py) to the Github Action pysmurf core CI tests.

Does this PR break any interface?

• Yes
• No

---

v4.2.0

Major changes that affect users:

#564 which updated the smurf-rogue base image version to R2.8.2, introduced a rogue version with an incompatibility interface changes to previous versions. Rogue changed its underlaying ZMQ interface between server and remote clients (i.e. GUI) applications, this means that pysmurf server applications using this version (or higher) need to use smurf-rogue version R2.8.2 or higher in order to connect a remote GUI.

---

Pull Requests Since v4.1.0

Client

Interface-change

1. #572 - Remove local copy of the schema python package
2. #570 - Set the type of the FrameCnt local variable to UInt64

Bug

1. #550 - Critical fix for decode_data function that eliminates glitching seen at NIST.
2. #542 - Resolving bugs in analyze_noise_vs_bias
3. #555 - Fix for resetting RF DACs after first setup in latest software versions.
4. #552 - PySmurf should never set TestBiasDacCtrlRegCh to 0x0
5. #532 - Fixing doc string for get_tes_bias_bipolar_array
6. #530 - Swapping UC/DC att call at beginning of estimate_phase_delay
7. #557 - Fixes bug that was causing find_freq to crash if not given a start/stop_freq or subband arg.
8. #570 - Set the type of the FrameCnt local variable to UInt64
9. #547 - Change register_file default
10. #509 - Expanded refPhaseDelay config file limits from 4 to 5 bits.

Enhancement

1. #527 - Thermal test scripts and bug fixes for long Westpak run with 6x carriers.
2. #572 - Remove local copy of the schema python package
3. #537 - Issue524
4. #548 - Adding flag to skip IV analysis functions after running S.run_iv()
5. #533 - changing do_plot to make_plot in smurf_util
6. #573 - Pin version of the python packages used in the client docker image
7. #539 - RTM reset

Core

Interface-change

1. #569 - Disable insertion of TES bias values in the SMuRF packet header in SW
2. #577 - SMuRF header: Use Unix Epoch time for word 6
3. #570 - Set the type of the FrameCnt local variable to UInt64
4. #564 - Update smurf-rogue base image to version R2.8.2

Bug

1. #565 - Fix mutex lock scopes related to the unwrapper data
2. #536 - Fix SIGABRT on DualDataBuffer destruction
3. #570 - Set the type of the FrameCnt local variable to UInt64
4. #567 - Add more restrictions to the find method used to check if JesdHealth exists
5. #564 - Update smurf-rogue base image to version R2.8.2

Enhancement

1. #560 - Migrate CI/CD script from Travis to GitHub Actions
2. #569 - Disable insertion of TES bias values in the SMuRF packet header in SW
3. #577 - SMuRF header: Use Unix Epoch time for word 6

Other

Bug

1. #566 - Github Actions: Fix release generation script, which was still referencing Travis env vars
2. #579 - Fix GitHub Action job used to generate release notes
3. #578 - Fix the GitHub token used during the generation of release notes

Enhancement

1. #562 - Update version of the verify-pr-label-action action to support PRs from forks

Pull Request Details

Fix GitHub Action job used to generate release notes

Author:	Jesus Vasquez [email protected]
Date:	Fri Dec 18 13:40:04 2020 -0800
Pull:	#579 (2 additions, 0 deletions, 1 files changed)
Branch:	slaclab/fix-github-action-release-2
Labels:	bug

Notes:

Issue

No issue associated to this PR.

Description

The release note generation script needs to be run with a deep checkout, so that is can find the tags.

Does this PR break any interface?

• Yes
• No

---

Fix the GitHub token used during the generation of release notes

Author:	Jesus Vasquez [email protected]
Date:	Fri Dec 18 11:31:58 2020 -0800
Pull:	#578 (1 additions, 1 deletions, 1 files changed)
Branch:	slaclab/fix-github-action-release
Labels:	bug

Notes:

Issue

No issue associated to this PR.

Description

Fix the GitHub token used during the generation of release notes.

Does this PR break any interface?

• Yes
• No

---

SMuRF header: Use Unix Epoch time for word 6

Author:	Jesus Vasquez [email protected]
Date:	Fri Dec 18 10:57:59 2020 -0800
Pull:	#577 (3 additions, 3 deletions, 2 files changed)
Branch:	slaclab/issue505
Issues:	#505
Labels:	core, enhancement, interface-change

Notes:

Issue

This PR resolves #505

Description

Use the UNIX Epoch Time (expressed in nanoseconds) in word 6 of the SMuRF header.

Does this PR break any interface?

• Yes
• No

Which interface changed?

Word 6 in the SMuRF packet header.

What was the interface before the change

Word 6 contained the time obtained using steady_clock, which was a monotonic clock that will never be adjusted (in nanoseconds).

What will be the new interface after the change

Word 6 now contains the time obtained using system_clock instead, which is the wall clock time from the system-wide realtime clock, converted to time since Epoch in nanoseconds.

---

PySmurf should never set TestBiasDacCtrlRegCh to 0x0

Author:	Shawn Henderson [email protected]
Date:	Thu Dec 17 17:40:00 2020 -0800
Pull:	#552 (20 additions, 7 deletions, 1 files changed)
Branch:	slaclab/issue-551
Issues:	#551
Labels:	bug, client

Notes:

Issue

This PR resolves #551 .

Description

PySmurf should never set TestBiasDacCtrlRegCh to 0x0, this will cause it to rail to -10V. Power on default is 0xE, enabled is 0x2. Let's make PySmurf only ever set 0x2 or 0xE. The command will default to 0x2 if it's anything else.

Does this PR break any interface?

• Yes
• No

---

Disable insertion of TES bias values in the SMuRF packet header in SW

Author:	Jesus Vasquez [email protected]
Date:	Wed Dec 9 09:28:55 2020 -0800
Pull:	#569 (2 additions, 72 deletions, 9 files changed)
Branch:	slaclab/ESCRYODET-763
Issues:	#559
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-763
Labels:	core, enhancement, interface-change

Notes:

Issue

This PR resolves #559 and ESCRYODET-763.

Description

This PR disables the insertion of TES bias values in the SMuRF packet header in SW, and let the values that now come from the FW application verbatim.

This PR also removes the code that was overwriting the version number in the SMuRF packet header, leaving the value that comes from FW verbatim as well.

Does this PR break any interface?

• Yes
• No

Which interface changed?

The TES bias values (offset 8) and protocol version number (offset 0) in the SMuRF packet header.

What was the interface before the change

• The TES bias values in the SMuRF packet header were inserted in SW based on the values TES bias control registers as:

v1 = self.FpgaTopLevel.AppTop.AppCore.RtmCryoDet.RtmSpiMax.TesBiasDacDataRegCh[{(2*idx)+2}]		
v2 = self.FpgaTopLevel.AppTop.AppCore.RtmCryoDet.RtmSpiMax.TesBiasDacDataRegCh[{(2*idx)+1}]
tesBias[idx] = (v1 - v2) // 2

• The protocol version value in the SMuRF packet header was set to 1.

What will be the new interface after the change

The TES bias value and the protocol version number in the SMuRF packet header are not longer written in SW, so they now come verbatim from the FW application. So, this change is backward compatible.

---

Set the type of the FrameCnt local variable to UInt64

Author:	Jesus Vasquez [email protected]
Date:	Wed Dec 9 08:58:42 2020 -0800
Pull:	#570 (4 additions, 2 deletions, 2 files changed)
Branch:	slaclab/ESCRYODET-726
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-726
Labels:	bug, client, core, interface-change

Notes:

Issue

This PR resolves ESCRYODET-726.

Description

Set the type of the FrameCnt local variable to UInt64 in order to support it trough the EPICS interface. Without this change, the variable it treated as a 32-bit value and it will overflow after a few day of data streaming.

The EPICS server will treat this value as a string now, so the client command that reads this variable was also updated to read the value as as string, and convert it back to int.

Does this PR break any interface?

• Yes
• No

Which interface changed?

The type of the variable ${epics_prefix}:AMCc:SmurfProcessor:FrameRxStats:FrameCnt through the EPICS interface.

What was the interface before the change

The variable was treated as a 32-bit number by the EPICS server.

What will be the new interface after the change

Now, the EPICS server sees this value as a 64-bit value, and treat it as a string.

However, the client command was updated as well, so a pysmurf client application won't noticed any difference, making this change backward compatible from the pysmurf point of view.

---

Remove local copy of the schema python package

Author:	Edward [email protected]
Date:	Tue Dec 8 16:59:44 2020 -0800
Pull:	#572 (1 additions, 776 deletions, 2 files changed)
Branch:	slaclab/issue37...

v4.1.0

Pull Requests Since v4.0.0

Client

Interface-change

1. #389 - Resolves all sphinx_build errors and warnings, upgrades schema to 0.7.1, adds new SmurfConfigPropertiesMixin, expands documentation to include core code and user guide outline.
2. #373 - Fix noise_vs_bias band requirement, offline mode of analyze_noise_vs_bias, play_tes_bipolar_waveform non-continuous, n_samp for read_stream_data
3. #343 - Changing default of high_current_mode in overbias_tes
4. #428 - Create new rogue variable EnabledBays with a list of enabled bays

Bug

1. #373 - Fix noise_vs_bias band requirement, offline mode of analyze_noise_vs_bias, play_tes_bipolar_waveform non-continuous, n_samp for read_stream_data
2. #354 - Fixes the noise_vs_tone function. The plotting routine had a bug in it which would cause it to crash.
3. #491 - IV functions renamed, now using tone_power and nsamp instead of varying names.
4. #367 - Fixes the noise_vs_tone analysis script - wasn't handling channel assignment properly
5. #451 - Fixes wrong hardcoded clock rate in get_flux_ramp_freq, adds docstrings to get_flux_ramp_freq and set_flux_ramp_freq.
6. #343 - Changing default of high_current_mode in overbias_tes
7. #488 - Fix set_defaults_pv method to determine if the system configuration sequence ended
8. #350 - Fixes readthedocs auto builds
9. #442 - Fixing which_bays backwards compatibility
10. #441 - Revert "Backwards compatibility fix for which_bays"
11. #440 - Backwards compatibility fix for which_bays
12. #430 - Fixes readthedocs badge and docstring convention note in README.
13. #473 - Fw refactor
14. #422 - Adding scaling relation for faster frequencies in play_sine_tes
15. #496 - Passing bias_line_resistance from run_iv to analyze_iv
16. #409 - Changed publish location for mask.txt file
17. #436 - Fixing a couple of fstrings with missing fs
18. #501 - Adding packaging to requirements.txt.

Enhancement

1. #389 - Resolves all sphinx_build errors and warnings, upgrades schema to 0.7.1, adds new SmurfConfigPropertiesMixin, expands documentation to include core code and user guide outline.
2. #485 - JESD health checks
3. #454 - Migrates remaining pysmurf configuration parameters into the SmurfConfigPropertiesMixin class as properties.
4. #420 - Comprehensive docstrings for SmurfConfig class.
5. #445 - loopback mode for profile_band, eta_phase estimator using IQ noise, fast data taking example scripts
6. #229 - Issue95 - Dan's RF initialization and enhancements for thermal testing
7. #481 - Adds a profile_slot script which extends profile_band to an entire slot, adds docstrings to smurf_atca_monitor submodule.
8. #474 - Use new registers which give the state of the configuration process during setup
9. #373 - Fix noise_vs_bias band requirement, offline mode of analyze_noise_vs_bias, play_tes_bipolar_waveform non-continuous, n_samp for read_stream_data
10. #433 - Rogue3 thermal test, shawnhammer upgrades, hardware logging docstrings.
11. #494 - Stephen's AMC testing script.
12. #384 - Introduce optional variable to turn off phase highlighting in find_peak plotting
13. #361 - Add core & user guide to pysmurf autodocs
14. #390 - Moves the running part of profile_band into a callable function.
15. #396 - Fixes docstrings to have valid reStructuredText (RST)
16. #269 - Issue266 - function to find bias groups by playing a sine wave on the TES bias
17. #317 - Publisher actions
18. #429 - pysmurf cfg file can be specified in smurf_startup.cfg, no longer requiring an init script.
19. #403 - Determine which bays are on from hardware, not pysmurf cfg.
20. #394 - Reorders imports in compliance with pep8.
21. #357 - Fixing offline code for reading ADC/DAC data.
22. #401 - Eliminates utf8 byte array return type option for str register gets.
23. #370 - Adding high_current_mode bool to identify_bias_group
24. #452 - Adds test script to help check for deterministic latency & fixes script_to_run bug in shawnhammer.
25. #456 - Add checks and fixes to help solve configuration issues
26. #428 - Create new rogue variable EnabledBays with a list of enabled bays
27. #410 - Added streaming context manager
28. #380 - Use list instead of np.array in read_stream_data for speed up.
29. #372 - Adding overbias_wait to noise_vs_bias
30. #392 - Add travis job to test documentation build for errors and warnings
31. #378 - Add the schema python package to the pysmurf.client docker image
32. #496 - Passing bias_line_resistance from run_iv to analyze_iv

Core

Interface-change

1. #389 - Resolves all sphinx_build errors and warnings, upgrades schema to 0.7.1, adds new SmurfConfigPropertiesMixin, expands documentation to include core code and user guide outline.
2. #428 - Create new rogue variable EnabledBays with a list of enabled bays

Bug

1. #430 - Fixes readthedocs badge and docstring convention note in README.
2. #499 - Fix broken link
3. #477 - Update smurf-rogue base image to version R2.8.1

Enhancement

1. #389 - Resolves all sphinx_build errors and warnings, upgrades schema to 0.7.1, adds new SmurfConfigPropertiesMixin, expands documentation to include core code and user guide outline.
2. #485 - JESD health checks
3. #420 - Comprehensive docstrings for SmurfConfig class.
4. #448 - Optimize performance of the data processing code
5. #396 - Fixes docstrings to have valid reStructuredText (RST)
6. #438 - Server hardware auto-detection: abort server startup if unsupported HW is detected
7. #458 - Move the start_server.sh argument parsing and usage function to server_common.sh
8. #394 - Reorders imports in compliance with pep8.
9. #459 - Update elastic buffer check, and add new variable SystemConfigured
10. #460 - Move all the initialization routines to server_common.sh
11. #456 - Add checks and fixes to help solve configuration issues
12. #461 - Catch rogue-level exception during configuration
13. #428 - Create new rogue variable EnabledBays with a list of enabled bays
14. #392 - Add travis job to test documentation build for errors and warnings

Other

Bug

1. #342 - Fixes setup.py to reflect rogue4 change in pysmurf code directory structure
2. #503 - Update reference to the old master branch to main
3. #344 - Fixes location of version_file in pysmurf package setup.cfg
4. #339 - Fix the releaseGen.py to catch issues numbers that are followed by a dot.

Enhancement

1. #347 - Crate issues and PR templates
2. #360 - Add Ed and Shawn as code owner of GitHub files
3. #400 - Adding readthedocs to readme

Unlabeled

1. #219 - Accelerated processing of TES bytes
2. #268 - Move version to pysmurf, instead of pysmurf.client

Pull Request Details

Update reference to the old master branch to main

Author:	Edward [email protected]
Date:	Tue Aug 11 09:26:49 2020 -0700
Pull:	#503 (2 additions, 2 deletions, 1 files changed)
Branch:	slaclab/issue502
Issues:	#502
Labels:	bug

Notes:

Issue

This PR resolves #502

Description

Fix the reference to the deprecated master branch to main.

Does this PR break any interface?

• Yes
• No

---

Adds a profile_slot script which extends profile_band to an entire slot, adds docstrings to smurf_atca_monitor submodule.

Author:	Shawn Henderson [email protected]
Date:	Tue Aug 11 00:48:28 2020 -0700
Pull:	#481 (807 additions, 122 deletions, 6 files changed)
Branch:	slaclab/issue480
Labels:	client, enhancement

Notes:

Issue

This PR resolves #480, #478, #358, and #497 .

Description

Introduces the script profile_slot.py which tests all the bands in a slot. Adds docstrings for smurf_atca_monitor submodule routines and for the smurf_util submodule estimate_phase_delay routine..

Does this PR break any interface?

• Yes
• No

Which interface changed?

estimate_phase_delay now returns an output.

What was the interface before the change

estimate_phase_delay used to not have a return

What will be the new interface after the change

Now it returns refPhaseDelay, refPhaseDelayFine, processing_delay_us, dsp_corr_delay_us

---

Fix broken link

Author:	Shawn Henderson [email protected]
Date:	Mon Aug 10 23:57:00 2020 -0700
Pull:	#499 (1 additions, 1 deletions, 1 files changed)
Branch:	slaclab/issue498
Issues:	#498
Labels:	bug, core

Notes:

Issue

This PR resolves #498

Description

Fix a broken link on README.SmurfProcessor.md to the Data Transmitter documentation.

Does this PR break any interface?

• Yes
• No

---

Adding packaging to requirements.txt.

Author:	Edward [email protected]
Date:	Mon Aug 10 15:12:17 2020 -0700
Pull:	#501 (1 additions, 0 deletions, 1 files changed)
Branch:	slaclab/packagingreq
Labels:	bug, client

Notes:

Issue

No existing issue.

Description

Just adds the packaging module to requirements.txt.

---

Stephen's AMC testing script.

Author:	Edward [email protected]
Date:	Fri Aug 7 17:43:43 2020 -0700
Pull:	#494 (355 additions, 50 deletions, 4 files changed)
Branch:	slaclab/stephen-script
Labels:	client, enhancement

Notes:

Issue

This PR doesn't resolve any issues but adds a script that Stephen wrote to qualify AMCs.

Description

Adds new startup cfg file for qualifying AMCs, cfg_files/rflab/rflab_smurf_startup_stephen_test.cfg. Adds _reg suffix to all smurf_atca_monitor register path variables. Ad...

v4.0.0

Pull Requests Since v4.0.0-rc21

Client

Interface-change

1. #327 - Adding and cleaning up comments - also fixes bug in take_noise_psd plot label

Bug

1. #310 - Script to automatically generate a summary html
2. #327 - Adding and cleaning up comments - also fixes bug in take_noise_psd plot label
3. #291 - Pton 2carriercfg 20200306
4. #284 - Fix the test_loopback script
5. #328 - Scratch update for eyy - bug in profile_band loading of .npy files
6. #308 - Issue306 - fixing bias bump - introduce estimate_bias_voltage function that estimates where to bias using an input IV curve
7. #271 - Issue270 - incorrect comments for TES high/low current mode
8. #301 - Fixing passing for setup_flux_ramp argument in check_lock.
9. #294 - Intel enpirion to em22xx
10. #304 - Fix TES bias extraction
11. #336 - Bug fix: trying to set a register to an empty array is not supported
12. #334 - Fix pysmurf-client docker image, by adding the cryo user's home directory

Enhancement

1. #310 - Script to automatically generate a summary html
2. #298 - Synching ucsd with master
3. #229 - Issue95 - Dan's RF initialization and enhancements for thermal testing
4. #272 - Issue264 - function to find bias point
5. #327 - Adding and cleaning up comments - also fixes bug in take_noise_psd plot label
6. #313 - Issue311 - remove amplifier bias hardcode
7. #278 - Nist cmb pre release
8. #269 - Issue266 - function to find bias groups by playing a sine wave on the TES bias
9. #280 - Adding read_stream_data_gcp_save for backwards compatibility.
10. #277 - Issue276 - estimate_flux_ramp_amp instead of LMS freq
11. #309 - exposing find_freq optional arguments for finding peaks
12. #308 - Issue306 - fixing bias bump - introduce estimate_bias_voltage function that estimates where to bias using an input IV curve
13. #295 - Add argument to publish streamed data file
14. #282 - Catch & report if saturating ADC during full_band_resp
15. #288 - Add the GitPython python package to the pysmurf-client docker image

Core

Bug

1. #334 - Fix pysmurf-client docker image, by adding the cryo user's home directory
2. #287 - Update smurf-rogue base image to version R2.6.1
3. #290 - Add missing imports

Enhancement

1. #285 - Detect type of carrier and enable the EM22xx device if it is C03+
2. #305 - Auto detect presence, type and version of AMCs during startup
3. #307 - Fix formatting of the log message from the start_server.sh
4. #314 - Expose the server startup script name and list of arguments
5. #316 - Adds default server_port when IP Address is not defined
6. #318 - Re-enable the "enable_em22xx" flag from the top script.

Other

Enhancement

1. #319 - Add release notes generation script
2. #315 - Fix shawnhammer smurf server detection
3. #323 - Setup Github workflow to auto label PR as core or client
4. #333 - Extract issue references from PR body
5. #325 - Verify that all PRs are labeled with bug, enhancement, or interface-change
6. #332 - Define code owners
7. #302 - Makes shawnhammer shutdown smurf-streamer dockers

Pull Request Details

Bug fix: trying to set a register to an empty array is not supported

Author:	Shawn Henderson [email protected]
Date:	Sun Mar 22 09:02:01 2020 -0700
Pull:	#336 (2 additions, 2 deletions, 1 files changed)
Branch:	slaclab/issue335
Issues:	#335
Labels:	bug, client

Notes:

In this case this was done during SmurfControl.setup, but this is not allowed in general.
In this case we can set the Mask to an single element [0] instead.

This resolves #335 .

---

Fix pysmurf-client docker image, by adding the cryo user's home directory

Author:	Jesus Vasquez [email protected]
Date:	Sat Mar 21 21:34:50 2020 -0700
Pull:	#334 (2 additions, 2 deletions, 2 files changed)
Branch:	slaclab/ESCRYODET-630
Issues:	#324
Jira:	https://jira.slac.stanford.edu/issues/ESCRYODET-630
Labels:	bug, client, core

Notes:

This PR resolves #324.

The base docker image smurf-base didn't contained the home directory for the cryo user. So, when the container was running the directory /home/cryo didn't exit. It seems that ipython write access to the home directory for its tab complete feature to work. That was fix in this PR, which is available in version tidair/smurf-base:R1.1.1.

This was affecting only the client docker image. However, I updated its base docker image as well to include that fix. It is available in the version tidair/smurf-rogue:R2.6.2.

In this PR I update the two base docker images to these new versions.

Jira ticket: https://jira.slac.stanford.edu/browse/ESCRYODET-630

---

Define code owners

Author:	Shawn Henderson [email protected]
Date:	Sat Mar 21 11:02:49 2020 -0700
Pull:	#332 (18 additions, 0 deletions, 1 files changed)
Branch:	slaclab/issue330
Issues:	#330
Labels:	enhancement

Notes:

This PR resolves #330.

It defines Shawn and Ed as defaults owners. And Ryan and I will be owners for the code related to the core.

The only goal of defining owner users, is to automatically assign the owner of specific file as reviewer when a PR modifies the files he owns.

We can add others users as well. I just did this as an initial list.

---

Issue306 - fixing bias bump - introduce estimate_bias_voltage function that estimates where to bias using an input IV curve

Author:	Shawn Henderson [email protected]
Date:	Sat Mar 21 10:53:52 2020 -0700
Pull:	#308 (32 additions, 12 deletions, 2 files changed)
Branch:	slaclab/issue306
Labels:	bug, client, enhancement

Notes:

---

Scratch update for eyy - bug in profile_band loading of .npy files

Author:	Shawn Henderson [email protected]
Date:	Sat Mar 21 10:46:44 2020 -0700
Pull:	#328 (31 additions, 14 deletions, 1 files changed)
Branch:	slaclab/ed-scratch-update-20200320
Labels:	bug, client

Notes:

np.load needs an allow_pickle=True statement for saved dicts.

---

Extract issue references from PR body

Author:	Jesus Vasquez [email protected]
Date:	Sat Mar 21 00:10:20 2020 -0700
Pull:	#333 (20 additions, 3 deletions, 1 files changed)
Branch:	slaclab/detect_issues
Issues:	#123
Labels:	enhancement

Notes:

This change will scan the pull request body for the #123 pattern in an attempt to find references to issues.

---

Add release notes generation script

Author:	Jesus Vasquez [email protected]
Date:	Fri Mar 20 15:49:29 2020 -0700
Pull:	#319 (181 additions, 3 deletions, 2 files changed)
Branch:	slaclab/rel_notes_gen
Issues:	#322
Labels:	enhancement

Notes:

Please take a look. I am not sure where the proper home is. I did not attempt to integrate into Travis.

---

Verify that all PRs are labeled with bug, enhancement, or interface-change

Author:	Jesus Vasquez [email protected]
Date:	Fri Mar 20 14:27:21 2020 -0700
Pull:	#325 (21 additions, 0 deletions, 1 files changed)
Branch:	slaclab/issue321
Issues:	#321
Labels:	enhancement

Notes:

This PR solved #321.

It adds the GitHub actions verify-pull-request-labels to verify that all PRs have at least one of these labels before they can be merged: bug, enhancement, interface-change.

This action works around the issue I mentioned in #321 regarding checks trigger with different conditions, and instead works in the following way:

• If a PR does not have one of the valid labels, then the Github check will succeed, but it will create a pull request review, requesting for changes. This will prevent the PR to be merge until the review is addressed.
• If a PR have at least one valid label, then the Github check will also succeed, and it will create a pull request review, approving it. This will clear the review requesting for changes from a previous failed check.
• The action will be triggered when a pull request is open, updated, and when labels are added/removed.

As the Github Check alway succeed, we will not have the problem I mentioned in #321. Instead, the PR will be block using normal pull request workflow.

An example scenario is:

• A PR is open, without one of the mandatory labels. The action runs and creates a PR review, requesting for changes. This will prevent the PR to be merged.
• One of the mandatory labels is added to the PR. The action will be triggered again, and will create a new PR review, approving it. This will dismiss the previous request for changes.
• If the label is removed, it will trigger the action again, and step 1 will happen again.
• If the PR labeled correctly before opening the PR, then the first action will just crate a PR review approving it.

So, in order for this new feature to work, we will need to increase the number of Required approving reviewers from 1 to 2: one review will always come from this action, and the other review will be the one needed from a real reviewer.

This PR itself will be an example of how this feature works. I'm opening the PR without any label, in order to let the action to crate the PR review, request changes. Then I ...

v4.0.0-rc21

New pysmurf version, combining server and client code, and using Rogue v4.
Release candidate 21

This release candidate includes:

• Merge pre-release into master, and start releasing from master now. The pre-release branch is not used anymore,
• Add Flake8 test to Travis (#238), and code linting cleanup,
• Move __version__ from pysmurf.client to pysmurf (#268)
• Server:
  • Disable zero copy on DDR streams (#265),
  • Change log messages to warning level, so that they don't appear by default in the console (#260),
  • Add workaround: set the SmurfProcessor.ChannelMapper.Mask to [0] by default (#256),
• Client:
  • Accelerated processing of TES bytes (#219),
  • Fix issue #76,
  • Fix issue #254,
  • Fix issue #252,
  • Fix issue #249,
  • Bug fix: properly decode f, dF streams for either bay0 or bay1 (#246),
  • Fixing bug with loading mask (#240),
  • Adding basic offline functionality,
  • Adding ability to append to plot filenames (#233),
• Update Princeton configuration files and scripts,

v4.0.0-rc20

New pysmurf version, combining server and client code, and using Rogue v4.
Release candidate 20

This release candidate includes:

• Server:
  • Update the smurf-rogue base image to version R2.6.0. This release updates Rogue to version v4.10.1 (see release notes here)
  • Migrate the GUI interface to PyDM by default. Added an input option --use-qt which can be used to use the old QT interface instead.
• Client:
  • Adc saturation check,
  • Checking the updated NIST pysmurf cfg files into rogue4 for 1000x demo,
  • Scripts and fixes to make NIST setup compatible with rogue 4 for 1000x demo,
  • Checking in offline analysis script

v4.0.0-rc19

New pysmurf version, combining server and client code, and using Rogue v4.
Release candidate 19

This release candidate includes:

• Server:
  • Include C++ speedup register access engine, for the SysgenCryo device.
• Client:
  • No changes.

v4.0.0-rc18

New pysmurf version, combining server and client code, and using Rogue v4.
Release candidate 18

This release candidate includes:

• Server:
  • Change common argument parsing to use argparse.
  • Update the smurf-rogue base image to version R2.5.2. This version includes this patch to fix a bug in Rogue, missing a import in _HelperFunctions.py.
• Client:
  • No changes.