Skip to content

Understanding Dirtiness in Dplug

Jump to bottom
p0nce edited this page Jan 10, 2023 · 61 revisions

Here is a WIP document about how dirtiness and graphics work in Dplug as of v7. And how Issue #127 will be implemented. This is actually the most complicated topic in Dplug.

How it works in Dplug v7

The Dirty List

In Dplug the UI is made of several UIElement organized in a tree.

Upon parameter changes and animation, some UIElement get invalidated (possibly partially) through UIElement.setDirty or UIElement.setDirtyWhole. This starts a chain reaction that ultimately lead to them being redrawn with onDraw. However the details are super tricky.

A rectangle based on the window origin is stored in a synchronized manner in the "Dirty List" which is global to the UI (and as such, held by UIContext).

class UIContext
{
...
    // This is the global UI list of rectangles that need updating.
    // This used to be a list of rectangles per UIElement,
    // but this wasn't workable because of too many races and
    // inefficiencies.
    DirtyRectList dirtyList;
...
}

The Dirty List contains a list of disjointed rectangles whose areas have been updated in the diffuse, material or depth map. It signals that such areas should be onDraw and then recomputed by the PBR compositor and then updated on the screen.

The IWindow interface

At the other end, the window actively (through a timer) pulls "dirtiness" information by calling those related callbacks in GUIGraphics.

    void onDraw(WindowPixelFormat pf);
    void recomputeDirtyAreas();
    box2i getDirtyRectangle();

Almost all windowing backends approximately do this:

   on timer:
       call onAnimate on widgets
       GUIGraphics.recomputeDirtyAreas()
       rect R <- GUIGraphics.getDirtyRectangle()
       if R is not empty:
           tell the OS this window needs a graphical update in R

   on OS redraw(rect R): // Important: such as to decorrelate actual redraw from dirtiness, in case of CPU overload
       call GUIGraphics.onDraw() on the union of dirty rectangles from one recomputeDirtyAreas() call
       update region R of the screen

The GUIGraphics object

GUIGraphics is an ugly god object which is at the uncomfortable trisection of:

  • the IWindow, abstract window interface
  • the UI tree (of which it's the root)
  • and the generic plugin client

This unfortunate mission gave birth to the IWindowListener.

  • GUIGraphics.recomputeDirtyAreas() is implemented like this:

    1. Pulls all rectangles from the Dirty List (synchronized) into _areasToUpdateNonOverlapping. This step make them disjointed again because rectangels from previous recomputeDirtyAreas accumulates until drawn.
    2. Create another list of the same rectangles _areasToRender but with a margin to account for PBR layer affecting the final layer

  • GUIGraphics.getDirtyRectangle() is implemented like this:

    1. Return the union of every rectangles in _areasToRender

  • GUIGraphics.onDraw() is implemented like this:

    1. calls onDraw on every (even partially) dirty UIElement and whatever is below or above it, in order. This is where widgets get rendered. Widgets are stable-sorted by z-order before so that the calls are in the right order. If the positions are disjointed, this step may happen in parallel. After this step, the PBR buffers are now updated.
    2. Mipmaps are updated in disjointed(_areasToRender). The 2 different mipmaps are computed in parallel.
    3. the PBR Compositor is called, which is potentially the most expensive part
    4. _areasToUpdateNonOverlapping is made empty

As complicated as it sounds, there is some essential complexity in this task that perhaps warrant all this. It can probably be simplified, but not this time unfortunately.

How it works in Dplug v8 (with Issue #127 solved)

Issue #127 when fixed will enable:

  • Much more practical PBR, no more need to precompute areas. It happens internally. Possibility to change lighting dynamically.
  • As such, more incentive to draw RGBA pixels directly in a RGBA buffer, which is currently impossible (the diffuse map is hijacked instead).
  • Mipmapping happens only for PBR layer changes, not all changes.
  • RGB correction curves becomes optional.
  • Physical channel can be removed.

Two layers

  • There are now two "layers", PBR (material before compositing) and Raw (composited/rendered) layer. Changing Raw layer is much less CPU intensive than changing the PBR layer.
  • Two separate calls to signify a change in a widget: setDirty(UILayer.guessFromFlags)(default) and setDirty(UILayer.onlyRaw)
  • Two separate methods to draw on them: onDrawPBR and onDrawRaw (for compatiblity onDrawPBR is still called onDraw for a while). onDrawPBR should be called on areas dirtied with layerPBR onDrawRaw should be called on areas dirtied with either, extended in the case of PBR

Because compositing is expensive, onDrawPBR is called with a list of rectangles and must perform a partial update. This is the same for onDrawRaw, for now.

The new dirty lists

class UIContext
{
...  
    DirtyRectList dirtyListPBR; // rectangles for which the widgets report having changed at the PBR level
    DirtyRectList dirtyListRaw; // rectangles for which the widgets report having changed at the Raw level
...
}

The new GUIGraphics implementation

  • GUIGraphics.recomputeDirtyAreas() is now implemented like this:
    1. Pulls all rectangles from dirtyListPBR (synchronized) into _rectsToUpdatePBR. Make them disjointed again eventually.
    2. Pulls all rectangles from dirtyListRaw (synchronized) into _rectsToUpdateRaw. Make them disjointed again eventually. Note that there is race between the two lists (in the case of a widget drawing on both layers, rectangles pulled might not correspond), but it can't lead to adverse conditions.
    3. Create two sets of rectangles _rectsToComposite and _rectsToDisplay which follows the forumula:
     _areasToComposite <- Union(Margin(_areasToUpdatePBR))
     _areasToDisplay <- Union(_rectsToComposite, _rectsToUpdateRaw)

  • GUIGraphics.getDirtyRectangle() is now implemented like this:

    1. Return the union of every rectangles in _areasToDisplay

  • GUIGraphics.onDraw() is now implemented like this:

    1. calls onDrawPBR on every (even partially) UIElement and whatever is below or above it, in z-order, that touches _areasToUpdatePBR (BREAKING unless we call this "onDraw" for a while)
    2. Mipmaps are updated in the corresponding extended _rectsToUpdatePBR, with the proper margin etc.
    3. the PBR Compositor is called, on those rectangles in _rectsToComposite.
    4. At this point a RGBA buffer _compositedBuffer exists, with rendered, non-corrected, pixel data.
    5. For every rect in _rectsToDisplay, this buffer is copied to another _renderedBuffer buffer which is also the output buffer
    6. calls onDrawRaw on _renderedBuffer for every (even partially) UIElement and whatever is below or above it, in z-order, that touches _rectsToDisplay.
    7. Apply Color-correction and R/B swap, can be handled by special top-level UIElements. (BREAKING) R/B swap could not be implemented by a widget (that would be an abstraction-leak from windowing)

  • make R/B swap work again

  • Update every UI element to use the right onDrawxxx call and have the right flags

  • make two UIBufferedElement that can work with the Raw or Pbr layer uniquely

  • widgets have immutable flags

  • setDirty chooses what to invalidate according to their flags. For PBR widgets wanting to only udpate thez Raw layer, this must be explicit.

  • Make a color correction-widget

More performance potential:

  • using widgets flags, avoid useless calls to onAnimate
  • using widgets flags, avoid useless calls to onDrawRaw
  • using widget flags, avoid useless calls to onDrawPBR
  • Later: remove the Physical channel which is now useless

Pseudo-code as of 2022

Need to distangle this mess at one point:

PSEUDOCODE OF WINDOW

    onResize();

    loop:
        recomputeDirtyRectangle();
        if (getDirtyRectangle != nothing)
            then onDraw();

    onResize();

    loop:
    recomputeDirtyRectangle();
    if (getDirtyRectangle != nothing)
        onDraw();

PSEUDOCODE OF GRAPHICS:

    // if true, it means the whole resize buffer and accompanying black
    // borders should be redrawn at the next onDraw
    bool _redrawBlackBordersAndResizedArea;

    // if true, it means the whole resize buffer and accompanying black
    // borders should be reported as dirty at the next recomputeDirtyAreas, and until
    // it is drawn.
    bool _reportBlackBordersAndResizedAreaAsDirty;        


    /// Render the window in software in the buffer previously returned by `onResized`.
    /// At the end of this function, the whole buffer should be a valid, coherent UI.
    ///
    /// recomputeDirtyAreas() MUST have been called before this is called.
    /// The pixel format cannot change over the lifetime of the window.
    ///
    /// `onDraw` guarantees the pixels to be in the format requested by `pf`, and it also
    /// guarantees that the alpha channel will be filled with 255.
    onDraw
        * by contract, make sure things in _rectsToUpdateDisjointedRaw and _rectsToUpdateDisjointedPBR 
          right now will be redrawn

        * redraw the UI, puts it in _rectsToResizeDisjointed. 
          Refait les bordures noires, et copie toute l'UI, si _redrawBlackBordersAndResizedArea == true

    /// The drawing area size has changed.
    /// Always called at least once before onDraw.
    /// Returns: the location of the full rendered framebuffer.
    onResized(logical size)
        * if logical size (the size the DAW gives us) has changed:
              then _reportBlackBordersAndResizedAreaAsDirty = true
        * if user size (the internal size of UI within the plugin) has changed:
              then insert a full dirty PBR rectangle in _rectsToUpdateDisjointedPBR and 
              call recomputePurelyDerivedRectangles(). Ahem...
  
    recomputeDirtyAreas
        * pull everything that is in the global UI dirty list, puts those rects in
          _rectsToUpdateDisjointedRaw and _rectsToUpdateDisjointedPBR
          call recomputePurelyDerivedRectangles()

    recomputePurelyDerivedRectangles()
        * make sure that _rectsToUpdateDisjointedRaw and _rectsToUpdateDisjointedPBR are still valid
          valid (a bit strange...)
        * purely derived   = _rectsToUpdateDisjointedRaw,  // The rectangles we engaged ourselves on redrawing
                             _rectsToUpdateDisjointedPBR,  // The rectangles we engaged ourselves on redrawing
                             _rectsToCompositeDisjointed   // The areas that must be effectively re-composited.
                             _rectsToDisplayDisjointed     // The areas that must be effectively redisplayed, which also mean the Raw layer is redrawn.
                             _rectsToResizeDisjointed     // The areas that must be effectively redisplayed, in logical space (like _userArea).
        * if _reportBlackBordersAndResizedAreaAsDirty == true
            then _redrawBlackBordersAndResizedArea = true

    getDirtyRectangle
        * if _reportBlackBordersAndResizedAreaAsDirty == true:
              then report the whole area to be dirty
              else it's just the union of _rectsToResize
Clone this wiki locally