Skip to content

System DT Meeting Notes 2020

Jump to bottom Edit New page
Nathalie Chan King Choy edited this page Jan 24, 2020 · 9 revisions

Table of Contents

2020-Jan-22

Attended:

Joakim Bech, Tomas Evensen, Ilias, Rob Herring, Mathieu Poirier, Stefano Stabellini, Ed Mooring, Mark Dapoz, Nathalie Chan King Choy, Bruce Ashfield, Loic Pallardy (ST), Etsam Anjum (MGC), Dan Driscoll, Saravanna Kannan (Google)

Action items:

  • Stefano: Document a little bit more how the model works
    • Remove reserved memory
    • Add top-level use-case FAQ (e.g. how to do peripheral assignment to SW)
    • Consider putting a qualifier word before "domain" to make it more specific
  • Everyone: Try to poke holes in the model. Good to have hard questions to think through & answer
  • Rob: Prototype proposal of changing root
  • Nathalie: co-ordinate next call over email (2 weeks from now doesn't work b/c Rob can't make it)

Notes:

  • Tomas: Recap:
    • Trying to expand current concept of device tree to describe the whole system
      • Add features to talk about multiple CPU clusters, multiple masters
      • Add config info to say which resources (CPU, memory, I/O device) belong to which domain
    • Look at from physical & virtual point-of-view: Would like to be able to describe both AMP & hypervisor use case.
    • Splitting how the HW looks from the configuration
      • DT is more about hardware description than configuration
      • Different ppl are coming with the different info
        • HW info: Board, SoC, vendor
        • Who decides how you split up the resources (e.g. between Linux, FreeRTOS, Zephyr) is different (e.g. System architect)
    • Stefano: added some features & not yet made proposal for config
  • Stefano: Rob suggested to write out both HW (new CPU clusters) & configuration (e.g. OpenAMP domain): How do they interact w/ existing?
    • Writing an overview was useful to get ppl on the same page & how these features fit together
    • Highlighted at least 1 problem:
      • Most top level nodes are to describe the HW (memory, amba bus, cpus).
      • Originally put OpenAMP domain under chosen to mark as configuration. Not really final destination for these domains, but made easy to identify the config & not drastically change structure of DT.
      • Reserved memory is one of few nodes that's already there for configuration. So, why not use that for describing memory? But, this is a bit awkward w/ multiple domains & multiple OS. You want 1 chosen & 1 reserved memory for each domain.
  • Rob:
    • Think you captured it pretty well
    • Domain is probably an over-used term
      • HW domains
      • Each HW domain is 1 or more config domains (e.g. R5, Cortex-A, may be divided into secure/non-secure)
      • Stefano: Agree: Domain is overloaded term
      • Loic: In ST we call it "execution context", where we can run SW component. Could be Cortex A secure on non-secure, or Cortex M. And, all the peripherals we can access.
      • Tomas: We had tried "execution context" in past & no one had understood it. Xen is calling this "domains". Tricky: In some SoCs this is very configurable. Don't want to change the structure of the DT if you change the configuration.
  • Stefano:
    • Makes sense to keep HW description all together & immutable
      • Used CPU attribute
      • Access list that points to buses
      • Memory was controversial case. Had 2 ways:
        • access list (could go to reserved memory - bit controversial),
        • special memory attribute (memory config carve out for domain)
          • Top-level memory node describes physical memory
          • Config of memory & what devices accessible & what cpu cluster we're running described in the domain under chosen (or clearly config section)
  • Discussion on how much effort should go into making it possible for Linux to understand a System DT. Grant urges to do so for default case. Otherwise, Lopper gives "legacy" OS a DT that it can understand.
    • Rob:
      • Doing that w/ memory is trade-off for how OS parses its memory
        • OS can't look at common node describing all of memory
    • Stefano:
      • Compatibility w/ legacy OS & domain description is bit of a balance
    • Rob:
      • If we change how OS parses, OS has to know how to parse both ways b/c might not have System DT. Legacy OS isn't necessarily "legacy" b/c not getting rid of it. With tool, only have to consider 1 way of doing it.
    • Stefano:
      • Just realized an assumption I had: A "legacy" OS will only have to support default. Either we'll get System DT will give the "legacy" OS the default view or have tool Lopper to give the OS the info it needs.
    • Rob:
      • So you're not going to run Linux on R5?
    • Tomas:
      • We intended that you would run Lopper first if it's the secondary OS.
      • To get system DT to feed directly into OS came later. Figured System DT is on host side & would always run Lopper. Grant wanted to see if we could have Linux understand System DT.
    • Rob:
      • Is it worth the messiness to avoid running Lopper on 1 view of the system vs. doing something more symmetrical & cleaner?
    • Tomas:
      • For us, RTOS won't use DT directly. It will use DT to get a .h with #defines in them. B/c resource constrained.
    • Rob:
      • Think that's true today but will diminish over time. Think more processors will run Linux in same system
    • Tomas:
      • You're saying when running multiple Linux instances, 1 will get default & others will need Lopper
    • Etsam:
      • Think system Dt should just work if only 1 execution context in system (just 1 Linux), or if there are multiple contexts then have to run Lopper b/c have to do memory partitioning & device assignment.
      • Tomas; Think this is Stefano's intention
      • Stefano: Yes
    • Stefano:
      • Rob suggesting: Is it worth trying to maintain max compatibility if makes things awkward?
      • We could maintain backward compatibility, but it doesn't have to be super nice. Will make it easier to write changes to spec if more seamless & not revolutionary. Then if want to run "legacy" OS, it will boot for primary case.
      • Trying to make multiple domain case w/ Lopper nice. Reserved memory makes this awkward
    • Etsam:
      • For multiple execution contexts or multiple domains, currently we are putting all the resource assignment info into chosen node. Problem is that OS will have to parse that info.
      • Dan proposed if we separate out the resource assignment to domains vs. what's needed to capture heterogeneity in SoCs
      • Even with that, having multiple execution contexts will need a tool to be run
      • If we move the domain concept out of the main system DT, the changes in OS will be minimal. Will not need to parse domain nodes in chosen
    • Stefano:
      • Have not had a chance to read Dan's yet, but think my proposal will require minimal changes. No change for legacy OS in default. Lopper use -> don't need to change OS.
      • If want to enhance an OS to understand system DT w/o Lopper at runtime, but amt of new info to read is quite minimal, if it knows which CPU cluster to use, all the info is there. There is a bit of link jumping & reference checking, but just 3 new attributes to read. Think it will be feasible to add support for that
      • And, is it worth optimizing for this case when you can use Lopper & not need any OS changes?
    • Rob: Back to my proposal of domains & changing what the root is
      • If we look at flattened DT, that may only require zeroing out the top level node to make domain at 0 node the root node, which would be a very minimal change
      • Or, at most, when you unflatten it, you could point to a different location & make that your root
      • If you have everything under a domain node & nothing else outside, then it would be a trivial change to the OS. Or, even the bootloader could do it before you give it to the OS. You could do Lopper as part of your boot flow.
    • Etsam:
      • Which OS will pick which node from the domain nodes when you have multiple execution contexts?
    • Rob:
      • You could say that the default node is domain 0. At some point, you have to pick which is which. Probably a couple ways to do that.
    • Tomas:
      • If you do that, you have decided on the domains ahead, but the domains are decided later by system architect (e.g. when you have TrustZone) with same device & multiple domains
    • Rob:
      • Probably need 2 levels of domains:
        • HW domains: Pretty much fixed: Cortex-A, R5, microcontroller
        • Within those, you have config domains which handle asymmetric cases, or TZ, or hyper
    • Dan:
      • Could have hypervisor running 4 different guests. A cluster could be multiple domains from a SW perspective. So, can't break up HW into subsystem cluster. It's how you map the SW to any execution context.
    • Rob:
      • You'd have 2 levels: 1 level is dictated by HW. Within a HW domain, you're creating config or openamp domains
    • Dan:
      • OK, in the end, yoy have a HW cluster that all the devices & buses fall under. Right now there's no relationship back to which CPU cluster
    • Stefano:
      • What about device that has been assigned to one of the OpenAMP domains. How would you express that?
    • Rob:
      • Good question. If the devices are all viewable by all the HW domains, then it would be at the top level.
    • Tomas:
      • Way to think about our system is CPU, memory, devices are fairly independent. Can assign as you want, mostly.
      • Interrupt controllers are some exceptions, but Stefano has way of expressing that.
      • General use case is to have someone else select from HW resources & create the domains. For CPU clusters, how do you tie those to the fixed HW - think Stefano was doing w/ these different buses.
    • Stefano:
      • If you consider that devices remain top level (e.g. amba), and link to them under Rob's domains. Not too different. Then OpenAMP domain would be directly under cpu clusters.
    • Loic:
      • Benjamin from ST has proposal for domain controller. Working on for kernel & u-boot. He proposed some bindings in each peripheral node to have info about which execution context can access. It's directly defined in the peripheral's node, like pin control. You can then say which domain it is associated.
      • Tomas: Then do you have to change tree when you reconfigure?
      • Dan: Would have to show that it could refer back to multiple execution contexts, whether it can access it or not. Not to do with domain definition, just the HW association
      • Loic: Yes
      • Dan: Seem have same issue w/ interrupt controller
      • Stefano: Solved in last example by describing all interrupt controllers. MPSoC has multiple interrupt controllers.
      • Dan: Is it changing the bindings or adding new?
      • Stefano: Thanks to Grant's suggestion, could do without any change. You use as interrupt parent of all devices like an interrupt muxer.
      • Dan: Then Lopper has to generate the right piece that this is the interrupt controller that I have
      • Rob: We only solved how to describe more than 1 parent, but not how to select which parent is the correct one. If you have HW domain, then you can stick it in HW domain.
      • Stefano: There is also HW restrictions where something is not visible or accessible between clusters. This is currently done with address map & indirect bus.
      • Dan: OK, in that case the info is all in the system DT & Lopper not necessary. So, we are discussing some cases where don’t need Lopper & other cases where Lopper makes life easier. At some point, we have to choose.
      • Tomas: We didn't originally think it could be backward compatible until we had the conversation w/ Grant. In a way, having default domain makes a lot of sense.
  • Tomas: Are there unsolved issues being able to express all the domains (SW context), HW that is only attached to only 1 cpu cluster & not others? Or, are all the issues kind of solved in your mind & more syntax to work out?
    • Stefano:
      • Not aware of open issues for the heterogeneous clusters case. It's about syntax & how we want to express & make nicer for 1 type of parsing
      • There are issues for VM use case. Want to wait for these ideas to settle 1st. Think could be solved additions
        • Memory for VM needs physical & pseudo-physical ranges
        • Where to place virtual-only devices
    • Rob:
      • Disagree that there's not open issues & it's just syntax. Syntax is open issues & need to know that they will work for the different cases.
      • Feel like we've gone down this path & each step there's different issues to solve & thinking about if we should back up & go down a different path w/ less issues to solve
    • Tomas:
      • Want to make sure that the model holds up
      • Lots of good energy on the topic. Think we all want the same thing & just need to clarify.
Add a custom footer
Add a custom sidebar
Clone this wiki locally