-
Notifications
You must be signed in to change notification settings - Fork 61
Open
Labels
A-layoutTopic: Related to data structure layout (`#[repr]`)Topic: Related to data structure layout (`#[repr]`)S-not-opsemDespite being in this repo, this is not primarily a T-opsem questionDespite being in this repo, this is not primarily a T-opsem questionT-lang
Description
From #31: If you have homogeneous structs, where all the N
fields are of a single type T
, can we guarantee a mapping to the memory layout of [T; N]
? How do we map between the field names and the indices? What about zero-sized types?
A specific proposal:
- If all fields have the same type
T
(ignoring zero-sized types), then the layout will be the same as[T; N]
whereN
is the number of fields - If the struct is defined with named fields, the mapping from fields to their indices is undefined (so
foo.bar
maps an undefined index) - If the struct is defined as a tuple struct (or a tuple), then the indices are derived from the definition (so
foo.0
maps to[0]
)
This is basically because it's convenient but also because it's about the only sensible thing to do (unless you imagine we might want to start inserting padding between random fields or something, which connects to #35).
Note that for tuple structs (and tuples), the ordering of (public) fields is also significant for semver and other purposes -- changing the ordering will affect your clients. The same is not true for named fields and so this proposal attempts to avoid making it true.
Metadata
Metadata
Assignees
Labels
A-layoutTopic: Related to data structure layout (`#[repr]`)Topic: Related to data structure layout (`#[repr]`)S-not-opsemDespite being in this repo, this is not primarily a T-opsem questionDespite being in this repo, this is not primarily a T-opsem questionT-lang
Type
Projects
Milestone
Relationships
Development
Select code repository
Activity
the8472 commentedon Oct 11, 2018
Would there ever be a reason to automatically insert tail padding for homogenous structs?
hanna-kruppe commentedon Oct 11, 2018
The most plausible reason for not guarantee this would be to raise the alignment of the aggregate to allow copying the type with fewer or more efficient instructions (e.g., align a struct of four
u8
s to 32 bit and use aligned 32 bit load and store instructions for it).Gankra commentedon Oct 11, 2018
I can also imagine two hypothetical but very dubious scenarios to add to the pile:
We explicitly want to break code that's relying on the order without an annotation because someone might reorder the fields syntactically without knowing it's important (in some sense, using repr(C) as a "I promise this type will have some kind of stable ABI" annotation)
With profile-guided optimization (PGO), it is determined that (say) Foo.7 being first in the type is better due to cache effects.
gnzlbg commentedon Oct 12, 2018
I think we should be able to do this for arrays too, and also for homogenous parts of aggregates like
struct A { a: u8, x: f32, y: f32, z: f32, w: f32, b: u8 }
where it might be worth it to reorder{xyzw}
to the front not only to reduceA
's size, but also to be able to perform SIMD operations on these fields, which might require raising the alignment ofA
from4
to, e.g.,16
. I don't think any implementation will do these optimizations in the near future (maybe never), but if leaving the door open for these does not causes many headaches it might still be worth doing nevertheless. We can always provide more guarantees later.hanna-kruppe commentedon Oct 12, 2018
That would break ABI compatibility with C arrays.
Also, one reason why one might not want to raise the alignment of any aggregate is that it tends to cause extra padding, either internally (in your example
struct A
, increasing alignment to 16 bytes also increases the size of the struct from 20 bytes to 32 bytes) or externally, i.e., requiring extra padding in an aggregate that contains a field whose alignment requirement was raised.gnzlbg commentedon Oct 12, 2018
Yes,
repr(Rust)
arrays would then differ fromrepr(C)
arrays - I don't know how big of a deal this would be.Yes, that's a very good reason not to do that. FWIW I wasn't suggesting to do this unconditionally, but rather that we don't have to forbid an implementation from doing this if it deems it worthwhile right now.
Gankra commentedon Oct 12, 2018
You can't put a repr on a builtin type, so there's no such thing as a "repr(c) array". Also we've long guaranteed their layout, and that ship has sailed.
gnzlbg commentedon Oct 12, 2018
An alternative would be to asymmetrically require that tuples are layout compatible with arrays, but to not require the layout of tuples and arrays to be identical. That is, arrays would not be layout compatible with tuples.
This would allow the compiler to increase the alignment of homogeneous structs and tuples, and also allow
&(T, T, T, T) as &[T; 4]
. It would not allow&[T; 4] as &(T, T, T, T)
because the alignment of[T; 4]
might be smaller than the alignment of(T, T, T, T)
. An union likeU { a: [T; 4], t: (T, T, T, T) }
could, however, be used to re-interpret the bytes of an array as a tuple because the union will properly align the array.AFAICT it would be backwards compatible to, in the future, make the stronger guarantee that the layout of homogeneous tuples and arrays is identical, which would allow
&[T; 4] as &(T, T, T, T)
.RalfJung commentedon Oct 13, 2018
These ordering issues make me think that for homogeneous non-tuple-structs, while we could guarantee that the layout matches that of an array, we should not guarantee how the fields are mapping to array indices. Getting that from the definition order seems strange to me.
I do not have a strong opinion on this, though.
nikomatsakis commentedon Oct 18, 2018
@RalfJung
This was, I believe, what I meant when I wrote "If the struct is defined with named fields, the mapping from fields to their indices is undefined (so
foo.bar
maps an undefined index)".This also seems to address @gankro's first concern:
With respect to the PGO or SIMD scenarios, if we adopted this proposal, the idea would probably be that one out to explicitly opt in to "non-standard" layouts of this kind (e.g.,
#[repr(opt)]
or something). I think this is probably reasonable -- basically#[repr(Rust)]
gives you a compact layout by default that is optimized to do reasonably well on any program with any usage pattern. If you have specific needs (e.g., simd, avoiding false sharing) then those would be something you declare.hanna-kruppe commentedon Oct 18, 2018
This has been my position for the longest time, but recently I'm less sure since the scenario outlined in #36 (comment) doesn't really need SIMD or any hypothetical whole-program/profile-guided optimizations to be profitable. It would be nice to be able to raise e.g. the alignment of
struct Rgba { r: u8, g: u8, b: u8, a: u8 }
to four bytes because doing so would rarely increase data structure sizes (if the struct is private we can even sometimes prove this locally) and on many targets it could make copies ofRgba
use 4x fewer instructions.This is relatively niche compared to reordering to remove internal packing, and considering the nice guarantees it would conflict with, I am not arguing we should reserve that freedom, but I do regret that more than I regret PGO-driven or SIMD-targeted layout changes.
25 remaining items
kornelski commentedon Jul 5, 2020
Can we get some progress on this? I'd love to have homogeneous tuples defined.
Lokathor commentedon Jul 5, 2020
To be clear: nothing defined anywhere in the UCG is actually officially defined and you can't rely on the info you find in the UCG docs until it's actually accepted via RFC.
nikomatsakis commentedon Jul 27, 2020
I agree with @RalfJung on this, but it's true that we should say one way or the other what we consider to be a breaking change. In other words, it seems quite reasonable to me for the rule to be that:
and, therefore, one should not assume that one can transmute or whatever else unless there is an affirmative comment promising not to reorder fields.
This seems fairly consistent to me, in that we generally make our "semver rules", which are more focused on "API" than on "ABI". But as @RalfJung said, I could be persuaded either way I imagine. I think though that we should definitely state explicitly the result, and we should try to be consistent overall.
Lokathor commentedon Jul 27, 2020
the8472 commentedon Jul 27, 2020
RalfJung commentedon Jul 28, 2020
If you want to continue discussing semver-related library API guarantees, could you please open a new issue for that discussion? This thread here is about what the compiler guarantees.
the8472 commentedon Jul 28, 2020
Ok, see #242
Remove potential UB for homogeneous tuples