Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducible release tarballs #1550

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Reproducible release tarballs #1550

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
17b1a28
build: Sort distributekernel METALOG when using -DNO_ROOT
patmaddox Dec 13, 2024
a6fbdd7
build: Set METALOG time when using -DWITH_REPRODUCIBLE_BUILD
patmaddox Dec 13, 2024
a5eb00d
build: Wait to build mandoc.db when making distributeworld
patmaddox Dec 13, 2024
6bc903d
release: Document reproducible release tarballs
patmaddox Dec 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions Makefile.inc1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,7 +462,7 @@ SUBDIR+= ${_DIR}
# by calling 'makedb' in share/man. This is only relevant for
# install/distribute so they build the whatis file after every manpage is
# installed.
.if make(installworld) || make(install)
.if make(installworld) || make(install) || make(distribute)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be make(distributeworld)? I'm not sure, but I don't fully understand the distribute pseduo-target.

SUBDIR+=.WAIT
.endif
SUBDIR+=etc
Expand Down Expand Up @@ -1441,7 +1441,11 @@ DEBUG_DISTRIBUTIONS=
DEBUG_DISTRIBUTIONS+= base ${EXTRA_DISTRIBUTIONS:S,tests,,}
.endif

MTREE_MAGIC?= mtree 2.0
MTREE_MAGIC?= mtree 2.0
.if ${MK_REPRODUCIBLE_BUILD} == "yes"
MTREE_TIME= time=${SOURCE_DATE_EPOCH}.000000000
SED_REPLACE_TIME= -e "s/ time=[[:digit:].]*/ ${MTREE_TIME}/"
.endif

distributeworld installworld stageworld: _installcheck_world .PHONY
mkdir -p ${INSTALLTMP}
Expand Down Expand Up @@ -1470,6 +1474,7 @@ distributeworld installworld stageworld: _installcheck_world .PHONY
.if defined(NO_ROOT)
-mkdir -p ${METALOG:H}
echo "#${MTREE_MAGIC}" > ${METALOG}
echo "/set ${MTREE_TIME}" >> ${METALOG}
.endif
.if make(distributeworld)
.for dist in ${EXTRA_DISTRIBUTIONS}
Expand Down Expand Up @@ -1548,7 +1553,7 @@ distributeworld installworld stageworld: _installcheck_world .PHONY
@# the relevant mtree line.
cd ${DESTDIR}/${DISTDIR}; \
find ./${dist}${path} | ${METALOG_SORT_CMD} -u ${METALOG} - | \
awk 'BEGIN { print "#${MTREE_MAGIC}" } !/ type=/ { file = $$1 } / type=/ { if ($$1 == file) { sub(/^\.\/${dist}/, "."); print } }' > \
awk 'BEGIN { print "#${MTREE_MAGIC}"; print "/set ${MTREE_TIME}" } !/ type=/ { file = $$1 } / type=/ { if ($$1 == file) { sub(/^\.\/${dist}/, "."); print } }' > \
${DESTDIR}/${DISTDIR}/${dist}${suffix}
.endfor
.endfor
Expand Down Expand Up @@ -1903,25 +1908,23 @@ distributekernel distributekernel.debug: .PHONY
false
.endif
mkdir -p ${DESTDIR}/${DISTDIR}
.if defined(NO_ROOT)
@echo "#${MTREE_MAGIC}" > ${DESTDIR}/${DISTDIR}/kernel.premeta
.endif
rm -f ${DESTDIR}/${DISTDIR}/kernel.premeta
${_+_}cd ${KRNLOBJDIR}/${INSTALLKERNEL}; \
${IMAKEENV} ${IMAKE_INSTALL:S/METALOG/kernel.premeta/} \
${IMAKE_MTREE} PATH=${TMPPATH:Q} ${MAKE} KERNEL=${INSTKERNNAME} \
DISTBASE=/kernel DESTDIR=${INSTALL_DDIR}/kernel \
METALOG=${METALOG:S/METALOG/kernel.premeta/} \
${.TARGET:S/distributekernel/install/}
.if defined(NO_ROOT)
@sed -e 's|^./kernel|.|' ${DESTDIR}/${DISTDIR}/kernel.premeta > \
${DESTDIR}/${DISTDIR}/kernel.meta
echo "#${MTREE_MAGIC}" > ${DESTDIR}/${DISTDIR}/kernel.meta
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to keep the @s presumably

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO the @s are bugs. These aren't steps with should be hiding from the caller.

echo "/set ${MTREE_TIME}" >> ${DESTDIR}/${DISTDIR}/kernel.meta
sed -e 's|^./kernel|.|' ${SED_REPLACE_TIME} ${DESTDIR}/${DISTDIR}/kernel.premeta | \
${METALOG_SORT_CMD} >> ${DESTDIR}/${DISTDIR}/kernel.meta
.endif
.endif
.if ${BUILDKERNELS:[#]} > 1 && ${NO_INSTALLEXTRAKERNELS} != "yes"
.for _kernel in ${BUILDKERNELS:[2..-1]}
.if defined(NO_ROOT)
@echo "#${MTREE_MAGIC}" > ${DESTDIR}/${DISTDIR}/kernel.${_kernel}.premeta
.endif
rm -f ${DESTDIR}/${DISTDIR}/kernel.${_kernel}.premeta
${_+_}cd ${KRNLOBJDIR}/${_kernel}; \
${IMAKEENV} ${IMAKE_INSTALL:S/METALOG/kernel.${_kernel}.premeta/} \
${IMAKE_MTREE} PATH=${TMPPATH:Q} ${MAKE} \
Expand All @@ -1930,9 +1933,11 @@ distributekernel distributekernel.debug: .PHONY
METALOG=${METALOG:S/METALOG/kernel.${_kernel}.premeta/} \
${.TARGET:S/distributekernel/install/}
.if defined(NO_ROOT)
@sed -e "s|^./kernel.${_kernel}|.|" \
${DESTDIR}/${DISTDIR}/kernel.${_kernel}.premeta > \
${DESTDIR}/${DISTDIR}/kernel.${_kernel}.meta
echo "#${MTREE_MAGIC}" > ${DESTDIR}/${DISTDIR}/kernel.${_kernel}.meta
echo "/set ${MTREE_TIME}" >> ${DESTDIR}/${DISTDIR}/kernel.${_kernel}.meta
sed -e "s|^./kernel.${_kernel}|.|" ${SED_REPLACE_TIME} \
${DESTDIR}/${DISTDIR}/kernel.${_kernel}.premeta | \
${METALOG_SORT_CMD} >> ${DESTDIR}/${DISTDIR}/kernel.${_kernel}.meta
.endif
.endfor
.endif
Expand Down
10 changes: 10 additions & 0 deletions share/man/man7/release.7
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,16 @@ This is intended for use only when
.Fa /usr/ports
is expected to exist by alternative means.
.El
.Sh REPRODUCIBLE BUILDS
Different builders can produce bit-identical release tarballs using
.Va PKG_TIMESTAMP
and
.Va REVISION :
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to differentiate PKG_TIMESTAMP from REVISION -- the former is an actual (literal) env variable used by Makefile*, but the latter is a placeholder in the example command below. I'm not sure off hand what the markup for that should be.

I wonder if we should have newvers.sh collect the commit time (e.g. from git show -s --format=%ct) and have the default SOURCE_DATE_EPOCH derived from that.

.Bd -literal -offset indent
git checkout <REVISION>
make -DWITH_REPRODUCIBLE_BUILD buildworld buildkernel
make -C release -DWITH_REPRODUCIBLE_BUILD PKG_TIMESTAMP=<TIMESTAMP> packagesystem
.Ed
.Sh EMBEDDED BUILDS
The following
.Fa release.conf
Expand Down