dune subst no longer works on empty directories in ocaml-ci without Dune 3.17

@mtelvers

Expected Behavior

dune subst (invovked by opam) continues to work on empty directories of vendored projects for pinned dependencies, as it did up until 3.16

Actual Behavior

Failes with

### output ###
# File ".", line 1, characters 0-0:
# Error: There is no dune-project file in the current directory, please add one
# with a (name <name>) field in it.
# Hint: 'dune subst' must be executed from the root of the project.

Reproduction

This is not at all minimal, but is a starter, taken from an ocaml-ci job (where this problem has been encountered):

git clone --recursive "https://github.com/ocurrent/ocaml-ci.git" -b "master" && cd "ocaml-ci" && git reset --hard bc5c5e55
cat > Dockerfile <<'END-OF-DOCKERFILE'
FROM ocaml/opam:ubuntu-22.04-ocaml-4.14@sha256:1c5bf3c2bec3ee454185e5155264394ba79f70e5c3985b66f7bd57810d9cbebb
# ubuntu-22.04-4.14_opam-2.3
USER 1000:1000
ENV CLICOLOR_FORCE="1"
ENV OPAMCOLOR="always"
WORKDIR /src
RUN sudo ln -f /usr/bin/opam-2.3 /usr/bin/opam
RUN opam init --reinit -ni
RUN uname -rs && opam exec -- ocaml -version && opam --version
WORKDIR /src
RUN sudo chown opam /src
RUN cd ~/opam-repository && (git cat-file -e 9d811310def03190e9b9b1db9c9ea19b4f954c17 || git fetch origin master) && git reset -q --hard 9d811310def03190e9b9b1db9c9ea19b4f954c17 && git log --no-decorate -n1 --oneline && opam update -u
RUN mkdir -p 'solver-service/' 'ocurrent/' 'ocluster/' 'ocluster/obuilder/' 'ocluster/' 'ocaml-version/' 'ocaml-dockerfile/'
COPY --chown=1000:1000 solver-service/solver-worker.opam solver-service/solver-service.opam solver-service/solver-service-api.opam solver-service/
COPY --chown=1000:1000 ocurrent/current_web.opam ocurrent/current_ssh.opam ocurrent/current_slack.opam ocurrent/current_rpc.opam ocurrent/current_gitlab.opam ocurrent/current_github.opam ocurrent/current_git.opam ocurrent/current_examples.opam ocurrent/current_docker.opam ocurrent/current.opam ocurrent/
COPY --chown=1000:1000 ocluster/ocluster.opam ocluster/ocluster-worker.opam ocluster/ocluster-api.opam ocluster/
COPY --chown=1000:1000 ocluster/obuilder/obuilder.opam ocluster/obuilder/obuilder-spec.opam ocluster/obuilder/
COPY --chown=1000:1000 ocluster/current_ocluster.opam ocluster/
COPY --chown=1000:1000 ocaml-version/ocaml-version.opam ocaml-version/
COPY --chown=1000:1000 ocaml-dockerfile/dockerfile.opam ocaml-dockerfile/dockerfile-opam.opam ocaml-dockerfile/dockerfile-cmd.opam ocaml-dockerfile/
COPY --chown=1000:1000 ocaml-ci.opam ocaml-ci-web.opam ocaml-ci-service.opam ocaml-ci-gitlab.opam ocaml-ci-client.opam ocaml-ci-api.opam ./
RUN opam pin add -yn solver-worker.dev 'solver-service/' && \
    opam pin add -yn solver-service.dev 'solver-service/' && \
    opam pin add -yn solver-service-api.dev 'solver-service/' && \
    opam pin add -yn current_web.dev 'ocurrent/' && \
    opam pin add -yn current_ssh.dev 'ocurrent/' && \
    opam pin add -yn current_slack.dev 'ocurrent/' && \
    opam pin add -yn current_rpc.dev 'ocurrent/' && \
    opam pin add -yn current_gitlab.dev 'ocurrent/' && \
    opam pin add -yn current_github.dev 'ocurrent/' && \
    opam pin add -yn current_git.dev 'ocurrent/' && \
    opam pin add -yn current_examples.dev 'ocurrent/' && \
    opam pin add -yn current_docker.dev 'ocurrent/' && \
    opam pin add -yn current.dev 'ocurrent/' && \
    opam pin add -yn ocluster.dev 'ocluster/' && \
    opam pin add -yn ocluster-worker.dev 'ocluster/' && \
    opam pin add -yn ocluster-api.dev 'ocluster/' && \
    opam pin add -yn obuilder.dev 'ocluster/obuilder/' && \
    opam pin add -yn obuilder-spec.dev 'ocluster/obuilder/' && \
    opam pin add -yn current_ocluster.dev 'ocluster/' && \
    opam pin add -yn ocaml-version.dev 'ocaml-version/' && \
    opam pin add -yn dockerfile.dev 'ocaml-dockerfile/' && \
    opam pin add -yn dockerfile-opam.dev 'ocaml-dockerfile/' && \
    opam pin add -yn dockerfile-cmd.dev 'ocaml-dockerfile/' && \
    opam pin add -yn ocaml-ci.dev './' && \
    opam pin add -yn ocaml-ci-web.dev './' && \
    opam pin add -yn ocaml-ci-service.dev './' && \
    opam pin add -yn ocaml-ci-gitlab.de
8000
v './' && \
    opam pin add -yn ocaml-ci-client.dev './' && \
    opam pin add -yn ocaml-ci-api.dev './'
# RUN echo '(lang dune 3.0)' > 'solver-service/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocurrent/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocluster/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocluster/obuilder/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocluster/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocaml-version/dune-project' && \
#    echo '(lang dune 3.0)' > 'ocaml-dockerfile/dune-project' && \
#   echo '(lang dune 3.0)' > './dune-project'
ENV DEPS="0install-solver.2.18 alcotest.1.8.0 alcotest-lwt.1.8.0 angstrom.0.16.1 ansi.0.7.0 arp.3.1.1 asetmap.0.8.1 asn1-combinators.0.2.6 astring.0.8.5 atd.2.15.0 atdgen.2.15.0 atdgen-runtime.2.15.0 awa.0.3.0 awa-mirage.0.3.0 base.v0.16.3 base-bigarray.base base-bytes.base base-threads.base base-unix.base base64.3.5.1 bigarray-compat.1.1.0 bigarray-overlap.0.2.1 bigstringaf.0.10.0 biniou.1.2.2 bos.0.2.1 ca-certs.0.2.3 ca-certs-nss.3.101 camlp-streams.5.0.1 capnp.3.6.0 capnp-rpc.1.2.3 capnp-rpc-lwt.1.2.3 capnp-rpc-net.1.2.3 capnp-rpc-unix.1.2.3 caqti.2.1.2 caqti-lwt.2.1.1 carton.0.7.2 carton-git.0.7.2 carton-lwt.0.7.2 cf.0.5.0 cf-lwt.0.5.0 checkseum.0.5.2 cmdliner.1.3.0 cohttp.6.0.0 cohttp-lwt.6.0.0 cohttp-lwt-unix.6.0.0 conduit.6.2.3 conduit-lwt.6.2.3 conduit-lwt-unix.6.2.3 conf-bash.1 conf-capnproto.2 conf-git.1.1 conf-gmp.4 conf-gmp-powm-sec.3 conf-graphviz.0.1 conf-libev.4-12 conf-libffi.2.0.0 conf-libssl.4 conf-pkg-config.3 conf-sqlite3.1 conf-which.1 cppo.1.8.0 crunch.3.3.1 csexp.1.5.2 cstruct.6.2.0 cstruct-lwt.6.2.0 cstruct-unix.6.2.0 csv.2.4 ctypes.0.23.0 ctypes-foreign.0.23.0 current.dev current_docker.dev current_git.dev current_github.dev current_gitlab.dev current_incr.0.6.1 current_ocluster.dev current_rpc.dev current_web.dev decompress.1.5.3 digestif.1.2.0 dns.7.0.3 dns-client.7.0.3 dns-client-lwt.7.0.3 dns-client-mirage.7.0.3 dockerfile.dev dockerfile-opam.dev domain-name.0.4.0 dream.1.0.0~alpha6 dream-httpaf.1.0.0~alpha3 dream-pure.1.0.0~alpha2 duff.0.5 dune.3.17.0 dune-build-info.3.17.0 dune-configurator.3.17.0 dune-private-libs.3.17.0 dune-site.3.17.0 duration.0.2.1 dyn.3.17.0 easy-format.1.3.4 emile.1.1 encore.0.8 eqaf.0.9 ethernet.3.2.0 extunix.0.4.3 faraday.0.8.2 faraday-lwt.0.8.2 faraday-lwt-unix.0.8.2 fmt.0.9.0 fpath.0.7.3 fsevents.0.3.0 fsevents-lwt.0.3.0 git.3.15.0 git-mirage.3.15.0 git-paf.3.15.0 git-unix.3.15.0 github.4.4.1 github-data.4.4.1 github-unix.4.4.1 gitlab.0.1.8 gitlab-unix.0.1.8 gmap.0.3.0 graphql.0.14.0 graphql-lwt.0.14.0 graphql_parser.0.14.0 h2.0.13.0 happy-eyeballs.0.6.0 happy-eyeballs-lwt.0.6.0 happy-eyeballs-mirage.0.6.0 hex.1.5.0 hkdf.1.0.4 hmap.0.8.1 host-arch-x86_64.1 host-system-other.1 hpack.0.13.0 http.6.0.0 httpaf.0.7.1 httpun-types.0.2.0 hxd.0.3.3 ISO8601.0.2.6 inotify.2.6 integers.0.7.0 ipaddr.5.6.0 ipaddr-cstruct.5.6.0 ipaddr-sexp.5.6.0 irmin-watcher.0.5.0 jsonm.1.0.2 ke.0.6 lambdasoup.1.1.1 logs.0.7.0 lru.0.3.1 lwt.5.9.0 lwt-dllist.1.0.1 lwt_ppx.5.8.0 lwt_ssl.1.2.0 macaddr.5.6.0 macaddr-cstruct.5.6.0 magic-mime.1.3.1 markup.1.0.3 menhir.20240715 menhirCST.20240715 menhirLib.20240715 menhirSdk.20240715 metrics.0.4.1 mimic.0.0.6 mimic-happy-eyeballs.0.0.6 mirage-clock.4.2.0 mirage-clock-unix.4.2.0 mirage-crypto.0.11.3 mirage-crypto-ec.0.11.3 mirage-crypto-pk.0.11.3 mirage-crypto-rng.0.11.3 mirage-crypto-rng-lwt.0.11.3 mirage-flow.3.0.0 mirage-kv.6.1.1 mirage-net.4.0.0 mirage-random.3.0.0 mirage-runtime.4.8.2 mirage-time.3.0.0 mirage-unix.5.0.1 mtime.2.1.0 multipart_form.0.6.0 multipart_form-lwt.0.6.0 num.1.5-1 obuilder-spec.dev ocaml.4.14.2 ocaml-base-compiler.4.14.2 ocaml-compiler-libs.v0.12.4 ocaml-config.2 ocaml-options-vanilla.1 ocaml-syntax-shims.1.0.0 ocaml-version.dev ocamlbuild.0.15.0 ocamlfind.1.9.6 ocamlgraph.2.1.0 ocluster-api.dev ocplib-endian.1.2 omigrate.0.3.2 opam-0install.0.5 opam-core.2.3.0 opam-file-format.2.1.6 opam-format.2.3.0 opam-repository.2.3.0 opam-state.2.3.0 optint.0.3.0 ordering.3.17.0 paf.0.5.0 parsexp.v0.16.0 pbkdf.1.2.0 pecu.0.7 pgx.2.2 pgx_lwt.2.2 pgx_lwt_unix.2.2 pp.2.0.0 ppx_compare.v0.16.0 ppx_custom_printf.v0.16.0 ppx_derivers.1.2.1 ppx_deriving.6.0.3 ppx_deriving_yojson.3.9.0 ppx_sexp_conv.v0.16.0 ppxlib.0.33.0 prettym.0.0.3 prometheus.1.2 prometheus-app.1.2 psq.0.2.1 ptime.1.2.0 randomconv.0.1.3 re.1.12.0 res.5.0.2 result.1.5 routes.2.0.0 rresult.0.7.0 seq.base session.0.5.0 session-cohttp.0.5.0 session-cohttp-lwt.0.5.0 sexplib.v0.16.0 sexplib0.v0.16.0 sha.1.15.4 solver-service.dev solver-service-api.dev spdx_licenses.1.2.0 sqlite3.5.2.0 ssl.0.7.0 stdint.0.7.2 stdio.v0.16.0 stdlib-shims.0.3.0 stdune.3.17.0 stringext.1.6.0 swhid_core.0.1 tailwindcss.dev tar.2.6.0 tar-unix.2.6.0 tcpip.8.0.0 timedesc.3.1.0 timedesc-tzdb.3.1.0 timedesc-tzlocal.3.1.0 tls.0.17.3 tls-lwt.0.17.3 tls-mirage.0.17.3 topkg.1.0.7 tyxml.4.6.0 uchar.0.0.2 unstrctrd.0.4 uri.4.4.0 uri-sexp.4.4.0 uuidm.0.9.9 uutf.1.0.3 x509.0.16.5 yojson.2.2.2 zarith.1.14"
ENV CI="true"
ENV OCAMLCI="true"
RUN opam update --depexts && opam install --cli=2.3 --depext-only -y ocaml-ci.dev ocaml-ci-web.dev ocaml-ci-service.dev ocaml-ci-gitlab.dev ocaml-ci-client.dev ocaml-ci-api.dev $DEPS
RUN opam install $DEPS
COPY --chown=1000:1000 . /src
RUN opam exec -- dune build @install @check @runtest && rm -rf _build

END-OF-DOCKERFILE
docker build .

Note that the commented out block where we create dummy dune-project files is the current workaround we have hacked into ocaml-ci to prevent spurious CI failures for users.

Running the above produces the following errors during the final opam install:

313.3 #=== ERROR while compiling solver-service-api.dev =============================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/solver-service)
313.3 # path        ~/.opam/4.14/.opam-switch/build/solver-service-api.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/solver-service-api-7-831243.env
313.3 # output-file ~/.opam/log/solver-service-api-7-831243.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling ocluster-api.dev ===================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocluster)
313.3 # path        ~/.opam/4.14/.opam-switch/build/ocluster-api.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/ocluster-api-7-607c66.env
313.3 # output-file ~/.opam/log/ocluster-api-7-607c66.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling ocaml-version.dev ==================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocaml-version)
313.3 # path        ~/.opam/4.14/.opam-switch/build/ocaml-version.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/ocaml-version-7-b1c25b.env
313.3 # output-file ~/.opam/log/ocaml-version-7-b1c25b.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling obuilder-spec.dev ==================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocluster/obuilder)
313.3 # path        ~/.opam/4.14/.opam-switch/build/obuilder-spec.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/obuilder-spec-7-c37df3.env
313.3 # output-file ~/.opam/log/obuilder-spec-7-c37df3.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling dockerfile.dev =====================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocaml-dockerfile)
313.3 # path        ~/.opam/4.14/.opam-switch/build/dockerfile.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/dockerfile-7-554e0b.env
313.3 # output-file ~/.opam/log/dockerfile-7-554e0b.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling current_rpc.dev ====================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocurrent)
313.3 # path        ~/.opam/4.14/.opam-switch/build/current_rpc.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/current_rpc-7-89bbf3.env
313.3 # output-file ~/.opam/log/current_rpc-7-89bbf3.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.
313.3
313.3
313.3 #=== ERROR while compiling current.dev ========================================#
313.3 # context     2.3.0 | linux/x86_64 | ocaml-base-compiler.4.14.2 | pinned(file:///src/ocurrent)
313.3 # path        ~/.opam/4.14/.opam-switch/build/current.dev
313.3 # command     ~/.opam/4.14/bin/dune subst
313.3 # exit-code   1
313.3 # env-file    ~/.opam/log/current-7-aaf51c.env
313.3 # output-file ~/.opam/log/current-7-aaf51c.out
313.3 ### output ###
313.3 # File ".", line 1, characters 0-0:
313.3 # Error: There is no dune-project file in the current directory, please add one
313.3 # with a (name <name>) field in it.
313.3 # Hint: 'dune subst' must be executed from the root of the project.

PR with a reproducing test: WIP

Specifications

Version of dune (output of dune --version): dune.3.17.0
Version of ocaml (output of ocamlc --version): ocaml.4.14.2
Operating system (distribution and version): all tested in ocaml-ci

Additional information

First reported in ocaml-ci is breaking builds that use pin depends ocurrent/ocaml-ci#990 and ocaml-ci is breaking builds that use pin depends ocurrent/ocaml-ci#990
@mtelvers did the debugging, diagnosis, and produced the workaround currently used in ocaml-ci in Fake dune-project file ocurrent/ocaml-ci#993
this workaround is a bit fragile, and has already required another workaround downstream in Remove vendored ppx_sexp_conv opam file ocaml-ppx/ppxlib#543
@Leonidas-from-XIV has suggested that this in behavior may have been introduced in fix: allow [$ dune subst] to work without a repo #11028 and it seems Regression in #11028 when using dune subst on a project with an opam file #11045 may be related (tho the specific problem is different).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Expected Behavior

Actual Behavior

Reproduction

Specifications

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Description

Expected Behavior

Actual Behavior

Reproduction

Specifications

Additional information

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions