Skip to content

test(flagd): adopt the OpenFeature Provider TCK - #1847

Draft
aepfli wants to merge 25 commits into
feat/provider-tckfrom
feat/provider-tck-flagd
Draft

aepfli wants to merge 25 commits into
feat/provider-tckfrom
feat/provider-tck-flagd

Conversation

@aepfli

@aepfli aepfli commented Sep 11, 2026

Copy link
Copy Markdown
Member

Stacked on #1830. Base is feat/provider-tck, so this diff shows only the new work. Review
#1830 first — it describes the suite, the adoption API and the capability vocabulary, and none of
that is repeated here.

Part of #1829 and open-feature/spec#417.

Adopts the conformance suite in the flagd provider, in both resolver modes.

Split out of #1830 so the suite and its first adopter are reviewed as separate questions: whether the
TCK is the right contract, and whether flagd satisfies it.

What it is

A shared abstract base and two subclasses differing only in resolver and port. The TCK brings its own
Gherkin, step definitions and Compose lifecycle, so a mode needs no registration and no build
configuration.

  • tck/AbstractResolverTest — shared configuration and the capability declaration
  • tck/RpcTest, tck/InProcessTest — one per resolver
  • tools/flagd-testbed/docker-compose.yaml — shared with the OFREP adoption; wraps the unmodified flagd-testbed image, which
    already serves both flagd and the launchpad control API. No host port bindings: the suite discovers
    dynamically mapped ports after startup.
  • pom.xml — the TCK test dependency, the tck profile, and the directory exclusions below

The ~18 other files in providers/flagd/.../e2e/ are untouched; they predate this work.

The adoption is a directory, not a naming convention. It lives in …/flagd/tck/ beside the e2e
package rather than inside it, and every selector is a path: **/tck/*.java for the tck profile's
includes, **/e2e/*.java,**/tck/*.java for the default build's exclusions. The class names lost the
Tck the selector used to be spelled with. testExclusions is a Surefire and not a compiler
exclusion, so the package still compiles in the default build — verified by injecting a reference to a
Capability constant that does not exist and watching the default build fail on it.

What it declares

Capability.declarableExcept(REINITIALIZATION) — everything else, in both resolvers. That now
includes @string-typing and @fully-typed-values: flagd carries a JSON type per flag and both
resolvers preserve it, so a non-string flag asked for as a string is a mismatch rather than a
rendering. Both were verified as executed, not merely declared.

@reinitialization is a choice the specification offers, not a gap. Requirement 2.5.2 says a
provider SHOULD revert to its uninitialized state after shutdown, and its supporting text says "some
providers MAY allow reinitialization from this state". Permitted, not required. An earlier revision of
this file recorded it as a KnownDeviation, which was wrong in the direction that matters — it
asserted a defect against a provider exercising an option the requirement gives it.

Note what is not in the list to remove: @large-integers. Java's integer accessor is a 32-bit
Integer, so 2^53 − 1 cannot be asked for by any Java provider — that is a property of the SDK rather
than of flagd, and the implementation now refuses the capability outright rather than expecting each
adopter to know it. Its scenario is still skipped, with a reason naming the accessor.

The one known deviation

@numeric-coercion is declared, its scenario fails, and knownDeviations() records why — the
shape Appendix F asks for. flagd narrows 0.5 to 0 through the integer accessor with no error code,
where the lossy half of its own accepted coercion ADR requires TYPE_MISMATCH with the code default.
The lossless half works and is not the defect. Both resolvers behave identically, which places it in
the shared provider layer rather than in either transport, so it is declared once. Tracked as
flagd#1996; delete the declaration and the
deviation together once it is fixed.

The backend stack

One Compose file serves both conformance suites in this repository, at
tools/flagd-testbed/docker-compose.yaml: the unmodified flagd-testbed image, service backend,
publishing 8013, 8015, 8016 and 8080 with no host port bindings — the suite requires dynamically
mapped ports and discovers them after startup. Each suite asks the harness only for the ports it
needs, so the extra ones cost nothing. It is a plain directory rather than a Maven module, so neither
provider reaches into the other's tree; both composeFile() overrides climb out of their module and
say why.

It used to be one file per adoption. Four hand-maintained copies across the four languages had
drifted on the one field that decides what is being tested — the image tag — which is exactly the kind
of divergence a conformance suite exists to prevent, occurring in the suite itself. That is now
resolved: all four languages run flagd-testbed:v3.10.1
, the current release, aligned up from the
v3.8.0 three of them were pinned to.

The bump fixes nothing on its own, which is worth saying because a version bump reads like a fix.
flagd-testbed#392 and
#394 are both still open, so v3.10.1 carries
neither the three missing precision flags nor the /start readiness fix — the fixture failures and the
readiness race are unchanged. The one behavioural change in range is
#390, which increases the simulated downtime,
and that is exactly the timing the @stale and @unavailable scenarios depend on.

The tallies below have been re-measured on v3.10.1 and are unchanged — same counts, same
failures by name. That is the expected result given the above: nothing in range touches what these
suites assert.

Deliberately not the testbed's own Compose file: that one bind-mounts a flags directory the
launchpad writes into, names its service flagd, and runs an envoy sidecar only the flagd e2e suites
need.

Running it, and the tally

mvn -pl tools/tck -am -DskipTests install     # once — tools/tck is not published yet
mvn -Ptck -pl providers/flagd test

Do not add -am to the second command: it drags tools/tck and tools/flagd-core into the
reactor and runs their unit suites first, which puts a conformance failure and a unit-test failure back
on one signal.

Each resolver runs 65 scenarios: 59 pass, 4 fail, 2 skip. 130 tests over the two classes, nothing
else in the log. The four failures are the same four by name in both resolvers — the coercion deviation
above, plus three where flagd-testbed serves neither integral-float-flag nor large-integer-flag,
which flagd-testbed#392 fixes. The three
fixture gaps get no deviation entry, because an entry there would attribute a backend gap to the
provider.

If a run reports far more than four, check for a leftover testbed stack before reading it as a
regression: one run here gave 29 failures, every one expected: TYPE_MISMATCH but was: FLAG_NOT_FOUND,
because a previous run's containers were still up and the second testbed served no flags at all. The
fix was to stop leaking containers, not to add a sleep.

Independence

This branch imports only base types — BackendEndpoint, Capability, ContainerizedProviderTckTest,
KnownDeviation. It needs nothing from the reporting branch (#1841), which is its sibling rather than
its parent, and git merge-tree reports zero collisions between them.

@aepfli
aepfli requested a review from a team as a code owner September 11, 2026 10:33
@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Runs the conformance suite against the flagd provider in both resolver modes.
The whole adoption is a shared abstract base and two subclasses that differ only
in resolver and port: the TCK brings its own Gherkin, its own step definitions
and its own Compose lifecycle, and works out which suite is running from the
JUnit test plan, so a mode needs no registration and no build configuration.

The Compose stack wraps the unmodified flagd-testbed image, which already serves
both flagd and the launchpad control API that this TCK's control API contract was
derived from. No host port bindings: the TCK discovers dynamically mapped ports
after startup, so the suite runs in parallel and does not collide with a
developer's local flagd.

capabilities() is declarableExcept(NUMERIC_COERCION). Evaluating float-flag (0.5)
through the integer API returns 0 with no error code rather than TYPE_MISMATCH
with the code default -- the value is silently truncated. Coercion as such is
permitted; it is the lossy case being accepted that is the defect, tracked as
open-feature/flagd#1996. Both resolvers behave identically, which places it in
the shared provider layer rather than in either transport, so it is declared once
here. Delete the override when the defect is fixed.

Split out of #1830 so that the suite and its first adopter are reviewed as
separate questions: whether the TCK is the right contract, and whether flagd
satisfies it.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
capabilities() withholding @numeric-coercion reads, from the outside, exactly
like a provider with no streaming transport declining @configuration-change: the
scenarios are skipped either way and nothing in the run says which of the two
happened. One is a limitation, the other is a bug, and a consumer comparing
providers needs to be able to tell.

knownDeviations() is the only place that can say so, because only the provider
author knows. Tracked against open-feature/flagd#1996. The summary names the
half of the coercion rule that is broken -- the lossy one -- because "flagd
coerces numbers" on its own reads as intended behaviour rather than as a defect.

Delete this and the capabilities() override together, once evaluating float-flag
(0.5) through the integer API reports TYPE_MISMATCH.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…s not a deviation

Three of the flags the suite's assets added are absent from flagd-testbed v3.8.0:
large-integer-flag, huge-integer-flag and integral-float-flag. Only the first is
reached by a scenario that runs here -- huge-integer-flag is asked for solely
under @large-integers, which is not applicable in Java, and integral-float-flag
solely under @numeric-coercion, which this provider withholds -- so exactly one
untagged scenario, the 32-bit precision one, fails FLAG_NOT_FOUND in both modes.
open-feature/flagd-testbed#392 is open for it; the Compose tag gets bumped when
it lands, which is why the note lives next to the tag as well as in the class.

A missing flag is a gap in the stack, not in the provider, so it is documented
rather than declared as a KnownDeviation. A deviation says the provider is wrong,
and the provider was never given the flag to get wrong.

The three falsy flags used to fail the same way and no longer do, which is worth
writing down because the failure looked identical. The testbed's zero-flags.json
already served boolean-zero-flag, integer-zero-flag and string-zero-flag with
zero/non-zero variants, while the canonical set called them false-flag,
zero-flag and empty-string-flag; the base moved the canonical names onto the
testbed's rather than the other way round, so those three scenarios now resolve
against flags that were always there.

Also says why capabilities() calls declarableExcept rather than
EnumSet.complementOf, which now matters more than it did: the complement would
claim @large-integers as well as the two reserved tags, and the suite refuses
that declaration at startup.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Two declarations examined rather than asserted, one of which retracts a mistake.

@reinitialization is withheld, and no KnownDeviation accompanies it. shutdown()
sets the sync resources' isShutDown flag and never clears isInitialized
(FlagdProvider.java:136-155, FlagdProviderSyncResources.java:27-28, 112-115), so
a later initialize() returns at its first check without rebuilding the resolver,
the gRPC channel it shutdownNow()'d, the retry scheduler it terminated or the
final errorExecutor it tore down (FlagdProvider.java:121-125). A shut-down flagd
provider is terminally shut down.

That is permitted. Requirement 2.5.2 says a provider SHOULD revert to its
uninitialized state after shutdown, and its supporting text says "some providers
MAY allow reinitialization from this state". Reuse is an option, not an
obligation, and declining it is one of the choices the requirement offers. An
earlier version of this file recorded it as an untracked KnownDeviation against
@lifecycle, which was wrong twice over: the scenario was mandatory only because
the spec's assets had not yet gated it, and the entry asserted a defect against a
provider behaving inside the requirement. Withholding the tag is the whole of
what is owed; the one scenario it gates is now reported as skipped with that
reason instead of failing in RPC mode.

The lesson is more useful than the correction. Nothing had checked whether 2.5.2
requires reuse before the failure was written up as a defect -- the scenario
failed, so a deviation was recorded. Find the numbered requirement first. This is
the third rule in the suite found asserted more strongly than the spec states it.

@lifecycle stays declared. flagd reaches its backend during initialisation in
both modes, so the remaining lifecycle scenarios assert something real, and Java
declaring it is what made the cross-language divergence visible in the first
place -- Go and JavaScript withhold it and are being changed to match.

@Stale is declared for both resolvers and that is correct. PROVIDER_STALE is
emitted from FlagdProvider.onError (FlagdProvider.java:258-264), which the shared
onProviderEvent switch reaches on PROVIDER_ERROR from either resolver
(FlagdProvider.java:197, 236), so the emit is in the provider layer rather than a
transport -- and the scenario passes in RPC mode as well as in-process. Go's flagd
provider withholds the tag for RPC; on this evidence that is a difference between
the implementations, not a property of the transport.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The TCK no longer sets @large-integers apart as "not applicable in Java". It is an
ordinary declarable capability, so declarableExcept(...) no longer leaves it out on
its own and this suite has to say so.

Nothing about the run changes. The scenario was skipped before and is skipped now,
for the same reason in substance: the SDK's integer accessor is a 32-bit Integer
and 2^53 - 1 has no room in it, so no Java provider can hold the tag. What changed
is where that is written down -- Appendix F, once, rather than a field in every
report -- and that the suite now expects the harness to withhold it rather than
refusing to let it be declared.

It stays out of knownDeviations, for the same reason the missing testbed flags do:
flagd is not at fault for a value the accessor cannot carry.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…ill owes

@Variants and @targeting arrived with the base's submodule bump, and
declarableExcept picks both up without a line of this file changing. That is the
right outcome and the reason nothing here said so, which is the problem: the
declaration grew by two capabilities and the file that argues every other one
either way was silent about them.

Two runs against flagd-testbed v3.8.0, one per resolver. RPC: 45 pass, 5
skipped, 2 failed. In-process: the same, after a first attempt whose opening
scenario timed out on a cold sync stream and passed on rerun -- a warm-up flake,
not a result. @targeting's three scenarios resolve targeting-key-flag through
flagd's own rule evaluation and pass in both modes on the image already pinned,
so the tag cost no bump; the Compose header now says so, because the note beside
that tag is where someone would otherwise go looking for a reason to bump it.

Both failures are the one testbed gap, and the @Variants outline reaches it a
second time: once for the untagged precision scenario, once for the row asking
for large-integer-flag's max-int32 variant. So the header's "one untagged
scenario" is a scenario short. Withholding @Variants would hide both -- and
would be a claim about flagd made to accommodate a missing flag, which is the
one thing a declaration must not be.

The capabilities() javadoc also still called @targeting reserved, in the
paragraph explaining why complementOf is the wrong call. It is declarable now,
and the paragraph is stronger for losing it: the reserved set shrinks as the
vocabulary fills up, so what protects the declaration is the form of the call
rather than the size of the set.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The tag arrives declared, because capabilities() is spelt "everything
declarable except", and it is right to arrive that way here — but that is
not the same as having been checked, so it was run. Both modes now
execute 56 scenarios and pass 49 of them, with the same five skips and
the same two testbed failures as before: all four rows of the new outline
pass in RPC and in-process alike. The provider substitutes the caller's
default for a flag whose state is DISABLED and reports no error code.

Nothing was needed upstream. The four disabled-* flags are
flagd-testbed's own, from flags/disabled-flags.json, present in the
v3.8.0 image already pinned and combined into the served set by the
launchpad; the canonical definition took the testbed's names and values
rather than inventing its own, as it did for the falsy flags. Recorded
next to the image tag, where the flags the testbed does *not* serve are
recorded.

Worth stating rather than assuming, because @disabled-flags is gated on
architecture rather than on quality, and flagd is on the side of that
line that can hold it: the in-process resolver evaluates locally, and the
RPC resolver still decides locally what to do with a response carrying no
value.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Import-path and coordinate churn only: the artifact is
dev.openfeature.contrib.tools:tck, the version range starts at 0.1.0, and
the four imports come from dev.openfeature.contrib.tools.tck. No
behavioural change, and no change to what is declared or withheld.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The first two in-process scenarios failed reproducibly on a slower host
with `Initialization timeout exceeded; did not complete within the 10000
ms deadline` out of FlagdProviderSyncResources.waitForInitialization. The
in-process resolver syncs the whole ruleset before it reports ready, and
the first scenario pays for a cold container on top of that; flagd doubles
the configured deadline, so 5000 gave it 10s and that was not enough.
15000 gives it 30s and both modes are clean apart from the two
flagd-testbed gaps already documented here -- 56 scenarios, 49 passing,
5 skipped, 2 failing, in RPC and in-process alike.

Diagnosed rather than guessed, because this looked at first like fallout
from the base dropping its 50ms post-command settle. It is not: the settle
was restored locally at 50ms and at 3000ms and fixed nothing, and the
failure is present on flagd-testbed v3.10.1 as well as on the pinned
v3.8.0. What it covers is the provider's own initialisation, which belongs
in a bound the scenario can see rather than in a sleep after an unrelated
control call. UNAVAILABLE_DEADLINE_MS is untouched, so the
initialisation-failure scenarios still assert promptness.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Switches the @numeric-coercion deviation from withheld-and-skipped to
declared-and-failing, which is the shape the TCK's settled guidance prefers, and
records why paying its cost is the honest report.

The guidance says withdrawing a capability in order to turn a failing scenario
into a skip is the failure mode the field exists to prevent, and that is exactly
what the old shape did here. Measured on the pinned testbed, in both modes: of
the tag's three scenarios, "An integer requested as a float is widened without
loss" passes. flagd therefore does coerce, and gets the narrowing direction
wrong - a skip cannot distinguish that from "flagd declines to coerce", and only
the second reading was available before.

The argument for the old shape was real and is recorded rather than dropped:
declaring the tag also fails "An integral float requested as an integer is
coerced without loss", because integral-float-flag is absent from flagd-testbed
v3.8.0. That cost is accepted because it is not a new kind of cost - this
adoption already carries two failures from the same missing flags and records
them plainly - and because the alternative hides a real defect behind a stack
gap.

Measured result, both modes: 56 scenarios, 2 skipped (@reinitialization,
@large-integers), 4 failing - one provider defect and three testbed gaps.

Also records something the previous pass reported as fixed and which does not
hold on a loaded host: the first scenario of errors.feature still errors in
in-process mode with an initialisation timeout against the doubled 30000 ms
deadline. Reproduced three times, and reproduced identically with
@numeric-coercion withheld, so it is not a consequence of this change. Thirty
seconds is not a plausible sync time for this ruleset and only the mode that
must establish a sync stream after the first POST /start is affected, so it
reads as stack-side readiness - the class of defect
open-feature/flagd-testbed#394 closes. The deadline stays at 15000 rather than
being raised again: a suite that sleeps instead of holding the control API to
its promise stops being able to detect when the promise breaks.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The two TCK suites have been excluded from the default build since they were
added, via <testExclusions>**/e2e/*.java</testExclusions>, and nothing said so.
An exclusion nobody writes down is indistinguishable from an oversight - which
is how providers/ofrep came to run a Docker-dependent suite in its default build
unnoticed, the same mistake in the other direction.

So the README now states the policy and its reasoning: the suites are
Docker-gated, excluded from every job that exists, and run by hand by a
maintainer before merging a change to resolution, event or lifecycle behaviour,
with the result quoted in the pull request. A scheduled or path-filtered
workflow was considered and declined. It also gives the two commands and says
which failures are expected, so that a reader can tell a regression from the
recorded testbed gaps and the recorded coercion defect.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
… not doing

Narrows the e2e profile's testExclusions from empty to **/e2e/*TckTest.java, so
the exclusion the default build applies is no longer undone in the one job that
matters.

A previous pass reported that the TCK suites "are excluded from every job that
exists". That was wrong, in two steps that have to be read together:
providers/flagd's e2e profile sets <testExclusions/>, clearing the exclusion, and
ci.yml's `main` job activates that profile on every push. Verified rather than
re-read:

  $ mvn -Pe2e -pl providers/flagd help:evaluate -Dexpression=testExclusions
    (empty)
  $ mvn      -pl providers/flagd help:evaluate -Dexpression=testExclusions
    **/e2e/*.java

So FlagdRpcTckTest and FlagdInProcessTckTest were running in CI, on a runner
that does have a Docker daemon, and they are expected to fail - three testbed
gaps and one recorded coercion defect. Every unrelated pull request touching this
module would have gone red for a reason that has nothing to do with it.

Narrowing rather than clearing is what keeps both halves true: the legacy
Run*Test suites over the test-harness submodule still run under -Pe2e exactly as
they do on main, and only the two TCK suites stay out. The pattern matches
AbstractFlagdTckTest.java as well, which is harmless - it is abstract and
surefire would not select it - and testExclusions only filters what runs, never
what compiles.

Not runnable locally as a cross-check: providers/flagd/test-harness is an
uninitialised submodule on this machine, so the legacy suites cannot be executed
here to prove they still get selected. The pattern is checked against the
directory's file names instead, which is deterministic:
Run{File,InProcess,Rpc}Test do not end in TckTest.

The README says all of this too, in the section added with the exclusion policy,
because the narrowing is the half a reader would not guess from the POM alone.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Appendix F now carries "Running the suite in CI", promoted there because the
same reasoning restated in four adoption READMEs is where it drifted. So this
section keeps the mechanism -- the testExclusions property, what the e2e profile
narrows it to and why the narrowing rather than the clearing, that the exclusion
is Surefire's and not the compiler's, and the help:evaluate command that resolves
it -- and links to the appendix instead of paraphrasing the argument.

The resolved values are checked, not read:

  mvn -Pe2e -pl providers/flagd help:evaluate -Dexpression=testExclusions
      -> **/e2e/*TckTest.java
  mvn       -pl providers/flagd help:evaluate -Dexpression=testExclusions
      -> **/e2e/*.java

The locally decided part stays stated here rather than deferred: a scheduled or
path-filtered workflow was considered and declined, and a maintainer quotes a
hand-run result in the pull request.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The TCK gained @standard-reasons, and this suite declares every declarable
capability except @reinitialization and @large-integers -- so it picked the new
tag up by default and started running reason.feature without anyone deciding
that it should. That is the right answer here, but it was measured before it was
written down rather than after.

All nine scenarios pass in both modes, including the two that compose with
@targeting and @disabled-flags: STATIC for the rule-less flags, TARGETING_MATCH
and DEFAULT either side of targeting-key-flag's rule, DISABLED for a disabled
flag, and ERROR beside FLAG_NOT_FOUND and TYPE_MISMATCH. So the claim the tag
makes -- the standard vocabulary with the standard meanings -- holds for the RPC
resolver and the in-process one alike.

Each mode is now 65 scenarios, 59 passing, 2 skipped and 4 failing, up from 56
and 50. The four failures are the same four as before and none of them is new:
the lossy numeric coercion that flagd#1996 tracks, and the three flags
flagd-testbed v3.8.0 does not serve. The cold-start initialisation error recorded
on CONNECTED_DEADLINE_MS did not reproduce in this run; it is intermittent and
host-dependent, so the note stays and the count says which run it comes from.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
capabilities() is now declarableExcept(REINITIALIZATION). @large-integers goes,
and its going is the whole point: it was never a decision this suite took. flagd
does not decline to resolve 2^53 - 1 -- Client.getIntegerDetails is a 32-bit
Integer, so no Java provider can be asked, and the TCK refuses the capability
centrally rather than having each adoption remember. The paragraph that used to
explain it here was one of four saying the same thing about the same language.

@reinitialization stays withheld, and stays this suite's call. That one is a fact
about flagd: FlagdProviderSyncResources keeps isInitialized and isShutDown as
separate flags and refuses initialize() when either is set, which Requirement
2.5.2 permits, so withholding is the honest report and no KnownDeviation is owed.
Nothing about the SDK's accessor has any bearing on it, which is why one moved and
the other did not.

Measured, both resolvers, on the pinned testbed image: 65 scenarios, 59 passed,
2 skipped, 4 failed -- identical to the previous pass, because this changes why a
scenario is skipped rather than whether it is. The two skips now read differently
in the results, which is the observable part:

  Skipped: provider does not declare capability REINITIALIZATION (tag
  @reinitialization). Declared capabilities: [...]

  Skipped: the Java SDK cannot express capability LARGE_INTEGERS (tag
  @large-integers) - Client.getIntegerDetails takes and returns a 32-bit Integer
  ... not the provider under test declining

A reader of the report can now tell which of the two absences describes flagd.
The four failures are unchanged: flagd#1996's lossy coercion plus the three flags
testbed v3.8.0 does not serve.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…hand

The javadoc beside knownDeviations() argues from first principles that declaring
@numeric-coercion is right even though one of its three scenarios asks for a flag
the pinned testbed does not serve. Appendix F now states that as a rule, so the
argument belongs upstream and the citation belongs here.

Both halves of it apply in order, which is worth spelling out because the first
half is what keeps the rule from over-reaching: flagd is attempting the coercion
-- the widening scenario passes, which is the evidence -- and two of the three
scenarios can be put to it. A provider that does not coerce at all stops at the
first half and withholds, as the SDK's in-memory provider does.

Also names the appendix's other consequence as the reason the deviation summary
names the testbed: a scenario that fails because the backend cannot serve its
fixture is not a provider defect, and a summary that did not say so would leave
the report accusing flagd of the stack's gap.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
`mvn --projects providers/flagd -P e2e test` does not run the conformance
suites. The `e2e` profile is the thing that keeps them out: it narrows the
exclusion to **/e2e/*TckTest.java so the legacy Run*Test suites run and the
TCK ones do not. Verified by running it rather than reading the POM - 788
tests in 12:18, RunFileTest, RunInProcessTest and RunRpcTest, and not one
mention of either TckTest class anywhere in the log.

Appendix F now asks for a step of its own rather than a corner of an existing
e2e suite, because of what a red build says: `-Pe2e` red means the provider's
own end-to-end suites regressed, while `-Ptck` red means conformance failed,
and a conformance run carries failures by design wherever AbstractFlagdTckTest
declares a knownDeviation.

The new `tck` profile clears the exclusion and narrows Surefire's includes to
**/e2e/*TckTest.java in the same breath, which is what makes it a conformance
step and not a wider one: `mvn -Ptck -pl providers/flagd test` runs
FlagdRpcTckTest and FlagdInProcessTckTest and nothing else - 130 tests, 65
scenarios per resolver, 59 passing, 2 skipped and 4 failing in both. Nothing
activates the profile in CI, for the reason the `e2e` profile's comment
already gives.

The exclusion stays Surefire's and not the compiler's, so both suites still
compile in the default build.

The README says not to add `-am` to that run, and why: it pulls tools/tck and
tools/flagd-core into the reactor and runs their suites too, so a failure in
either comes out as a `-Ptck` failure. A one-off `install` is what `-am` was
there for.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The two conformance suites lived under e2e/ and were selected out of it by
filename: `**/e2e/*TckTest.java`, in three places. They now live in a sibling
package, `src/test/java/.../flagd/tck/`, and every selector names the directory
instead.

Nesting them under e2e/ said "this is a kind of e2e test", which is the
conflation the separate `tck` step exists to undo. The two suites answer
different questions -- e2e tests this provider against flagd's own test harness,
tck tests it against the OpenFeature provider contract -- and they mean
different things by a red run: an e2e suite is expected green, while a
conformance suite fails scenarios by design wherever a knownDeviation is
declared. A file is now in the conformance directory or it is not, and no
convention about class names holds that line.

So the names drop what the directory now says. FlagdRpcTckTest in package
...flagd.tck said "tck" twice and "flagd" twice; it is RpcTest, beside
InProcessTest, over AbstractResolverTest. The `*Test` suffix stays because
Surefire's default includes need it -- that is not the selector being removed.

The three selectors, all directory-shaped now:

  * the module's default testExclusions gains tck, so both Docker-dependent
    packages stay out of `mvn verify`;
  * the `e2e` profile drops e2e from the exclusion and keeps tck, where it used
    to narrow to a filename pattern;
  * the `tck` profile is its mirror image -- drops tck, keeps e2e, and includes
    `**/tck/*.java`.

Both halves of the `tck` profile are still needed, for the reason they always
were: the include alone leaves the exclusion in force and runs nothing, and
dropping the exclusion alone runs the module's unit tests alongside the suites.
testExclusions is still a Surefire and not a compiler exclusion, so both
packages still compile in the default build.

RpcTest and InProcessTest now state their configuration() rather than deriving
it. The default derivation reads the class name, so the rename would have filed
their runs as "rpc" and "in-process" instead of "flagd-rpc" and
"flagd-in-process" -- a report is read away from this repository, where the
provider's name is the half that matters. The suites are otherwise unchanged.

Unchanged too: 65 scenarios per resolver, 59 passed, 2 skipped, 4 failed, in
both modes. Same scenarios, same results, a different directory.

The profile comment documenting the run had kept `-am` on it, which the READMEs
already warn against; it says the command that works now.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The TCK section was 5.5 KB of a 28 KB provider README, and most of it
was the base README's or Appendix F's: why an adoption suite is
excluded rather than gating, why it gets a step of its own, what a red
conformance build says, why both halves of the tck profile are needed,
and why the -am the command must not carry would mix signals.

What is left answers the three questions an adoption README owes a
reader. What this provider declares and why each absence is what it is -
by pointing at AbstractResolverTest, where the reasoning sits next to
the declaration and is measured rather than asserted. What the tally is
and which failures are expected - 65 scenarios, 59/2/4 per resolver,
three testbed gaps and one real defect. And the command that runs it.

One local fact stays at length because it is a trap and is recorded
nowhere else: -Pe2e does not run these suites, it is the profile that
keeps them out, and it used to clear the exclusion outright and run them
red in CI on every unrelated pull request.

2.4 KB, from 5.5. The adoption now adds 38 lines to this README rather
than 85.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…round it

Stripped of comments the two TCK Compose files in this repository were
the same stack: same image, same service name, same absence of host
bindings, differing only in which ports each listed. Two hand-maintained
copies is the mechanism by which they drift onto different images while
both claiming to ask the same questions, so there is now one file at
tools/flagd-testbed/docker-compose.yaml exposing 8013, 8015, 8016 and
8080. Extra ports cost nothing: with no host bindings the TCK maps each
container port dynamically and resolves only the ones a suite asks for,
so the flagd suites never look up 8016.

Placed outside both provider modules rather than in either, so neither
reaches into the other's test tree and a third adoption against the same
backend reads the same path. That costs a tools/ directory which is not
a Maven module -- no pom, no CHANGELOG, nothing released -- and a
composeFile() that climbs out of its own module, which both suites now
explain rather than leaving to look like a mistake.

The comment is down from 29 lines across two files to 13 in one, and
keeps only what is a trap here: that it is NOT the legacy e2e stack at
providers/flagd/test-harness, which bind-mounts ${FLAGS_DIR}, names its
service "flagd" and runs an envoy sidecar. Which flags the image does
not serve belongs to open-feature/flagd-testbed#392, and which scenarios
that costs is in AbstractResolverTest.

AbstractResolverTest loses the testbed-gap essay, the falsy-flag rename
history and two paragraphs about what an earlier revision of the file
declared. Every measured probe stays: the @numeric-coercion three-way
result, the @reinitialization and @Stale source-line evidence, the
@disabled-flags and @standard-reasons runs, and both findings recorded
on CONNECTED_DEADLINE_MS.

Comments and YAML comments only. No behaviour change.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Capability.DISABLED_FLAGS no longer frames the gate as where evaluation
happens, so the sentence here that echoed it says what was actually
measured instead: both resolvers are told the flag is disabled.

Comments only.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Measured this pass: an RPC run came back with a fifth failure,
FLAG_NOT_FOUND on an evaluation.feature row expecting no error code, and
the next run of the same tree was clean. Same shape the OFREP adoption
already records against open-feature/flagd-testbed#394, so it is named
here rather than left for the next reader to diagnose as a regression.

Comments only.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Four languages pinned this image by hand and had drifted: JavaScript was already
on v3.10.1 while Go, Java and Python sat on v3.8.0, so the suites whose results
are only comparable if they asked the same backend were asking two. v3.10.1 is
the current release, so aligning up rather than down.

What this does NOT fix, stated because the tag is easy to mistake for a fix:
open-feature/flagd-testbed#392 and #394 are both still open, so v3.10.1 carries
neither the three missing precision flags nor the /start readiness fix. The
fixture failures and the readiness race are unchanged.

The one behavioural change in range is open-feature/flagd-testbed#390, which
increases the simulated downtime -- and that is exactly the timing the @Stale and
@unavailable scenarios depend on. Tallies recorded against v3.8.0 have not been
re-measured on this image, including the failure ranges the adoption READMEs cite
by tag; those figures stay as they are because they are a record of what v3.8.0
did, and re-taking them is follow-up work rather than a rewrite.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
The four scenarios the new tag gates were mandatory and passing before it
existed, so the "everything except @reinitialization" default already declares
it. Recorded here anyway, because this file's standard is that each declaration
rests on evidence from a run and not on inheriting the default.

Measured in both resolver modes: 65 scenarios, 2 skipped, and the same four
failures as before -- the lossy numeric coercion (open-feature/flagd#1996) and
the three flags the pinned testbed image does not serve
(open-feature/flagd-testbed#392). None of the four @string-typing scenarios is
among them. flagd's flag definitions carry a JSON type per flag and both
resolvers preserve it, so a non-string flag asked through the String accessor
is a real mismatch here and is reported as one.

RPC additionally showed the intermittent "half" variant failure this branch
already records -- surefire's reruns had it pass four times in five, which is
the testbed readiness window of open-feature/flagd-testbed#394 and not a
property of any assertion. The clean-run tally is unchanged.

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
Appendix F split @string-typing at bda599f1: the outline keeps boolean-flag
and integer-flag, and float-flag and object-flag moved behind a new
@fully-typed-values that asks whether the store records a native type for
those two as well.

flagd is the case the split was not written for, and declaring both is how
that shows. Its flag definitions carry a JSON type per flag and both
resolvers preserve it, so all four questions have the same answer here. A
partially typed backend declares the first tag and withholds the second;
there is nothing partial about this one.

declarableExcept(REINITIALIZATION) already picks the new tag up, so this is
javadoc rather than a declaration change — but the tag was measured rather
than inherited, which is this file's standing rule. Both modes after the
re-pin: 65 scenarios, 59 passing, 2 skipped, 4 failing, identical to the
run before it. The two skips are still the withheld @reinitialization and
@large-integers; the surefire report has "A float flag is not returned as
its string representation" and "A structured flag is not returned as its
JSON text" as executed and passing, which is the part a tally alone would
not have shown — an undeclared @fully-typed-values would have turned both
into skips and left the failure count untouched.

The four failures are unchanged and none is a string-typing scenario: the
lossy numeric coercion (open-feature/flagd#1996) and the three flags the
pinned testbed image does not serve (open-feature/flagd-testbed#392).

Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
@aepfli
aepfli force-pushed the feat/provider-tck-flagd branch from 4ab1eea to aff72ac Compare September 16, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants