4digital Lab4digital Lab4digital Lab4digital Lab
← All research

Lab · Research

Can a leakage test miss... a leak?

In brief: In a controlled experiment, I deliberately introduced information from the future into the data. I then checked whether tests designed to detect such errors actually recognise them. A test that does not detect an error does not yet prove that the error is absent.

  • validation
  • data-leakage
  • time-series
  • testing

Objects of study

O-01
object of study Single-cut test: a comparison of computations on the whole series and on a prefix ending before the cut point.
O-02
object of study Point-wise test: a comparison of the value computed solely from data available up to a chosen point with the value for that point obtained on the full series.
Origin of the material

A study on synthetic data, designed and prepared with AI assistance.

The hypotheses and criteria were written down before the code was written; the experiment was run once on the author's own hardware.

Version
1.0

01 / Research problem

In a controlled experiment, I deliberately introduced information from the future into the data. I then checked whether tests designed to detect such errors actually recognise them.

For some configurations the result was unambiguous: the test did not detect a single error in 1,000 data series, even though the error was present in every one of them.

This did not mean the test had broken down. The comparisons ran according to the adopted procedure. The problem was that the very design of the test made it impossible to see a particular kind of irregularity.

So I asked: how do you check the reliability of a control mechanism that may itself fail to recognise the errors it is supposed to protect against?

How does information from the future get into the data?

Consider eight consecutive readings.

An illustrative example — the numbers serve to explain the mechanism and do not come from the experiment.

Reading12345678
Actual value101191213121415
After faulty aggregation1212121215151515

I divided the readings into two intervals of four observations each. I then assigned the closing value of each interval to all the points belonging to it.

As a result, the first reading receives the value 12, although that value will only appear at the fourth reading. Likewise, the fifth reading receives the value 15, known only at the eighth.

The calculations are arithmetically correct. The problem concerns when the information becomes available.

If an analysis result assigned to a given moment uses data that appeared later, we are dealing with leakage of information from the future.

Such an error can go unnoticed, because the values do not have to be wrong. They are real — they were simply used too early.

Why might a test not notice it?

One method of detecting leakage is to compare two computations. The first uses the whole data set. The second receives only its initial fragment, as if the later observations did not exist yet.

If the computation used the future, the results may differ. The test will detect the leak only if the difference shows up in the compared range.

Let us return to the example.

Cut after the fourth reading: both computations see a completed interval and assign the value 12 to its points. The results are identical. The test reports no error.

Cut after the third reading: the shorter fragment ends with the value 9, while the full interval ends with the value 12. The results differ. The test detects the error.

The same data set, the same transformation and the same test. Only the place where the data was cut off changed.

This is the first mechanism I put under study.

The second mechanism: an event recognised after the fact

Not all leaks come from aggregating data. Some appear during pattern recognition.

Consider three consecutive values:

3 → 5 → 4

The middle point is a local maximum. This can only be established, however, after the third value has been observed, because only then is it known that a decline has occurred.

If the analysis result marks the maximum already at the value 5, it uses an observation that was not available at that moment.

An additional problem appears at the end of the data set. It is not yet known whether the last point will turn out to be a maximum, because the subsequent observations are missing.

This uncertainty can be recorded as an empty value or as zero.

I decided to check this variant as well, because a seemingly technical decision can be of fundamental importance: a test that skips empty values may remove from the comparison exactly the places where the leak becomes visible.

02 / Hypotheses

I designed the experiment so as to deliberately introduce the two errors described and check the effectiveness of different ways of detecting them.

Before writing the code, I set out the hypotheses, the predictions and the evaluation criteria. The pre-registration was frozen before the experiment started, as confirmed by the time of the record on the repository server.

I wanted to settle three main questions.

First: does the position of the cut relative to the interval boundaries decide whether the aggregation error is detected?

Second: can the way undefined values are handled make the test miss the leak?

Third: does increasing the number of points checked improve the detection of errors that appear rarely?

In addition, I checked that the same tests do not report errors on correctly processed data.

Each hypothesis had a defined criterion of success or failure. I have included the full list, together with the numerical predictions, in the methodological appendix.

03 / Experimental method

I prepared 1,000 synthetic time series, each containing 17,280 observations.

I generated the data as a random process. Successive increments were independent, so earlier increments did not make it possible to predict later ones. The construction contained no predictable signal in future increments.

On the same data I applied two kinds of transformation: a correct one, using only the information available at a given moment, and a faulty one, deliberately using future observations.

I then compared two test designs.

The single-cut test computed a given feature on the whole history and on a shortened fragment, and then compared the results over the common range. I placed the cut at an interval boundary or at a random position.

The point-wise test selected a set number of observations and, for each of them, checked whether the value computed solely from the history available up to that moment matched the value obtained on the full data set.

I studied configurations using 10, 40 and 400 check points.

I also included two variants of comparing the results: one that treated a difference between an empty value and a defined value as an error, and one that skipped empty values.

As the primary measure I adopted the share of series in which the test reported at least one deviation.

On data with deliberate leakage this meant detection effectiveness. On the correctly processed counterparts — the frequency of false alarms.

I ran the experiment once. Before the results were analysed, checks of the instrument were also carried out, including a check that computations on shortened fragments agree with computations on full prefixes.

04 / Results

1. Interval boundary: zero detections

Result: 0 out of 1,000 series for each of the three interval sizes studied.

When the cut fell exactly on an interval boundary, the single-cut test did not detect the leakage caused by faulty aggregation.

This was not an accidental miss. In that position both computations used the same completed intervals and returned identical values.

When I chose the cut at random, the effectiveness depended on the interval length and came to 74.4%, 90.9% and 97.7% respectively.

The point-wise test, which checked the availability of information at many moments, detected this kind of leakage in all the series and configurations studied.

2. Empty values: zero or one hundred percent

Result: 0% versus 100% detections, depending on the comparison method.

In the second variant I used local maxima, whose recognition requires knowledge of later observations.

When I marked missing values as empty, the test that skipped them during comparison did not detect the leakage in any of the 1,000 series.

The same test, run on the same data but treating a difference between an empty value and a defined value as a deviation, detected the leakage in all the series.

In the point-wise test, the variant that skipped empty values did not obtain a single comparable point. I classified this as an inconclusive result, not as passing the test.

This is a particularly important distinction. No deviations detected and no possibility of making the comparison do not mean the same thing.

3. Rare events: forty points were not enough for reliable detection

Result: 45.7% detections at 40 points, versus 99.9% at 400 points in the configuration with the widest window studied.

A local maximum is an event that occurs only in some of the observations.

If a test checks only selected points of the series, it has to hit a place where the error becomes visible.

When undefined values were recorded as zero, the single-cut test detected the leakage in about 28–32% of the series, depending on the configuration.

In the point-wise test I observed detectability rising with the number of observations.

For the widest window studied I obtained:

Number of points checkedDetection
1011.0%
4045.7%
40099.9%

This does not mean that 400 points is a universally sufficient number. The result depends on the frequency of the event studied, the length of the series and the placement of the checks.

False-alarm check

In the 90 configurations of correct counterparts of the transformations that I studied, with 1,000 series each, I did not observe a single false alarm or inconclusive result.

I do not treat this as proof of zero risk of false alarms in all applications. The result shows only that in the tests carried out the problem was insufficient detectability of known errors, not an excessive number of reports on correct data.

05 / Deviations and falsification

The hypotheses on leakage detection met all the criteria set in advance. This does not mean, however, that every observed value was exactly in line with the mathematical prediction.

In the configurations using ten check points I obtained three results lower than expected: 71.0% instead of 74.4%, 39.4% instead of 43.0% and 11.0% instead of 13.3%.

The differences fell within the pre-registered tolerance of five percentage points, so the hypothesis met the formal criterion. At the same time, the 95% Wilson intervals for these three results did not include the predicted values.

I made all three measurements on the same series and check points. The results are therefore not independent observations, and I did not establish the cause of the common deviation.

I did not carry out an additional run to explain it. I leave this result as a documented unknown, not as grounds for changing the hypotheses or criteria after the fact.

06 / Conclusions

A test that does not detect an error does not yet prove that the error is absent. First it has to be shown that it can recognise an error whose existence is certain.

In the experiment carried out I identified three specific ways in which a control mechanism can fail to do its job.

First, a test may compare data at such a place that the error remains invisible, even though it occurs earlier in the analysed history.

Second, the way undefined values are represented and skipped may remove from the check precisely those observations that would reveal the irregularity.

Third, a spot check may have an insufficient chance of hitting a rare event, even if it correctly recognises the error at a point it checks.

On this basis I formulate a practical rule: it is worth checking a validation mechanism on controlled examples of the errors it is meant to detect. In such a check, the construction of the data, the comparison method and the number of observations all matter.

I believe these conclusions may also matter when preparing features for machine learning models. In this study, however, I did not train or evaluate any ML model, so I present no results on the quality of learning or prediction.

07 / Limitations

I carried out the study on synthetic series with independent, normally distributed increments. I did not study data from real systems, seasonality, conditional volatility, heavy tails or missing observations in the input data.

I evaluated two kinds of leakage and two test designs, in strictly defined configurations. I do not automatically extend the results obtained to all data transformations, validation methods or analytical systems.

I used one implementation of the instrument and carried out one run of the experiment. No independent implementation was made, and I have left the unexplained deviation at ten points open.

The experiment determines the effectiveness of detecting known, deliberately introduced leaks. It is not a general guarantee that data passing a given test is free of unauthorised use of information.

Methodological appendix — details of the experiment

In the main part I presented the research question, the leakage mechanisms, the results and their interpretation. Below I give the full hypotheses, definitions, parameters and results needed for a technical assessment of the experiment.

Hypotheses and criteria

IDSubjectPre-registered criterion
P1Aggregation, single cut at an interval boundaryDetection ≤ 1% for every size
P2Aggregation, random cutDetection close to 1 − 1/k, tolerance ±5 pp
P3Local maximum, NaN, skipping comparisonDetection ≤ 1%
P3bLocal maximum, NaN, strict comparisonDetection ≥ 99%
P4Local maximum, zero, single cutDetection close to w · p(w), tolerance ±5 pp
P5Local maximum, zero, point-wise testDetection close to 1 − (1 − p(w))^(N−1), tolerance ±5 pp
P6Local maximum, NaN, point-wise testStrict: ≥ 99% detections; skipping: ≥ 99% inconclusive results
P7Aggregation, point-wise test at 40 pointsDetection ≥ 99%
P8Clean counterparts of all transformations0 false alarms in 1,000 series in every cell

Data model

Each series consisted of 17,280 points. Successive increments of the logarithm of the value were drawn independently from a normal distribution with mean zero and standard deviation 0.001.

The value of the series was obtained by exponentiating the cumulative increments.

I used the generator numpy.random.default_rng(20261106 + i), where i took values from 0 to 999. The same series were used for the comparisons of all test variants.

Transformations

Aggregation F1: the interval length k was 4, 12 or 48 points. The leaking variant assigned to the points the value at the end of the current interval. The clean variant used the end of the previous, completed interval.

On the shortened data, the last, incomplete interval in the leaking variant took the last available value.

Local maximum F2: the window width on each side was w = 2, 5, 20. The leaking variant marked the maximum at its own point, using earlier and later observations. The clean variant marked it only once the required later observations had appeared.

Where the full window was not available, NaN or zero was used, depending on the configuration.

Test designs

O-01: a comparison of computations on the whole series and on a prefix ending before the cut point. The boundary cut was T = 8640. The random cut was drawn uniformly from the range between one quarter and three quarters of the series length, from the generator of the given series.

O-02: a comparison of the value computed solely from data available up to a chosen point with the value for that point obtained on the full series. 10, 40 or 400 evenly spaced points were checked, from index 200 to the end of the series, rounded to integer indices.

For numerical values, a deviation was a difference greater than 10⁻¹² × max(1, |b|), where b denoted the reference value. In the strict variant, a difference between NaN and a defined value was also counted.

No comparable points meant an inconclusive result.

Basis of the predictions

For a symmetric random walk with a continuous distribution of increments, the probability of a local maximum in the window from t−w to t+w is:

p(w) = (C(2w,w) / 4^w)²

This gives about 14.1% for w = 2, 6.1% for w = 5 and 1.6% for w = 20.

Because two such maxima cannot lie closer than w points apart, the probability of detecting a maximum among the last w points before the cut is w × p(w).

For m check points with non-overlapping windows, the probability of finding at least one maximum is:

P(detection) = 1 − (1 − p(w))^m

In the point-wise test studied, the last observation of the series could not produce a deviation for this feature, so for N points I took m = N−1. The remaining check points were far enough apart for their windows not to overlap.

Full results — aggregation

Interval lengthCut at the boundaryRandom cut
40.0%74.4% (75.0%)
120.0%90.9% (91.7%)
480.0%97.7% (97.9%)

Predictions are given in brackets. For the single-cut test, both versions of the comparison gave identical results. The point-wise test detected the leakage in 100% of the series for all three interval lengths and all numbers of check points.

Full results — local maximum, single cut

Each cell gives the result for the boundary cut and for the random cut.

Window widthNaN, skippingNaN, strictZero, strict
20.0% / 0.0%100% / 100%28.2% / 27.5%
50.0% / 0.0%100% / 100%29.8% / 30.6%
200.0% / 0.0%100% / 100%31.8% / 32.0%

The predictions for the zero variant were 28.1%, 30.3% and 31.4% respectively.

Full results — local maximum, point-wise test

Undefined values recorded as zero.

Window width10 points40 points400 points
271.0% (74.4%)99.9% (99.7%)100% (100%)
539.4% (43.0%)90.9% (91.3%)100% (100%)
2011.0% (13.3%)45.7% (46.1%)99.9% (99.8%)

Predictions are given in brackets.

For ten check points the 95% Wilson intervals were 68.1–73.7%, 36.4–42.5% and 9.2–13.1% respectively. None of the three included the predicted values.

For 40 points and a window width of 20 the Wilson interval was 42.6–48.8%.

Instrument check and execution

For 20 series, the correct versions of all features passed the point-wise test at all 3,074,400 points checked. In 9,000 checks, the computation on the shortened fragment was identical to the computation on the full prefix. In each of the 20 series, the leaking variant differed from the clean one.

I ran the experiment using Python 3.13.5 and NumPy 2.5.4. A single run using 48 parallel processes took 2.9 seconds.

The study uses synthetic data, so repeating it independently does not require access to private data sets. Statistical agreement of a separate implementation does not necessarily mean obtaining identical series with a different random generator.

08 / From the author

In engineering practice I often come across situations in which both the computational mechanism and its tests are technically and logically correct. Even so, the result obtained turns out to be wrong, and in some cases even temptingly optimistic. The cause lies outside the area the check covered.

This is also the case in the study presented here. The problem does not have to lie in the algorithm itself or in the implementation of the test, but in the way the data that reaches it is prepared and aggregated.

A lack of appropriate safeguards already at the stage of designing the data-processing mechanism can mean that the further computations are carried out correctly, but on the basis of information that should not yet be available.

In my experience, this is one of the less obvious categories of engineering problems. It is easy to overlook during design and hard to identify later, because the individual components may work exactly as they were designed to.

The correctness of the individual elements does not guarantee the correctness of the result of the whole process.

How to cite

Bartłomiej Pietrzyk, “Can a leakage test miss... a leak?”
4digital Lab, 7 November 2026, https://lab.4digital.cloud/en/can-a-leakage-test-miss-a-leak/

Terms for using the write-ups and data are agreed individually.