A correct migration that failed its own threshold

In a pilot benchmark, a change of data layout in an experimental JVM engine delivered 65.5% higher throughput.

It is the kind of result that makes it easy to start planning a migration of the whole system.

Instead of keeping a separate object for every data point, the new variant organised the values in separate arrays. Such a representation can reduce object overhead and make better use of the processor cache.

The result looked promising. It had, however, an important limitation: the pilot tested a simplified part of the computation, with no dependence on state carried between successive operations.

When the columnar variant was tested on a representative stateful workload, the throughput improvement was 1.80%.

The new implementation had been tested, and the preserved documents reported that both variants produced matching results. That still did not justify replacing the existing solution.

To understand why, we have to go back to the problem that led to this experiment.

The performance work was part of developing a private, experimental JVM computation engine.

The engine processed ordered sequences of data, performing repeated calculations whose result could depend on state built up in earlier steps. Both throughput and the cost of memory allocation mattered.

In a system like this, performance counts. Every additional configuration of the computation, every variant of a hypothesis under study and every rerun adds to the amount of work to be done.

That does not mean, however, that every element should be optimised at any cost.

The existing implementation was already optimised. Further changes therefore had to be judged against a real baseline, not against the theoretical benefit of using a particular technology.

I was considering several directions: reducing object allocation, changing the data representation, using vector operations and moving selected structures off the virtual machine's heap.

Each of these techniques could be useful. None of them, on its own, solved the problem yet.

What is more, the system covered workloads with different characteristics. Processing large data sets, repeatedly running full computation passes and operations that need a short response time do not have identical requirements. Optimising one kind of work does not have to improve the others.

After analysing the possible directions, I decided to split off a separate engineering track.

It was not a project to rewrite the engine on a new architecture. The first task was to define a data representation contract, the limits of optimisation and a way of checking whether further changes were justified at all.

Other parts of the system were being developed in parallel, so it also mattered to share these findings with the other work without prematurely imposing a new implementation on it.

Agreeing to explore a technical direction is not yet a decision to adopt it.

One of the directions under consideration was changing the data representation from an object layout to a columnar one.

The mechanism rested on a simple assumption: keeping values in contiguous arrays can lower the cost of data access, reduce allocations and make better use of the processor.

The question remained what this change would mean for the engine as a whole.

In an isolated computation loop, data access can account for a significant share of the cost. In the wider process there are also intermediate calculations, updates to the algorithm's state and other operations that a change of representation alone does not speed up.

The aim of the experiment was therefore not merely to show that the new data layout could be faster.

It had to establish whether using it in the existing engine would bring a sufficient benefit without changing the semantics of the computation or its results.

That required separate checks of correctness and of performance. A positive result in one of them could not stand in for the other.

Before the migration's cost-benefit measurement, two numerical criteria were written into the repository:

• a throughput improvement of at least 15%;
• a reduction in memory allocation of at least 50%.

In the historical contract these values were marked proposed. The preserved document is not, on its own, evidence that the owner formally approved them.

Their chronology, however, can be established.

The criteria entered the versioned contract before the deciding measurement. The gap was just under two hours.

The later reconstruction also checked whether the thresholds had been changed once the result was known. Their text stayed byte-for-byte identical over the following months.

This matters because a migration prepared at the cost of engineering work creates a natural temptation to find a new justification for adopting it after an unfavourable measurement.

The threshold can be lowered. A more favourable benchmark can be chosen. Or one can decide that, since the implementation is correct and partly ready, it would be a waste not to use it.

In this case, the result could be compared with criteria written down before the measurement.

The historical wording of the condition was not entirely unambiguous about the logical relation between the two values. That does not affect the assessment of the case: the tested variant met neither threshold.

The change was first tested on a simplified, stateless kernel.

The result was clearly positive:

These are two different workloads, with separate baselines. Putting them side by side does not mean that the throughput of the same program fell from 65.5% to 1.8%.

What it does show is that the effect of the optimisation depended heavily on what the program was actually doing.

The pilot document itself stated that a measurement on a stateless kernel does not settle whether migrating the full computation path is worthwhile.

The second benchmark covered a processing path in which successive operations depended on maintained state, and data access was only part of the total cost of the computation. Both variants ran the same logic on the same input data, with only the way the data was made available changed. It was a measurement closer to the target use than the simplified pilot, although it did not cover every possible workload of the system.

For this workload the raw allocation record was also preserved:

Allocation means the number of bytes allocated during one operation, not the amount of memory the process occupies.

The measurement was taken on one machine, with one virtual machine run per variant, five warm-up iterations and eight measurement iterations.

The throughput error intervals overlapped, so the +1.80% result on its own is not strong evidence of a lasting improvement.

There is no need, however, to settle the exact size of this small gain to see that the benefit required to justify the migration was not demonstrated.

What about correctness?

The change of data representation was not meant to change the processing logic or the results of the computation.

The preserved documents report positive results of named equivalence tests for both variants. They covered agreement of the algorithm's sequence of decisions and identity of the final record of results.

Within the verified scope, however, no raw output of those test runs was preserved. One can therefore speak of reported functional equivalence, but not of its independent reconfirmation while this article was being prepared.

Even if both implementations are accepted as equivalent, a second question remains: what does the system gain from the migration?

On the workload studied, the answer was not enough.

The +65.5% result was not wrong just because a later measurement showed +1.80%.

The pilot answered a different question.

It showed that in a specific part of the computation the new data representation can run much faster.

It did not prove, however, that the cost of that part dominates the full processing path.

That is the essential difference between measuring a technique's potential and measuring the value of applying it.

If the optimised part accounts for only some of the total cost, even a large improvement in its performance can translate into a small gain for the process as a whole.

This does not automatically mean we know the exact cause of the difference between the results. Explaining it would require a separate breakdown of the cost of individual operations in the representative workload.

The experiment did not have to settle each of these causes, though, to answer the question about the migration.

The pilot showed the technique's potential. The proper measurement did not show sufficient value in applying it.

After the proper measurement, a negative recommendation was recorded in the documentation: the cost-benefit criteria had not been met, so the migration should not go ahead.

The columnar variant was tested on a representative stateful workload, but it was not adopted as a replacement for the existing implementation.

Further performance work was based on the existing data representation. It was a separate scope of optimisation, and its later results should not be credited to the rejected migration.

The documentation makes it possible to establish the result of the experiment, the content of the recommendation and the actual later state of the project. Within the verified scope, no separate record of my final approval of the recommendation was preserved.

That distinction does not change the technical result.

Not adopting the solution did not invalidate the work done to test it. It provided information about the limits of the mechanism's usefulness under specific conditions.

The most important result is simple: the new variant did not replace the existing solution on the strength of reported correctness and an impressive pilot result alone.

The work on the columnar layout showed what it can do within a limited scope. It did not provide a sufficient argument for changing the existing engine.

That, too, is a useful result of engineering work.

At the start it was not certain whether a change of data representation would justify the cost and risk of a rebuild.

That is why the work on the contract, the experiment on a limited part and the measurement on a representative workload were kept separate.

A positive pilot result did not end the verification.

The proper benchmark made it possible to confront the promising variant with the conditions that were supposed to justify using it.

The experiment answered three different questions:

• Does the solution have technical potential? Yes, in the pilot studied.
• Does it preserve functional equivalence? The test documents report that it does, within the limits of the available material.
• Was a sufficient improvement for the target migration demonstrated? No, on the workload studied.

It did not thereby show that the columnar representation is a bad solution or that it would bring no benefit in another use.

It showed something narrower, but sufficient for the change under consideration: on this computation path, against the existing implementation, it did not reach the recorded cost-benefit threshold.

This is the point where a technical result meets an architectural decision.

It is not enough to know that something can be done faster. One also has to know whether we are speeding up the right part of the system, what benefits follow from it and whether they justify the added complexity.

A correctly executed change can at the same time be a change that is not worth adopting.

This is a private research experiment, with no users and no production deployment of the migration described.

The results come from one environment and a limited number of runs. They do not justify general claims that one way of representing data is superior to another.

The throughput measurement on the full computation path had overlapping error intervals. The documents reporting functional equivalence name specific tests, but the reconstruction did not recover their runs from raw records.

The criteria were written down before the measurement, but they had the status of a proposal. The repository history makes it possible to check that they existed beforehand and stayed unchanged afterwards, but it is not a complete record of authorisations.

The preserved consultations document the context of the problem and some of the decisions on how the work was organised. They do not cover the whole history of the implementation or all later decisions.

The technical sources remain private, so the publication does not allow a full, independent reproduction of the historical experiment.

The context of the project and the decision to split off the engineering work rest on conversations preserved from the time the project was running. The technical course of the experiment was reconstructed from the documentation, the change history and the recorded measurement results.

I led the work personally and actively supervised the whole process, from start to finish. Not all supervisory actions and decisions were recorded in the repository as separate approvals.

I used AI agents for design, implementation, measurement and documentation tasks.

In preparing the article, the preserved materials and results were analysed. The code under study and the benchmarks were not rerun. Implementation details, the full records of the consultations, the research data and the internal evidence materials remain non-public.

As the author, I am responsible for the choice of topic, the scope of disclosure and the published interpretation.

What interests me most here is a result that did not allow a technically promising solution to be adopted.

The pilot showed a significant improvement. A fuller measurement did not confirm it to the required extent.

There was no need to change the criterion or look for a substitute justification for the migration.

A good experiment does not have to confirm that a change makes sense. It can also provide sufficient reasons not to make it.