The question we usually get on this topic is “Is Databricks better than DataStage?” That's the wrong question. Both platforms process data reliably, and both are actively developed. The decision is made somewhere else entirely: in the billing model, in the real state of your job inventory, and in who is going to operate the platform afterwards.

Migration makes sense if your processing load is highly uneven and your team already works in SQL and Python. It doesn't if you run a stable, evenly loaded estate and your ETL knowledge sits with a couple of DataStage specialists.

Before you evaluate a migration, it helps to know that both platforms have moved recently.

IBM has not sunset DataStage — quite the opposite. It remains available in three deployment models: traditional on-premises, inside Cloud Pak for Data, and now as a fully managed service with a cloud-based design surface and a separate execution plane, so data can stay where it already lives. If you believed you were being forced off the product, you aren't.

Databricks, meanwhile, has consolidated its data engineering portfolio under the Lakeflow name: Lakeflow Connect for ingestion, Lakeflow Jobs for orchestration, Lakeflow Designer for preparation.

A proposal, article or internal document still using the previous product names is out of date — a useful freshness check for any consultancy pitch that lands on your desk.

This is the question that determines the effort, and it is almost always asked too late.

In a DataStage estate that grew over years, a substantial share of jobs is effectively dead: they feed reports nobody opens, populate tables no application reads, or duplicate logic from a project abandoned long ago. Migrating that estate without auditing it means paying to translate logic that no longer does anything.

So before you request a single quote, establish three things:

• Run history: which jobs actually ran in the last six months — and how often?
• Consumers: which target tables are genuinely read by a report or an application?
• Duplicates: which jobs carry identical or near-identical logic?

The most important difference between the platforms is commercial, not technical.

Databricks bills on consumption in DBUs, at per-second granularity, with better rates available through committed use. Classic ETL licensing works on capacity instead: you pay for a defined configuration whether or not you use it fully.

Which model is cheaper depends entirely on your load profile. If your processing happens in peaks — month-end close, seasonal trade, campaign analysis — and the infrastructure idles in between, a capacity model has you paying permanently for peak demand. Switching produces a real saving.

If your processing runs evenly, the effect reverses: at a constant base load, consumption billing is rarely cheaper, and it is harder to budget. For a mid-sized company on a fixed IT budget, a variable monthly invoice is a genuine drawback that tends to get overlooked in the enthusiasm for elasticity.

If you go this route, put budget alerts and per-department cost attribution in place from day one — otherwise you'll meet the cost side for the first time when an uncomfortable invoice arrives.

This is where mid-market migrations fail more often than they fail on technology.

DataStage is developed graphically. Databricks is fundamentally driven by SQL and Python, even though low-code preparation surfaces now exist. For a team that has modelled graphically for years, that isn't a change of tool — it's a change of job.

For an organisation with two or three people on the data side, this means budgeting either for real upskilling — not a two-day seminar, but supported project work over months — or for an external operations partner. Put that line in the migration budget. A platform only an external vendor understands after go-live hasn't reduced your dependency, it has relocated it.

There are situations where we advise against migrating — even though that argues against our own project business:

• Stable estate, even load, working operations. A migration ties up your best people for months. That budget is almost always better spent on data quality or with the business units.
• ETL knowledge sits with one or two people. The issue then isn't the platform, it's your staffing risk. You solve that with documentation and a second person, not with new technology.
• Regulatory questions are unresolved. In regulated industries, outsourcing to cloud services carries supervisory requirements. That assessment belongs at the start of the project, not at the end.
• The actual pain is elsewhere. Very often the problem isn't the ETL tool but absent data ownership, inconsistent master data, or reports nobody trusts. No platform fixes any of that.

If the three questions point towards migration, a phased approach beats a cut-over date:

1. Inventory, decommissioning every job that is no longer needed.
2. One bounded business domain as the pilot: genuinely valuable, but not business-critical.
3. Parallel operation with reconciliation — old and new must produce the same numbers, or you lose the business units' trust.
4. Incremental adoption of further domains, each with a documented close-out.
5. Decommissioning the legacy estate only once the business actually trusts the new platform.

Parallel operation costs double for a while. That is the price of being able to stop the project at any point — and for a mid-sized company, that fallback is worth more than a few months of saved elapsed time.

Migrating from DataStage to Databricks is a business decision, not a technical one. Test your load profile against the billing model, clean up your job inventory before you request any quote, and budget your team's upskilling as a full project line item. If your current environment runs reliably and your load is steady, staying is the economically correct call — even if it's the less exciting one.

• IBM DataStage – product overview and deployment options
• Databricks – Data Engineering and Lakeflow
• Databricks – pricing model and DBUs