Opening Pandora’s Box: Conquer the 7 “Bringers of Evil” in Data Cloud Migration and Greenfield…

Pandora, the first mortal woman, was created by the gods as part of Zeus‘s plan to punish humanity for Prometheus‘s theft of fire [1].

She was gifted with beauty and intelligence, and Zeus sent her to Epimetheus, Prometheus’s brother. For the wedding gift, Zeus gave Pandora a jar (often interpreted as a "box") and warned her never to open it [1].

Despite warnings, Pandora was curious. She opened the jar, releasing the world’s bringers of evils—leaving only hope trapped inside [2].

Since then, "to open a Pandora’s Box" has been synonymous with doing or starting something that will cause many unforeseen problems [3].

Comparing this to my professional life, the only occasion I felt like I had opened "Pandora’s Box" was when I began working on a data cloud migration/greenfield project several years ago.

And the funny thing is that this thought hasn’t changed years later, even after working on two additional and almost identical projects.

Not only did I experience new "bringers of evil" with every new data cloud migration project, but I also felt I had managed to release "hope" from the box.

Now, with (a bit) more wisdom, knowledge, and experience in a few data migration/greenfield projects, I will share the "7 bringers of evil" and how to overcome them.

Let’s dive in.

A Guide to Conquering Cloud Migration Challenges

I thought a lot about how to compare the mythical bringers of evil—envy, remorse, avarice, poverty, scorn, ignorance, and inconstancy [3]—to the real-world challenges I’ve seen or gone through in data cloud migration/greenfield projects.

And, as a result, I first created a one-sentence explanation of what specific bringer of evil can cause in the project.

Then, I provided the [hypothetical] project scenario for more context.

Lastly, I provided my input on best- and worst-case solutions, i.e., how to "conquer" a specific [hypothetical] scenario in case it happens in your project.

---

1. Envy

Comparing your migration project unfavourably to other projects, leading to poor project planning and unrealistic expectations.

Scenario:

You did your research, you talked to people, you called in consultants, and they all confirmed it: "The company XY, which is similar to your company, managed to migrate their whole data platform to the cloud in only 10 months."

By all means, this implies only one – you should be able to migrate in the same period, if not faster.

So, you start by creating a project plan, keeping in mind external inputs on the migration deadline. This leads to budget approval, which leads to the execution phase.

Then, suddenly, reality kicks in during the project execution phase.

• You start realizing that your on-prem infrastructure is more complex, and you have security restrictions that don’t allow you to connect to the source database(s). You understand then you can’t even use a 3rd party data integration tool to move your data to the cloud. You need to develop a new solution to overcome this problem.
• Then you realize that your legacy development depends on the data sources, which you are not allowed to move to the cloud without approval, and you don’t have this approval.
• Then you figure out that 15 to 20% of the data stored locally is not in use anymore from business, but you didn’t have time to do a proper analysis to design the clean cloud file storage or database landing layer, and you need to do it now.
• …..

And the problems just started piling up because the planning was rushed and biased by external inputs.

On top of that, project "what-if" scenarios were not developed beforehand to have a "best-case" vs. "worst-case" implementation plan to justify what is happening.

What follows is that you need to communicate this information to project sponsors and inform them of the increased project scope and budget.

Not a fun thing to do.

Solution(s):

["Best-case"] OR [What you can do to prevent this]

• Develop a customized migration plan: Create a migration plan that fits your organisation’s needs and infrastructure, rather than competing with external benchmarks.
• Think about all pillars during planning: During the planning, consider technical, regulatory, and cost pillars, and plan contingency money (plus "what-if" scenarios) for project implementation.
• Plan for Contingencies: From developed "what-if" scenarios, think and prepare how you will be able to resolve the "worst-case" ones if they happen.

["Worst-case"] OR [What you can do for damage control]

• Take responsibility and be transparent: Ask for additional resources (both human and monetary ones), but this time be transparent in presenting the maximum project over-budget scenario. Outline the valid arguments/blockers you have faced, and take responsibility for what happened.
• Highlight the long-term benefits: Emphasize to project sponsors that resolving these issues during the migration phase will lead to long-term cost savings on the data platform.
• Focus on quick wins: If feasible, ensure that you speed up your developments in components where you don’t have blockers, so you can show the progress and maintain a "positive image" in front of the stakeholders.

---

2. Remorse

Regretting the selection of the new technologies and development principles, leading to delays in migration and impacting the solution design of the new data components.

Scenario:

You started your cloud migration, designed the architecture, selected the cloud services, and finally – the development began.

—What could go wrong, ha?

• Then you realize that some of the newly selected cloud services may be missing part of the features that you were so used to/are critical in your legacy ones, and again – you need to find a solution or even additional service to cover the core purpose of these features.
• Then you understand that your legacy development standards should be adapted because you changed your data management system and didn’t develop the new development standards. Pressured by deadlines, you "freelance" in the development, creating a new, even bigger mess.
• With all the new pitfalls, you dwell on your destiny and regret not starting the migration project 5 years ago, when your dataset was smaller and business requirements manageable.
• …

Finally, in Cher‘s words, you sing (quietly) the title of the song "If I Could Turn Back Time." On a loop.

Solution(s):

["Best-case"] OR [What you can do to prevent this]

• Conduct feature analysis beforehand: Before selecting cloud services, do a feature comparison with the legacy services. Involve technical teams early in the evaluation and run proofs-of-concept (PoC) before the project starts.

• Create new development standards early: Think in advance about how cloud development principles differ from on-prem systems. Develop new standards, by creating a mapping table with the legacy ones.

• Again: contingency planning.

["Worst-case"] OR [What you can do for damage control]

• Reevaluate technology choices quickly: When you realize some cloud services don’t fit your needs, temporarily pause the affected data migration component and re-plan the development while focusing on resolving this issue.
• Temporarily assign extra or external resources to resolve blockers: To resolve missed feature gaps or lack of standards, assign additional resources (preferably consultants) to clean up these specific problems.
• Again: take responsibility, communicate issues/needs transparently, and highlight the long-term benefits.

---

3. Avarice

Overemphasizing cost savings and future time-to-market value at the expense of critical components, like data quality, security, or performance, leading to higher costs and poor data platform in the end.

Scenario:

You landed yourself a new job.

You were hired to develop the new data platform where architecture is already defined, services selected, and the first data integration pipelines are functional.

Your main task is to **** create value from the data as fast as possible, design the semantic layer from scratch, and engineer new data products for key business colleagues.

—No big deal.

• You start by focusing on transformations and data modelling, and voila – the first data products are here, and you are the company star. Then the business comes to you and starts questioning the accuracy of the delivered insights. You return to your data, compare it to the source systems, and then realize they don’t match. Something went wrong in the data integration part, but you didn’t see this before because you didn’t implement quality checks.
• In addition, you were so eager to deliver the data products faster that you didn’t even think about infrastructure nor tried to put in place multiple environments. Instead, you did your development in production. Yes, production.
• As a cherry on top, you started designing machine learning data products as fast as possible, realizing only later that their development and run costs are high with a low return on investment (ROI).
• ….

And here you are now.

You promised that your cloud costs would be lower than the on-prem ones, with the motto: "The storage is cheap in the cloud, and I don’t need SRE colleagues or a variety of data roles to develop a functional data platform."

Solution(s):

["Best-case"] OR [What you can do to prevent this]

• Data quality as a mission: Plan time and staff human resources to set up quality checks at every stage of the data pipeline design—from the data integration to the final products that contain insights.
• Design a stable and secure infrastructure: Plan time and staff human resources to set up multiple secure environments (dev, test, prod), even under tight deadlines. This allows you to develop quality and reliable data products.
• Assess the costs of the data products: Before committing to deliver new data products, perform a cost-benefit analysis to assess their (potential) ROI. Avoid starting expensive developments without an understanding of the financial impact.
• Again: contingency.

["Worst-case"] OR [What you can do for damage control]

• Shut down the high-cost, low-value data products quickly, and manage expectations: Implement the cost monitoring framework and shut down the data products with low ROI. Show the numbers to the business transparently, and propose scaling back when business needs are more mature.
• Again: take responsibility, communicate issues/needs transparently, and highlight the long-term benefits.

---

4. Poverty

Cutting down the budget and resources allocated to the migration project and not getting granted "feel good" benefits, leading to high stress and low morale.

Scenario:

You received your project budget, and implementation is going as planned.

However, the project sponsors expect you to keep the development costs under the assigned budget, while speeding up the delivery.

You got the assignment to build an additional, remote team in a more budget-friendly location.

• Although this was not originally planned, you acknowledge their input and start the hiring process. However, the process of staffing takes longer than expected. You initially invest 3 months in staffing the new resources, then another 3 months in repeated staffing to replace candidates who declined offers. But finally, the staffing is done, and the new team members are slowly joining the project.
• You start making organizational and backlog changes to consolidate the project development that is now split between two locations.
• In the meantime, your small local cloud team is investing daily effort to share the knowledge and onboard the new (remote) colleagues with the current project development.
• The collaboration is going well, except for minor delays in communication/providing feedback due to different time zones. Because of this, you would like to reward your local team members, who cover 2 to 3 roles simultaneously and work long hours to ensure the project gets successfully delivered.
• However, this is not feasible. You can’t get this extra money for travel to a remote location, team events, etc.
• The dissatisfaction only piles up when you realize that you and the team are not allowed to visit your remote team, with whom you work daily, but the business colleagues – are.
• …

Solution(s):

["Best-case"] OR [What you can do to prevent this]:

• Ensure a higher budget from the start: During the project planning phase, ask for a budget cut that covers project team-building events and travel. Try to argue that "feel-good" benefits are not only perks but are important for maintaining team morale.
• Plan for unforeseen changes: Even if you don’t expect to get an additional team in your project, you can adapt remote-friendly processes. Create project documentation (space), and adopt Scrum/Kanban to track your development. In addition, create communication channels/groups and organize virtual coffee sessions/open discussion weekly meetings. These will help with every new onboarding, regardless of the new team member’s location.

["Worst-case"] OR [What you can do for damage control]:

• Request project onboarding time: When building a new remote team, communicate to management the additional time and resources required for onboarding. Get the "buffer time" in the project to ensure proper knowledge transfer and collaboration from the start of the cooperation. Ensure that the local team doesn’t constantly work long hours. The team’s well-being should be a priority, even if this means prolonging project delivery.
• Focus on non-monetary recognition after budget cuts: Recognize the team’s hard work in a non-financial way. This includes flexible working hours/locations, skipping admin meetings, giving public acknowledgement, or even small tokens of appreciation like chocolate.
• Negotiate for minimal travel: If the travel between project locations isn’t feasible, negotiate for in-real-life meetings only to celebrate the project’s main milestones.

---

5. Scorn

Facing internal resistance from colleagues who feel their legacy systems are being dismissed, leading to passive and active obstruction of the migration project.

Scenario:

You know it, business knows it, everyone knows it – the cloud data platform is the fastest way to meet rising business requirements and, if developed properly, stop the rising on-prem maintenance costs.

• Yet, despite the obvious need for the migration, you begin hearing the same concerns from various colleagues: "Our legacy system is completely functional; why do you want to replace it?" – OR – "This cloud migration is just a trend, and you will fail in it." – OR – "The cloud is not secure enough, and we need to keep data in our on-prem platform." You only hear reasons why this can’t and won’t work.
• On top of this, the resistance comes in actions too. The colleagues are not sharing inputs in the project planning phase. They show a lack of interest in providing support in the development of proof of concepts. Consequently, project approvals get delayed.
• Then you manage to resolve the blockers in the project planning/preparation phase, staff the cloud-skilled colleagues, and the project takes off. However, the resistance is higher than ever. The same colleagues who blocked the project now feel excluded and share constant and negative feedback on the new development and colleagues. All this blocks the normal development of the project, as your focus needs to be on conflict resolution instead of delivery.
• …

You seek advice from your business coach, and he tells you the following sentence: "Welcome to big-corp business; now you need to learn how to deal with it." (Side input: it was only a starting sentence said as a joke.)

Solution(s):

["Best-case"] OR [What you can do to prevent this]:

• Include middle/high management to share the vision: Before the cloud migration starts, seek help from management and project sponsors to share the positive vision of the change and the strategic direction of why platform modernization is important for the business.
• Organize workshops: Seek consultancy support in organizing workshops for everyone to get insights into the new technology and present customer success stories. This, too, should result in creating awareness of the cloud platform’s advantages and benefits.
• Create a transition plan: To show the benefits of the cloud platform, create a plan for comparing data products on both platforms. In other words, during the PoC phase, compare the metrics of the same data product developed on-prem vs. cloud. Focus on improvements in performance, costs, and development steps, showing that the legacy work will not be disregarded but evolved.

["Worst-case"] OR [What you can do for damage control]:

• Address resistance head-on: If you encounter individuals who actively resist and block migration, try to address this behaviour in direct conversation. Address the possible issues of their concerns by providing positive aspects of the new development. Then observe if their behaviour will change after.
• Escalate if necessary: If resistance persists and affects the project delivery, escalate the issue to middle/higher management. Show them the impact of this behaviour, and if necessary, ask them to help out in distancing some individuals from the project. You will need support from leadership to help push through roadblocks caused by internal resistance.

---

6. Ignorance

Lack of expertise in cloud technologies and migration best practices, increasing the risk of failure and delays.

Scenario:

You were assigned a new data-stack modernization project from management, and the expectation is to deliver the first PoCs in the next quarter.

You have never worked on anything similar and don’t know where to start.

Your colleagues and you start comparing the existing cloud platforms and services, and you select "the one" you will work with.

• However, no one in the existing team is familiar with the new technologies or data migration best practices related to them.
• So, you staff one or two new colleagues, or "fresh blood," who are enthusiastic techies but haven’t had experience in similar projects before.
• They engage with new technologies, catch up very quickly, and manage to showcase new PoCs. And all of this is done before you get the official project approval and budget.
• The enthusiasm on your and the team’s side gets higher, and you think that everything from now on will be a smooth sail.
• Then the project officially starts, and you get a tangible budget. All happy, you again start staffing new colleagues. But this time, you pick the people who have worked on similar projects.
• And they all bring new ideas and migration concepts from their previous roles. This results in discarding the initial concepts and PoCs you had. And again, you are starting the development from scratch.
• As the initial effort of 3–4 months is gone, from the start of the project, you find yourself behind schedule.
• This causes pressure on your team and you, and some colleagues even leave the project due to this.
• …..

Solution(s):

["Best-case"] OR [What you can do to prevent this]

• Conduct pre-project skills gap analysis: Before starting the cloud migration, identify the skills you need for your cloud project. Then try to get a hiring budget before the project budget. Following this, ensure that the same people who work on PoCs will work on the cloud development.
• Commit to PoC development: Instead of rushing the PoC development with a quick-win solution, invest additional time in the design of several solutions. Approach this problem strategically, and for one data pipeline, try to develop and test two PoCs. On top of the "theoretical" one (read: best practices), a hands-on approach in selecting the optimal solution will bring your team confidence in the development phase.

["Worst-case"] OR [What you can do for damage control]

• Reassess the staffing and get expert consultants: If the mixed ideas on the development solutions start causing project delays, bring in external consultants who can help with their expertise. These people can assist in improving the team’s learning curve and ensure the project stays on track.
• Go with the suboptimal solution: Pick the suboptimal solution for part of your development if you have an existing skillset in the team. Existing hands-on experience can speed up the project’s delivery. For example, pick the programming language that everyone is already familiar with.
• Again: take responsibility, communicate issues/needs transparently, and highlight the long-term benefits.

---

7. Inconstancy

Changing project scope and priorities, causing confusion and disrupting planned project delivery.

Scenario:

• You: So, there are new requirements for this project component?
• Your colleague: Yes, and some additional ones are still being discussed.
• You: Really?
• Your colleague: Yes, really.
• You: But you know the project deadline is in 5 months? And what’s with that – is it still the same?
• Your colleague: Mhm.
• ……………………………………………
• Here we go again. For the fourth time since the project started, the business requirements have changed.
• Twice in the current release.
• Some work that you already delivered is now out of the project scope. However, the new one has been included in the scope. And you guessed it – with no extra time for this development.
• Similar to the previous iterations, you return this information to your team, which only confuses them more.
• And again – you know it, they know it, everyone knows it – this time, the changed project scope will break the deadline.
• ….

Solution(s):

["Best-case"] OR [What you can do to prevent this]

• Develop a change management process: Establish rules and guidelines for handling unplanned requirements before the project starts. Ensure that this process includes an impact analysis on the timelines and obtaining approval for the requested changes from the project sponsors.
• Introduce "hard deadlines": Lock down the changes in the project scope early enough and define the "hard deadlines" to get fixed requirements. In addition, track your project backlog delivery carefully and communicate the pending critical work to the requesters in case of new requirements.
• Plan project meetings: Organize weekly meetings with project managers, business requirements engineers, technical analysts, and the delivery team. Ensure that everyone is on the same page when it comes to understanding the pending backlog and the timelines.

["Worst-case"] OR [What you can do for damage control]

• Escalate fast: Escalate the changes in the project scope to your project sponsors. Create awareness of how these changes affect the deadline and the project outcome.
• Re-prioritize even faster: If the new requirements have an impact on critical components, prioritize this development by re-planning lower-priority tasks for later project phases.
• Again: take responsibility, communicate issues/needs transparently, and highlight the long-term benefits.

---

See no evil 🙈 , hear no evil 🙉 , speak no evil 🙊

Although cloud migration or greenfield projects can often feel like opening Pandora’s Box, it doesn’t mean they are not awarding to work on.

Despite all the challenges I experienced and saw in these projects, the learnings I got from them resulted in my personal and professional growth. Bigger and more positive than I was ever able to imagine.

I have learnt that good preparation, contingency planning, taking responsibility for mistakes, staffing skilled people, maintaining transparent communication towards ** sponsors, and consistently highlighting long-term benefit**s always lead to positive project outcomes.

In summary, it’s important to stay proactive in every challenging situation and find a solution for it.

Hence, in this blog post, I’ve shared my solutions in the hope you can reuse them to handle the challenges in your cloud migration/greenfield projects.

Until next time, happy ☁️ data migration planning & development!

---

Thank you for reading my post. Stay connected for more stories on Medium, Substack ✍️ and LinkedIn 🖇 ️.

---

References

[1] Theoi Greek Mythology: Pandora, https://www.theoi.com/Heroine/Pandora.html, accessed on September 25, 2024

[2] Britannica: Pandora, __ https://www.britannica.com/topic/Pandora-Greek-mythology, accessed on September 25, 2024

[3] Wikipedia: Pandora’s box, https://en.wikipedia.org/wiki/Pandora%27s_box, accessed on September 25, 2024