Senior Data Engineer

Join the 2,800+ professionals of Minnesota IT Services (MNIT) who connect Minnesotans to services that will improve their lives. This position serves our partners at the Department of Employment and Economic Development (DEED). The MNIT DEED team provides technical support to help improve employment conditions in Minnesota.

The team supports work that improves workforce and community development, increases business recruitment, and helps Minnesota small businesses to expand into international trade.

Job Summary

This position is temporary and includes full state fringe benefits; it is anticipated to last up to two (2) years.

This position exists to design, build, maintain, and support the Minnesota Department of Employment and Economic Development (DEED) data engineering and MLOps infrastructure. The role focuses on enabling scalable analytics and AI/ML workloads across Databricks and AWS services such as SageMaker, Bedrock, and Glue.

The ideal candidate will have hands-on experience delivering production-grade data pipelines and machine learning operations, and will ensure that DEED’s Databricks workspace, data lake environments, and cloud-based AI services operate securely, efficiently, and reliably.

Key responsibilities include administering Databricks workspaces, developing and maintaining notebooks and production pipelines, and serving as a technical advisor to application and data science teams implementing MLOps patterns.

The position also supports the ongoing administration of the enterprise data lake, contributes to data engineering best practices, and drives continuous improvement through evaluation and adoption of modern data and AI technologies.

This role requires applying project management and problem-solving skills to resolve technical issues, lead improvements, and ensure 24/7 availability and performance of production data engineering and MLOps services. Occasional work outside normal business hours may be required.

Primary Duties

• Databricks administration and operational support.
• Development and enhancement of Databricks notebooks, jobs, and pipelines.
• MLOps engineering support for teams using SageMaker, Bedrock, Glue, and other AWS services.
• Data lake management and data engineering for DEED programs.
• Evaluation and recommendation of new data engineering and AI technologies.

This position requires an employee to work onsite at 600 Robert Street N, St. Paul, Minnesota. at least 50% [or more or less if space limitations remain] of the time, with limited telework available. Telework eligibility depends on residence:

• Candidates living in Minnesota or within 50 miles of the worksite in a bordering state (IA, ND, SD, WI) may telework up to 50%.
• Minnesota residents living more than 50 miles from the worksite may be eligible to telework more than 50%.
• Candidates living 50 miles or more from the worksite in a bordering state are not eligible for hire unless relocating within 50 miles.

Qualifications

This position requires a minimum of four (4) years of experience in performing IT business analysis functions.

• High-level technical knowledge of Databricks, Delta Lake, Spark, Unity Catalog, and data lakehouse architecture.
• Demonstrated ability to administer, develop, optimize, and maintain production data lake house environments.
• Designing, maintaining, and troubleshooting complex data pipelines and workflows.
• Demonstrated ability to design scalable ETL/ELT pipelines using Python, SQL, Delta Live Tables, and Medallion architecture.
• Demonstrated ability to provide technical leadership and collaborate with cross-functional teams, including developers, business analysts, data analysts, and business stakeholders to support enterprise applications and services.
• Demonstrated ability to communicate clearly and effectively in a manner that facilitates mutual understanding.
• Customer-focused approach grounded in active listening, empathy, and solution-oriented problem-solving to deliver positive user experiences.

Education in Information Technology (IT) or an IT related field may substitute for experience as follows: a master’s degree for eighteen (18) months of experience; a bachelor’s degree for one (1) year; and an associate’s degree for six (6) months.

Strongly Preferred Qualifications

• Experience with CI/CD pipelines and DevOps practices, including GitHub Actions or similar automation platforms for infrastructure and application deployments.
• Ability to implement and maintain infrastructure-as-code using Terraform, GitHub Actions, YAML.
• Familiarity with AWS services related to data engineering (S3, RDS/Postgres, IAM, Glue, networking).
• Experience designing, implementing, and supporting enterprise data lake architectures for analytics, reporting, or data science initiatives.

Minnesota IT Services does not participate in the federal E-Verify program. Minnesota IT Services is unable to sponsor applicants for work visas, including F-1 STEM OPT extensions, and cannot complete Form I-9 or any other documentation requiring employer participation in E-Verify. All applicants must be legally authorized to work in the United States at the time of application.