[Remote] Databricks Developer – Medicare
Auto ImportNote: The job is a remote job and is open to candidates in USA. BigRio is a leading Medicare Advantage health plan, seeking an experienced Databricks Developer to build and operate data solutions on the Databricks lakehouse platform. The role focuses on managing data pipelines, optimizing data processing, and ensuring data governance within the healthcare payer domain.
Responsibilities
- Build, deploy, and maintain data pipelines natively within Databricks, using notebooks, Delta Live Tables (DLT), and Databricks Workflows/Jobs for orchestration
- Develop and optimize Spark (PySpark/Scala/SQL) logic for processing large-scale Medicare Advantage claims (EDI 837/835), eligibility, and provider data within the Databricks environment
- Manage and evolve the Delta Lake medallion architecture (bronze/silver/gold) to ensure clean, reliable, well-governed data
- Configure and maintain Unity Catalog for data governance, lineage, and access control across PHI/HIPAA-regulated datasets
- Tune cluster configurations, job performance, and cost efficiency (autoscaling, cluster policies, Photon, job/task-level optimization) within Databricks
- Use Databricks SQL and SQL warehouses to support ad hoc analysis, validation, and reconciliation of Medicare Advantage claims and eligibility data
- Monitor pipeline health, troubleshoot job failures, and resolve performance issues directly within the Databricks platform (Jobs UI, cluster logs, DLT event logs)
- Manage code and deployments via Databricks Repos and CI/CD integration (Databricks Asset Bundles, Azure DevOps/Git)
- Apply PHI/HIPAA-compliant data handling and role-based access practices throughout all Databricks workspace and pipeline configurations
- Document Databricks pipeline design, workspace structure, and job configurations
Skills
- 5+ years of hands-on Databricks experience — this is a Databricks-first role; deep platform fluency is the primary requirement
- Strong command of Spark (PySpark/Scala) and Databricks SQL for pipeline development
- Hands-on experience building and managing Delta Live Tables (DLT) pipelines
- Practical experience with Unity Catalog for data governance, access control, and lineage
- Experience with Databricks Workflows/Jobs for orchestration and scheduling
- Demonstrated ability to tune cluster and job performance (autoscaling, cluster policies, Photon, cost optimization) within Databricks
- Experience with Databricks Repos and Git-based CI/CD for notebook/pipeline deployment
- 3–10+ years of healthcare payer domain experience, ideally with Medicare Advantage data (claims ingestion, eligibility logic, EDI 837/835, CMS-aligned reporting)
- Working knowledge of PHI/HIPAA-regulated data handling and role-based security within a data platform
- Comfortable working within an Azure or equivalent cloud environment (ADLS, workspace/networking basics) as the underlying infrastructure for Databricks
- Strong problem-solving skills and ability to work independently in a remote environment
- U.S.-based, remote-capable (U.S. resident, citizen, or Green Card holder)
- Databricks Certified Data Engineer (Associate or Professional)
- Experience with MLflow or Databricks' ML/AI tooling
- Exposure to Azure Data Factory, Power BI, or Snowflake in a supporting/downstream capacity
- Experience migrating legacy ETL worklo into Databricks
- Prior experience in a client-facing or consulting environment
- Experience with EHR-adjacent systems (e.g., Epic Clarity) or provider-side data in addition to payer data
Benefits
- Work Model: Remote (U.S.-based candidates only)
Company Overview