In 2022, the field of Artificial Intelligence (AI) is expected to face challenges such as increased fairness and bias issues in AI systems, a need for better ML observability, and a rising demand for citizen data scientists. However, there are also opportunities for growth, including the adoption of ML monitoring and observability solutions, an increase in specialized ML infrastructure tools, and a potential talent crunch that could drive innovation in educational and job opportunities. To navigate these changes effectively, it is crucial to focus on making the industry more fair, inclusive, diverse, and transparent.
The growing complexity of machine learning models has made it increasingly difficult to understand why a model makes certain predictions, especially as these predictions can have significant impacts on our lives. Explainability is a technique designed to determine which features led to a specific model decision. It does not explain how the model works but offers a rationale for human-understandable responses. This piece aims to highlight different explainability methods and demonstrate their incorporation into popular ML use cases.
Click-through rate (CTR) models are critical for digital advertisers, with the average CTR in Google AdWords across all industries being 3.17% on the search network and 0.45% on the display network. However, machine learning systems can be impacted by various factors such as contextual relevance, user attributes, time of day or seasonal fluctuations, and data quality issues. To address these challenges, it's essential to implement best practices for ML monitoring and observability with CTR models, including tracking key metrics like log loss, precision recall AUC, and time series of predictions versus actuals. By identifying root causes of performance degradation and adaptingively retraining models, teams can ensure their CTR models stay relevant and effective in the ever-changing digital advertising landscape. Effective data quality assurance is also crucial to prevent "garbage in, garbage out" scenarios, where inaccurate or missing data can significantly impact model performance.
The article discusses the importance of maintaining high-quality data for machine learning (ML) models and how modern MLOps solutions need to address both code and data aspects. It highlights that ensuring good data quality is a continuous process, requiring ongoing investment. The article delves into the key dimensions of data quality, which include accuracy, completeness, consistency, privacy and security, up-to-dateness, relevance, reliability, timeliness, usability, and validity. It further explores how these dimensions can be addressed for structured and unstructured data using ML observability and Data Operations platforms respectively. The article concludes by emphasizing the benefits of investing in data quality management for unlocking the potential of an organization's structured and unstructured data.
Can AI Help Make Social Media More Accessible, Inclusive and Safe?` ShareChat, a rapidly-growing social media unicorn valued at over $3 billion, is leveraging AI to build a more inclusive and diverse platform for its 160 million active monthly users in South Asia. As the Lead AI Scientist, Ramit Sawhney, aims to democratize AI and create a safer online space by tackling issues like bias, hate speech, and preventing abuse. The company's unique vantage point on ethics and fairness in AI informs its approach, prioritizing human-centered design and considering diverse perspectives. By leveraging AI for personalization, detecting abusive content, and monitoring models, ShareChat is creating a more accessible and inclusive social media platform that balances performance with safety and ethics.
Deb Liu, President & CEO of Ancestry, shares her vision for the company which includes expanding its reach to everyone around the world who cares about their family history and making it easier for customers to craft stories by finding their history and telling their story. She also discusses her upcoming book "Take Back Your Power: 10 New Rules for Women at Work" which aims to help women thrive in an imperfect world, build allies, and take back their power where they can. Liu emphasizes the importance of building teams and overcoming biases in order to promote diversity and women in tech. She highlights the need for companies to allow failure as a learning experience and iterate quickly to improve products and services.
Demand forecasting is a crucial aspect for businesses across various industries. The advent of AI and machine learning has automated this process and made predictions more sophisticated and precise. However, recent events have raised questions about the reliability of these models' performance. Model monitoring and observability are essential to alert teams when these events happen, quantify their impact on models, and provide insights into root causes for quick remediation. Common challenges faced by demand forecasting models include regression model susceptibility to drift, limited feature diversity, and the impact of outlier events like COVID-19. To ensure satisfactory performance, it is crucial to monitor various metrics such as mean error, mean absolute error, mean absolute percentage error, and mean squared error. Observability platforms can help teams visualize and root cause issues quickly, especially during outlier events.
Ilya Reznik, a Senior Machine Learning Engineer at Twitter Cortex, views the ML engineer role as both exciting and challenging due to increased competition in the field. He believes it's a good time to enter the industry but notes that getting an initial foothold is more difficult than before. Ilya sees the rise of MLops and data-centric approaches, which will enable subject matter expertise and allow for better tooling around model development and evaluation. He emphasizes the importance of monitoring and observability in machine learning, citing examples such as addressing issues like racial bias and handling sudden changes like the pandemic. Ilya's work experience spans physics, OSHA, Adobe Analytics Cloud, and his current role at Twitter Cortex, where he aims to contribute to scaling ML infrastructure and improving model development processes.
Feast and Arize AI have partnered to enhance the ML model lifecycle by empowering online/offline feature transformation and serving through Feast's feature store and detecting and resolving data inconsistencies through Arize's ML observability platform. The integration of a feature store and evaluation store can help improve productionization of features, mitigate data inconsistencies, and facilitate troubleshooting to resolve performance degradations in an end-to-end ML model lifecycle.
The recent CDAO Fall panel discussion highlighted key challenges and takeaways for organizations implementing AI and ML initiatives. Due to the COVID-19 pandemic, data science teams have had to navigate uncertainty, prioritize ML monitoring, and focus on model observability. The panel emphasized that buy-in for AI projects is tied to delivering agile insights and business value at scale. Business stakeholders' expectations are high, creating a challenge of quickly proving the real value of initiatives. Striking a balance between tools and technologies and measuring incremental value is critical, with ways to categorize ROI varying depending on whether models have a direct impact on revenue or efficiency. Ultimately, balancing short-term wins with long-term success helps sustain initiatives and deliver ROI.
Neptune AI and Arize AI have partnered to improve continuous monitoring and improvements for machine learning (ML) models. The collaboration aims to help ML teams maintain model performance once deployed in production environments, where data may shift in distribution or integrity due to unexpected dynamics or upstream changes. By connecting Arize's ML observability platform with Neptune's metadata store for MLOps, the partnership enables more effective monitoring of production models and informed retraining decisions. The integration allows users to track every model version and its history, improving experimentation and optimization processes in machine learning workflows.
There are significant financial losses due to global fraud, with the economy losing over $5 trillion annually. Building and deploying sophisticated machine learning (ML) models is crucial in detecting and preventing fraud, but these models can be fragile and require monitoring for anomalies. ML practitioners face challenges such as imbalanced datasets, misleading traditional evaluation metrics, limited sensitive features, and not all inferences weighted equally. To address these issues, important metrics to watch include recall, false negative rate, and false positive rate. Identifying the slices driving performance degradation is critical, and having an ML observability platform can help surface feature performance heatmaps to patch costly model exploits quickly. Additionally, monitoring and troubleshooting drift or distribution changes over time is essential in fraud models, as tactics are always evolving, and it's crucial to account for drift to ensure models stay relevant. By being proactive with monitoring and measuring drift, counter-abuse ML teams can get ahead of potential problems and focus energy on the most sophisticated threats.
Arize AI has been recognized by Gartner as a Cool Vendor in its Enterprise AI Operationalization and Engineering report. The platform helps address three key challenges inhibiting AI operationalization, including automatic detection of problems such as data quality or drift issues, faster root cause analysis and problem resolution of ML models, and continuous improvement of model performance, interpretability, and readiness. Arize is a machine learning observability platform that assists ML practitioners in taking models from research to production with ease. The recognition underscores the importance of ML observability as a critical category in ML infrastructure and positions Arize as a leading pioneer in this emerging space.
The role of machine learning (ML) engineers at Chick-fil-A involves building and scaling analytic capabilities to support business strategies, delivering continuous value for the company through innovative solutions. Korri Jones, senior lead ML engineer, emphasizes the importance of hiring tenacious thinkers with big hearts and a natural curiosity, as well as having a shared ownership approach across teams to achieve goals. The organization prioritizes bridging the gap between data scientists and data engineers by providing the right tools and technology to achieve scale and performance without losing velocity. Jones also stresses the need for leadership awareness, support, and understanding of ML initiatives to drive success, citing Chick-fil-A's leadership as a key factor in their growth and innovation. The ultimate goal is to deliver unique value that empowers owner/operators to make an impact in their communities and provide exceptional customer experiences.
Amber Roberts, an astrophysicist and self-taught data scientist, has joined Arize as a Machine Learning (ML) Sales Engineer. She was drawn to Arize due to its focus on using ML as a powerful force for change in businesses, the economy, and society. Amber's background in astrophysics and her passion for solving big problems made her an ideal candidate for this role. In her new position, she will help address pressing issues such as bias, responsible AI, ML observability, and explainability. As a sales engineer, Amber will be crucial in understanding the technology, business case, and varying levels of complexity based on the organization's ML infrastructure.
The last decade has seen a surge in interest in machine learning, with numerous researchers attempting to solve complex problems using state-of-the-art techniques. This renewed interest has led to an explosion of applications using machine learning to deliver novel experiences. However, as these applications move from research labs to production environments, new challenges have emerged that must be addressed in order for the ML systems to succeed. In order to measure and improve service-level performance, it is no longer sufficient to only monitor data quality or system performance over time; rather, overall service performance must also be evaluated.
The AI industry is growing rapidly, with businesses investing heavily in artificial intelligence to gain a competitive advantage. However, most organizations lack purpose-built systems to scale their MLOps and tools for model performance, leading to the need for observability solutions. Arize AI has raised $19 million in Series A funding from Battery Ventures, which will help strengthen its commitment to providing ML practitioners with deeper understanding of model performance across all stages of the model development cycle. The company's vision is to overcome current ethical deficits in AI systems by developing tools that monitor, troubleshoot, explain, and provide guardrails on AI, ultimately benefiting businesses and society. Arize's toolset allows teams to observe, manage, and improve machine learning models through a single pane of glass, connecting points across training, validation, and production, and providing automated monitoring of key model performance metrics.
Arize AI has been listed as a Representative Explainability Vendor in Gartner's 2021 Market Guide for AI Trust, Risk and Security Management (AI TRiSM). The company's Machine Learning Observability platform helps teams take models from research to production with ease. Arize's automated model monitoring and analytics platform is used by top enterprises to detect issues, troubleshoot problems, and improve overall model performance. Gartner views AI TriSM as a set of tools that ensure AI model governance, trustworthiness, fairness, reliability, efficacy, security, and data protection. Arize's recognition highlights the growing need for solutions that provide transparency and introspection into historically black box systems to ensure more effective and responsible AI deployment.
The text discusses the importance of model explainability in machine learning (ML) as models grow increasingly complex and impactful on various aspects of life. It highlights the need for understanding why a model makes certain predictions, especially when they power significant experiences.
The transparency problem in AI is a significant issue, with 51% of business executives reporting its importance and 41% suspending deployment due to potential ethical issues. Technical complexities contribute to black box AI, as the sheer volume of data fed into ML models makes their inner workings less comprehensible. Misconceptions regarding transparency include losing customer trust, believing self-regulation is sufficient, thinking that not using protected class data eliminates bias, and fearing disclosure of intellectual property. However, adopting responsible AI practices helps establish trust with customers, enabling predictable and consistent regulation, allowing access to protected class data for mitigating biases, and ensuring transparency doesn't mean disclosing intellectual property. ML observability tools can help organizations build more transparent AI systems by transforming black box models into glass box models that are more comprehensible to human beings.
Lyft relies on Machine Learning (ML) Engineers to bridge the gap between data scientists who develop models and teams that operationalize them. The company's ML infrastructure is used for various solutions, including mapping, fraud detection, pricing optimization, and ETA estimates. Alex Zamoshchin, an engineering manager at Lyft, explains how ML engineers help get models from research into the real world while ensuring they achieve business objectives. They are involved in framing ML problems within the business context, converting models into working pipelines, and analyzing experimental and observational data to ensure model quality and performance once deployed. In a hypothetical world without ML engineers, issues with models could arise before or after implementation, leading to potential failure in production environments.
Machine learning models are becoming increasingly important in emerging products and technologies, transforming the process of building ML models from an art to an engineering practice. Ensuring a reliable experience for users is crucial when deploying models into production environments, where issues can have significant impacts on revenue and customer satisfaction. As the industry matures, reliability engineering has become essential to prevent model drift or failure, which can cause sudden and significant problems, such as plummeting sales and customer complaints. The importance of reliability engineering in ML initiatives cannot be overstated, requiring a structured approach to ensure successful model deployment and maintenance.
Operationalizing AI Ethics has become imperative due to the challenges posed by machine learning models aiming for real-life mirroring and prediction. Despite reputational, regulatory, and legal risks, many companies still lack the ability to identify, evaluate, and mitigate ethical risks associated with their AI/ML products. Reid Blackman suggests that implementing systems identifying ethical risks throughout an organization is crucial. His seven steps to operationalizing ethical AI include leveraging existing infrastructure, creating tailored risk frameworks, optimizing guidance for product managers, building organizational awareness, incentivizing employee involvement in risk identification, and monitoring impacts while engaging stakeholders. An approach focusing on integrating the most appropriate ML infrastructure tools and processes is recommended by Blackman to make AI socially and ethically responsible.
The article discusses the importance of best-of-breed ML monitoring and observability solutions in managing machine learning models. It emphasizes that model failure can have significant impacts on a company's revenue, public relations, and user safety. To handle these challenges, an effective ML observability and model monitoring platform is required to understand what's happening inside the model as it runs. The article also highlights two general types of machine learning monitoring and observability solutions: best-of-suite and best-of-breed platforms. While best-of-suite systems attempt to cover end-to-end visibility, they may not have a feature set to cover every possible use case. On the other hand, best-of-breed ML platform solutions focus on providing highly specialized tooling for specific use cases and can be interoperated together to take advantage of their strengths while avoiding their weaknesses. The article concludes by stating that best-of-breed monitoring and observability solutions are ideal for companies that take their AI investment seriously, as they provide valuable insights into the performance of machine learning models without requiring significant time or financial investments.
Data quality is crucial for machine learning (ML) systems as they rely heavily on data to function effectively. Poor data quality can lead to inaccurate model predictions, impacting the overall performance of ML models. As companies increasingly adopt ML technologies, ensuring high-quality data sources has become more important than ever. This article highlights the significance of monitoring and maintaining data quality throughout the entire process, from training to deployment.
Dr. Rana el Kaliouby, a leading expert on technology and empathy, explores the role of emotion in today's technology-driven landscape. She was inspired by her early exposure to technology at a young age, which led her to realize that most communication is conveyed through non-verbal cues, but these signals are often lost when interacting with devices. This lack of emotional intelligence in technology has significant implications for how we interface with machines and each other. Dr. el Kaliouby's company, Affectiva, aims to humanize technology by designing software that can understand emotional and cognitive states through facial expressions. Their focus is on addressing big problems where their innovations can improve or even save lives, such as in automotive safety and the in-vehicle experience. However, there are risks associated with integrating emotional intelligence into computing, including bias and manipulation, which must be addressed through diverse teams, data representation, and transparency. Dr. el Kaliouby emphasizes the importance of building a full ecosystem of women and diverse leaders to improve representation and provide role models for young girls.
Artificial Intelligence (AI) and Machine Learning (ML) are complex fields that require a lot of knowledge to navigate. AI refers to machines carrying out tasks requiring human intelligence, while ML is an application of AI where machines learn from data and transform it into action. The roles of Data Scientists and ML Engineers are often confused, with the former focusing on research environments and algorithm definition, and the latter on deploying models in production and monitoring their performance. Monitoring is not enough; observability digs deeper to understand why issues emerge. Responsible AI is an ongoing process that requires constant attention and adaptation to ensure fairness and bias mitigation.
The article discusses the ethical issues surrounding artificial intelligence (AI) and machine learning (ML) technologies. It highlights the lack of diverse teams responsible for developing these systems, which can lead to algorithmic bias affecting marginalized populations. To address this issue, companies should embrace fairness, transparency, and accountability in their hiring and research processes. Building diverse representation within data and engineering teams is crucial for mitigating negative impacts on society. Investing in a diverse workforce and overcoming the ethical deficit are essential steps towards ensuring that AI and ML technologies serve all people fairly and without harm.
The machine learning (ML) industry has come a long way since its inception fifty years ago, with ML now being an integral part of society and helping various aspects of life such as driving cars, job searching, loan approvals, and medical treatments. The future trajectory of the industry is uncertain but there are emerging tools and capabilities that are becoming standards for nearly every ML initiative. Beyond these tools, the roles that shape data teams are rapidly evolving, particularly in the area of ML ops, which involves integrating development and operational aspects of ML infrastructure. This has led to the emergence of a new class of expertise - the ML engineer, who bridges the gap between data scientists and operations teams to ensure models perform well once they leave the lab. Companies need to invest in ML engineers to overcome challenges such as performance degradation issues with models that don't perform after code is shipped, requiring both tools for model observation and teams understanding how to make them perform.
Model drift refers to changes in the distribution of a model's input or output data over time, which can lead to decreased performance. It is crucial for ML practitioners to monitor and troubleshoot drift to ensure their models stay relevant, especially in businesses where data is constantly evolving. Drift can be categorized into feature drift (change in input distribution) and concept drift (change in output or actuals). To measure drift, one can compare the distributions of inputs, outputs, and actuals between training and production using various distribution distance measures such as Population Stability Index (PSI), Kullback-Leibler divergence (KL divergence), and Wasserstein's Distance. It is essential to relate these metrics to important business KPIs and set up thresholding alerts on the drift in distributions. When a model drifts, retraining it may be necessary, but careful consideration of how to sample new data and represent it in the model is required to avoid overfitting or under-representation. Adjusting the tradeoff between these two competing forces can help strike the right balance. If significant changes have occurred in the business, a simple retrain might not suffice, and the entire model may need revision. Troubleshooting drift involves identifying which input features or outcome variables have changed, understanding how their distributions have shifted, and potentially adjusting the model structure or feature engineering to adapt to new dynamics. Regularly reviewing model performance and maintaining open communication with end-users can help proactively address drift issues and improve models over time.
Arize AI has grown and welcomed new team members, including Andy Lu as Senior Designer, Harrison Chu as Senior Software Engineer, Krystal Kirkland as Product Marketing Manager, David Monical as Application Engineer, Eric Senzig as Enterprise Sales Lead, and Francisco Castillo as Data Scientist. These individuals bring diverse backgrounds and experiences to the company, with a focus on building a brighter future in the AI/ML space. Arize's strong stance on diversity, fairness, ethics, and inclusion has fueled Andy Lu's passion for the role, while Harrison Chu aims to solve operational challenges faced by ML-based teams. Krystal Kirkland is excited about contributing to a team that prioritizes technology and society, while David Monical was drawn to the company's potential to benefit from its MLOps capabilities. Eric Senzig appreciates Arize's unique mix of people, technology, and market timing, and Francisco Castillo joins with the goal of holding AI accountable through transparent and ethical model observation.
The growing importance of responsible AI was discussed by Arize AI CPO Aparna Dhinakaran at Re-Work, highlighting challenges such as lack of access to protected attributes, no easy way to check for model bias, tradeoff between fairness and business impact, and responsibility diffused across individuals, teams, and organizations. The presentation emphasized how human bias can be introduced into data through proxy information and sample size data, leading to biased models that disproportionately affect certain groups. To optimize model fairness, Aparna recommends increasing organizational investment, defining an ethical framework, and establishing tools for visibility, such as ML observability, which enables the identification of problems before deployment and allows for troubleshooting and fixing issues.
UbiOps and Arize are partnering to accelerate model building and deployment by providing a comprehensive observability platform for machine learning engineers, enabling teams to develop models, deploy them to production, and gain full visibility and control of their performance. This integration allows data scientists and ML engineers to work together seamlessly, validating model quality and performance prior to deploying to production, accelerating model deployment without high ops overhead, and automatically diagnosing issues that emerge in production. The platform provides features such as performance heatmaps, enabling teams to find opportunities for improvement and retraining, ultimately delivering better model performance and faster time-to-value.
ML observability refers to tools and practices that help teams monitor and understand the performance of their machine learning (ML) models in real-world scenarios. This is particularly important as more teams adopt ML to streamline their businesses or turn impractical technologies into reality. The challenge lies in translating research lab models to production environments, where data and feature transformations can be inconsistent, leading to poor model performance. By applying evaluation stores and introspection techniques, teams can identify gaps in training data, detect underperforming model slices, compare model performances, validate models, and troubleshoot issues in real-time, ultimately improving their ML efforts.
Beyond traditional monitoring, observability is a crucial aspect of understanding the health of complex data-driven systems. It enables teams to identify issues such as duplicate or stale data, model drift, and biased training datasets that can lead to unintended consequences. Observability provides granular information about data quality, schema changes, lineage, freshness, distribution, volume, and other key pillars of data health, allowing teams to detect problems early and prevent them from becoming bigger issues. Unlike monitoring, observability enables active learning, root cause analysis, and collaboration across cross-functional teams to resolve data issues before they impact the business. By applying principles of software application observability and reliability to data and ML, teams can build more trustworthy and reliable systems, gain insight into model performance, detect drift, and identify the "why" behind broken data pipelines and failed models. Ultimately, observability is essential for building a culture of trust in data-driven systems and making informed decisions based on accurate insights.
The concept of data ethics is being reevaluated by a group of AI researchers in Africa, who argue that the origin, collection, and sharing of data are often overlooked but critical components of AI ethics. They highlight issues such as deficit narratives, extractive data practices, and moral distance between data collectors and communities, which can lead to harm and mistrust. The authors emphasize the importance of building trust, respecting local norms and contexts, and ensuring that data is shared in a way that benefits both communities and science. They also argue that AI ethics must start with data collection and sharing, rather than just model development, to ensure fairness, equity, and justice in AI systems.
Arize AI has been named to the Forbes AI 50 list of most promising artificial intelligence companies of 2021, recognizing its leadership in Machine Learning Observability. The company's platform helps make machine learning models work in production by providing real-time monitoring, explanation, and troubleshooting capabilities. Arize AI is focused on addressing performance degradation issues that arise after ML model deployment, which can have significant financial impacts on businesses. The company was evaluated by Forbes alongside other top North American companies using AI to transform industries, and its selection reflects the importance of software that enables AI models to perform as expected in real-world applications.
Arize AI has partnered with Algorithmia to help organizations improve their machine learning operations (MLOps) and observability. The collaboration aims to enable companies to deliver more models into production, maximize model performance, and minimize risks associated with ML models. By integrating Arize's ML observability platform with Algorithmia's enterprise MLOps platform, customers can leverage the combined power of both platforms to deploy models and manage their performance at scale. This partnership aims to streamline machine learning operations, bridge the gap between data science and ML engineering, and provide a comprehensive solution for managing AI investments in businesses.
Coded Bias: An Insightful Look At AI, Algorithms And Their Risks To Society` Coded Bias is Netflix's deepest dive into the state of artificial intelligence, highlighting issues that are uncomfortably relevant to society. Commercially available facial recognition programs have a severe algorithmic bias against women and people of color, as seen in research by M.I.T. Media Lab researcher Joy Buolamwini. Flaws in AI applications can spread discrimination across daily life, affecting hiring, medical treatment, education, financial credit, and prison terms. The film shines a light on the opportunity to improve AI systems through better data leverage and development practices. To address algorithmic harm, innovation is needed, including tools for accountability, to enable marginalized populations to benefit from better technology and experience a better world.
In 2020, global greenhouse gas emissions decreased by 7% due to the COVID-19 pandemic's impact on travel and transportation. However, as people return to their daily routines, there is concern that carbon emissions will rise again. De-urbanization trends may lead to increased dependence on driving in smaller cities and suburbs, which could contribute to unsustainable living and higher carbon emissions. Google Maps has introduced a new feature that optimizes routes for lower fuel consumption based on factors like road incline and traffic congestion. This AI-driven approach incentivizes users to make environmentally friendly decisions, potentially reducing the carbon footprint of increased vehicle use across the country.
The article discusses the importance of Machine Learning (ML) tools for business executives, drawing parallels between the evolution of software development and the current state of ML. It highlights that just as discrete steps in the software development lifecycle were addressed by various solutions and tools, similar categories are emerging around researching, building, deploying, and monitoring models in ML. The article emphasizes that understanding AI/ML initiatives is now crucial for business leaders, especially in data-driven businesses. It also provides an overview of some key ML tools such as Feature Store, Model Store, and Evaluation Store, which can help CEOs participate in technical discussions around ML/AI initiatives.
A machine learning toolbox is essential for teams to apply machine learning successfully in their products. The toolbox consists of three fundamental tools: a Feature Store, a Model Store, and an Evaluation Store. A Feature Store enables the centralization of feature transformations, allowing for offline and online serving, team collaboration, and version control. A Model Store serves as a central repository for models and model versions, enabling reproducibility, tracking lineage, and integration with other tools. An Evaluation Store provides performance metrics, monitoring, and evaluation capabilities to ensure model quality and continuous improvement. Additional tools like Data Annotation Platforms, Model Serving Platforms, and AI Orchestration Platforms can complement the toolbox by handling data annotation, model deployment, and workflow management, respectively. By leveraging these three core tools and additional complementary tools, teams can successfully apply machine learning in their products and achieve rapid innovation.
Tammy Le has joined Arize AI as Vice President of Marketing and Strategy, bringing her expertise in product marketing from roles at Atlassian and Adobe. As a self-proclaimed introvert who prefers written communication, Tammy developed a passion for storytelling through her work at Tubemogul, which was acquired by Adobe. With a background in psychology and a Professional Baking & Pastry Arts certification, Tammy is excited to bring her insatiable curiosity to Arize AI and nurture diverse perspectives in the team. She aims to define this emerging space in AI/ML and create simple and compelling product stories. In her free time, Tammy enjoys baking as a way to flex a different part of her brain.
Machine learning and artificial intelligence systems are increasingly used by major companies for critical business decisions, but they face challenges such as algorithmic bias. Companies like Apple have been accused of biased AI models, while the use of AI in the US judicial system raises concerns about perpetuating systemic biases. The responsibility for these issues often falls on users rather than creators, and society is struggling to catch up with the ethical implications of AI/ML technologies. To address this problem, steps include admitting the importance of ethical validation, making protected class data available to modelers, breaking down barriers between teams and data, and employing emerging technologies for accountability.
Model monitoring has become increasingly important as machine learning infrastructure matures. However, there is no foolproof playbook for measuring model performance in every situation. Performance analysis can be complex, especially when ground truth is not immediately available or biased. In such cases, proxy metrics and statistical distances can be used to monitor prediction drift. Additionally, measuring business outcomes alongside model metrics provides a comprehensive understanding of how models affect customers' experiences with the product.
Kunal Shah is the newest member of Arize's Front-end Engineering team. Kunal holds a bachelor's degree in Computer Science from the University of Southern California and has experience as a front-end engineer at Omada Health and Pandora. He was influenced by the tech scene in the Bay Area growing up and enjoys tinkering with mechanical components, having built computers for his family. Kunal is excited to join Arize AI and contribute to bridging the gap between humanity and technology, focusing on understanding ethics in AI/ML and mitigating risks for society. In his free time, he enjoys listening to music and maintains a blog at realhitsonly.com.
Model observability plays a crucial role in detecting, diagnosing, and explaining regressions in deployed machine learning models. Some potential failure modes include Concept Drift, where the underlying task of a model changes over time; Data Drift or Feature Drift, which occurs when the distribution of model inputs changes; Training-prod skew, where the distribution of training data differs from production data; and Cascading Model Failures, which happen when multiple models are interconnected. Additionally, Outliers can be problematic as they may represent edge cases or adversarial attacks on a model. Monitoring tools help identify these issues and enable teams to improve their models after deployment.
Eunice Kokor, the newest member of Arize AI’s Front-end Engineering team, joins from Hearst Magazines where she worked on content display as a Front-end Engineer. Eunice holds a degree in Computer Science from Columbia University and has a passion for technology and problem-solving through code. She envisions an exciting future for fair, responsible, and ethical AI, aiming to help Arize build more tooling to better understand mission-critical machine learning problems. With a background in robotics and hackathons, Eunice is eager to join the team and contribute to Arize AI’s mission.
Arize AI and Spell have partnered to bring model observability to the Spell platform, allowing users to easily transition from research to production and troubleshoot model performance without consuming data science cycles. The integration combines Spell's autoscaling online model APIs with Arize's powerful model monitoring, explainability, and troubleshooting capabilities, enabling teams to build trust between research and end-users. With this partnership, Spell users will have early access to Arize's model observability platform, while Arize users can leverage Spell as a powerful MLOps platform for building and managing machine learning projects.
ML Observability is a platform that enables teams to analyze model degradation and identify the root cause of issues by connecting points across validation and production environments. This allows for a deeper understanding of the "why" behind performance changes, which is different from traditional model monitoring that focuses on aggregates and alerts. The platform provides features such as explainability insights, production feature data analysis, and distribution drift analysis to help teams troubleshoot and improve their models. By using ML Observability, teams can gain confidence in their models' performance, scale their operations, and gain a competitive advantage in the market.