View source for AI ROI observability



Weinberg rules on quality:
https://www.shino.de/blog/wp-content/uploads/2022/10/Quality-Software-Management.pdf

AI didn't create new problems:
https://itrevolution.com/product/revenge-of-qa/

AI Summary

# Measuring AI Impact on Productivity and Code Quality:
Roland and the group discussed the challenge of measuring whether AI tools are genuinely improving productivity and code quality, with Roland sharing their dashboard approach and others contributing observations about review processes, metrics, and the limitations of current measurement methods.

# Dashboard Metrics and Observations: 
Roland described their dashboard, which tracks AI-assisted pull requests, adoption rates, and tool costs at the team level. The dashboard, though only a few days old, revealed that AI-assisted pull requests tend to have more commits but are not significantly larger in terms of lines of code. The tagging of AI involvement has been ongoing for three to four weeks, and while more changes and feedback are observed in AI-assisted pull requests, the meaning of these data points remains unclear.

# Review Process Changes: 
Participants noted that the review process has become weaker with increased AI involvement, with more pull requests being rubber-stamped and fewer humans deeply reviewing code. However, Jtf pointed out that rubber-stamping was already common before AI, suggesting that the issue is not new but more visible due to AI's prominence.

# Effort vs Impact Metrics:
The group discussed the ease of collecting effort metrics, such as number of commits or lines of code, versus the difficulty of measuring impact metrics like code quality or business outcomes. Roland highlighted the assumption that merged pull requests are of sufficient quality, but acknowledged the lack of direct metrics to confirm this.

# AI-Assisted Code Maintenance: 
There was debate about whether AI-generated code is harder to maintain, with some participants sharing examples of effective AI use for technical debt reduction and others cautioning about potential quality issues in projects with high AI-generated code.

# Management's Interest in AI ROI:
Roland explained that management is increasingly interested in AI's return on investment due to its visibility as a line item, prompting questions about productivity and cost justification. The group agreed that measuring improvement should not be limited to AI but should apply to all changes.

# Modelling and Metrics for Business Performance: 
The team explored how to build mathematical models to measure business performance, profitability, and quality, emphasising that such models should be used regardless of AI involvement and that profitability is the ultimate measure of success.
	Profitability as Key Metric: Participants agreed that profitability, particularly renewal rates, is the most reliable measure of business performance. They discussed the limitations of lagging metrics and the need for leading proxy metrics to estimate future profitability.
	Building Mathematical Models: The group advocated for constructing mathematical or symbolic models to reason about the impact of changes, including AI adoption. They shared examples of modelling user behaviour and market impact, highlighting the importance of refining models based on real-world data.
	Quality Measurement Approaches: Quality was defined as 'value to someone who matters,' with the team discussing how to measure quality in production and the importance of building products that meet customer needs. They noted that quality requirements vary by industry, with some sectors unable to tolerate failures.
	Model Application Examples: Jtf and others provided examples of using models to predict the impact of UI changes on user behaviour and market outcomes, including estimating sample sizes and timeframes needed to detect meaningful differences.

# Iterative Product Development and Learning: 
Roland and the group discussed the advantages of using AI to accelerate iterative product development, enabling faster learning and validation of ideas, while noting that speed does not always equate to lower costs.
	Accelerated Learning with AI: AI enables teams to quickly build multiple versions of a product, facilitating rapid experimentation and learning. This lowers the cost of iteration and allows for more frequent validation of hypotheses, especially in early-stage product development.
	Cost vs Speed Considerations: While AI can significantly reduce development time, participants noted that it may not always lower overall costs due to factors like token expenses and maintenance. The value of faster learning is difficult to quantify but can be substantial.
	Optimising for Learning: The group emphasised the importance of optimising for learning rather than just output, especially when product requirements are uncertain. AI's ability to support rapid prototyping is seen as a major opportunity.

# Challenges in Measuring Quality and Maintenance: 
The team examined the difficulties in measuring code quality and maintainability, especially with increased AI-generated code, and discussed the need for ongoing evaluation and adaptation of measurement approaches.
	Production Quality Measurement: Participants agreed that quality should be measured in production, with metrics such as user adoption, repeat usage, and ease of ongoing changes. They highlighted the risk that AI-generated code may be harder to maintain and adapt over time.
Industry-Specific Quality Requirements: 
The group discussed how quality requirements differ across industries, with some sectors unable to tolerate failures due to safety or financial risks, and the importance of adapting measurement models accordingly.
	Critical Industry Examples: Examples from banking, insurance, and emergency services were shared, illustrating the need for stringent quality controls and minimal bugs in software where failures can have severe consequences.