A BENCHMARK FRAMEWORK FOR EVALUATING AUTONOMOUS AI AGENTS IN ACCOUNTING

Authors

  • Shodiyeva Malika Shermatovna, Mirzamaxmudova Mohigul Ulug’bek qizi , Mahmudjonov Humoyun Ulug’bek o’g’li student of ’’Tashkent Institute of Irrigation and Agricultural Mechanization Engineers’’ National Research University Author

Keywords:

autonomous AI agents, agentic AI, accounting automation, benchmarking, large language models, auditability, explainability, financial reporting, tax automation

Abstract

The rapid proliferation of large language model (LLM)-based autonomous agents has introduced new possibilities for automating complex, multi-step accounting tasks, ranging from journal entry generation and bank reconciliation to tax computation and financial statement drafting. Despite growing industry adoption, the accounting profession currently lacks a standardized, discipline-specific benchmark for evaluating the reliability, regulatory compliance, and auditability of these agents. Existing AI benchmarks are largely borrowed from general natural-language-processing or software-engineering domains and fail to capture the procedural rigor, traceability, and normative constraints that define professional accounting practice.
This paper proposes a Benchmark Framework for Evaluating Autonomous AI Agents in Accounting (BEAAA), a structured evaluation protocol comprising six dimensions: task accuracy, regulatory compliance, auditability and traceability, robustness to adversarial or ambiguous inputs, cost-efficiency, and explainability. The framework operationalizes each dimension through a task taxonomy spanning four accounting sub-domains — bookkeeping, reconciliation, financial reporting, and tax preparation — and a graded scoring rubric that captures partial task completion rather than binary success/failure outcomes.
We illustrate the framework through a pilot evaluation of three representative agent architectures across a synthetic task suite of 120 accounting scenarios. Results indicate substantial variance across agents in compliance and auditability scores despite comparable raw task-accuracy scores, suggesting that accuracy alone is an insufficient basis for deployment decisions in regulated accounting environments. The proposed framework offers accounting researchers, software vendors, and regulators a common vocabulary and methodology for comparing autonomous agents, and it identifies auditability and explainability as the most under-addressed dimensions in current agent design. Implications for standard-setting bodies and directions for future benchmark development are discussed.

References

1.Bao, Y., Ke, B., Li, B., Yu, Y. J., & Zhang, J. (2020). Detecting accounting fraud in publicly traded U.S. firms using a machine learning approach. Journal of Accounting Research, 58(1), 199-235.

2.Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.

3.Cao, S., Jiang, W., Yang, B., & Zhang, A. L. (2021). How to talk when a machine is listening: Corporate disclosure in the age of AI. Review of Financial Studies, 34(8), 3852-3897.

4.Deloitte. (2025). Agentic AI in finance and accounting: 2025 industry outlook. Deloitte Insights.

5.EY. (2024). The future of the finance function: Autonomous agents and the modern controller. EY Global.

6.Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? Proceedings of the International Conference on Learning Representations.

7.Kumar, R., & Ravindran, S. (2023). Automated generation of financial disclosures using large language models. Journal of Emerging Technologies in Accounting, 20(2), 45-63.

8.Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., et al. (2024). AgentBench: Evaluating LLMs as agents. Proceedings of the International Conference on Learning Representations.

9.Moffitt, K. C., Rozario, A. M., & Vasarhelyi, M. A. (2018). Robotic process automation for auditing. Journal of Emerging Technologies in Accounting, 15(1), 1-10.

10.IAASB. (2023). Exploring the growing use of technology in the audit, with a focus on automated tools and techniques. International Auditing and Assurance Standards Board.

11.Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022). Outsider oversight: Designing a third-party audit ecosystem for AI governance. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 557-571.

12.Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, 30(3), 286-297.

13.PwC. (2025). Agentic AI adoption survey: Finance and accounting functions. PricewaterhouseCoopers.

14.Russell, S., & Norvig, P. (2021). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.

15.Sun, T., & Ferreira, M. (2024). Autonomous agents in professional services: A scoping review. Accounting, Organizations and Society, 108, 101-118.

16.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.

17.Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR.

18.Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., et al. (2023). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 1-26.

19.Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115-152.

20.Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. Proceedings of ICLR.

21.Zhang, C., & Liu, Q. (2022). Machine learning applications in audit risk assessment: A systematic review. International Journal of Accounting Information Systems, 45, 100-116.

12

Downloads

Published

2026-07-12