A BENCHMARK FRAMEWORK FOR EVALUATING AUTONOMOUS AI AGENTS IN ACCOUNTING
Keywords:
autonomous AI agents, agentic AI, accounting automation, benchmarking, large language models, auditability, explainability, financial reporting, tax automationAbstract
The rapid proliferation of large language model (LLM)-based autonomous agents has introduced new possibilities for automating complex, multi-step accounting tasks, ranging from journal entry generation and bank reconciliation to tax computation and financial statement drafting. Despite growing industry adoption, the accounting profession currently lacks a standardized, discipline-specific benchmark for evaluating the reliability, regulatory compliance, and auditability of these agents. Existing AI benchmarks are largely borrowed from general natural-language-processing or software-engineering domains and fail to capture the procedural rigor, traceability, and normative constraints that define professional accounting practice.
This paper proposes a Benchmark Framework for Evaluating Autonomous AI Agents in Accounting (BEAAA), a structured evaluation protocol comprising six dimensions: task accuracy, regulatory compliance, auditability and traceability, robustness to adversarial or ambiguous inputs, cost-efficiency, and explainability. The framework operationalizes each dimension through a task taxonomy spanning four accounting sub-domains — bookkeeping, reconciliation, financial reporting, and tax preparation — and a graded scoring rubric that captures partial task completion rather than binary success/failure outcomes.
We illustrate the framework through a pilot evaluation of three representative agent architectures across a synthetic task suite of 120 accounting scenarios. Results indicate substantial variance across agents in compliance and auditability scores despite comparable raw task-accuracy scores, suggesting that accuracy alone is an insufficient basis for deployment decisions in regulated accounting environments. The proposed framework offers accounting researchers, software vendors, and regulators a common vocabulary and methodology for comparing autonomous agents, and it identifies auditability and explainability as the most under-addressed dimensions in current agent design. Implications for standard-setting bodies and directions for future benchmark development are discussed.
References
1.Bao, Y., Ke, B., Li, B., Yu, Y. J., & Zhang, J. (2020). Detecting accounting fraud in publicly traded U.S. firms using a machine learning approach. Journal of Accounting Research, 58(1), 199-235.
2.Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
3.Cao, S., Jiang, W., Yang, B., & Zhang, A. L. (2021). How to talk when a machine is listening: Corporate disclosure in the age of AI. Review of Financial Studies, 34(8), 3852-3897.
4.Deloitte. (2025). Agentic AI in finance and accounting: 2025 industry outlook. Deloitte Insights.
5.EY. (2024). The future of the finance function: Autonomous agents and the modern controller. EY Global.
6.Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? Proceedings of the International Conference on Learning Representations.
7.Kumar, R., & Ravindran, S. (2023). Automated generation of financial disclosures using large language models. Journal of Emerging Technologies in Accounting, 20(2), 45-63.
8.Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., et al. (2024). AgentBench: Evaluating LLMs as agents. Proceedings of the International Conference on Learning Representations.
9.Moffitt, K. C., Rozario, A. M., & Vasarhelyi, M. A. (2018). Robotic process automation for auditing. Journal of Emerging Technologies in Accounting, 15(1), 1-10.
10.IAASB. (2023). Exploring the growing use of technology in the audit, with a focus on automated tools and techniques. International Auditing and Assurance Standards Board.
11.Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022). Outsider oversight: Designing a third-party audit ecosystem for AI governance. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 557-571.
12.Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, 30(3), 286-297.
13.PwC. (2025). Agentic AI adoption survey: Finance and accounting functions. PricewaterhouseCoopers.
14.Russell, S., & Norvig, P. (2021). Artificial Intelligence: A Modern Approach (4th ed.). Pearson.
15.Sun, T., & Ferreira, M. (2024). Autonomous agents in professional services: A scoping review. Accounting, Organizations and Society, 108, 101-118.
16.Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
17.Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR.
18.Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., et al. (2023). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 1-26.
19.Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115-152.
20.Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. Proceedings of ICLR.
21.Zhang, C., & Liu, Q. (2022). Machine learning applications in audit risk assessment: A systematic review. International Journal of Accounting Information Systems, 45, 100-116.
12




United Kingdom
Germany
France
Uzbekistan
Türkiye
Kazakhstan
Kyrgyzstan
Turkmenistan
Azerbaijan
India
Republic of Korea
Japan
United States of America
Kosovo
Italy
Spain
Poland
Netherlands
Belgium
Sweden
Norway
Finland
Denmark
Switzerland
Austria
Czech Republic
Hungary
Romania
Bulgaria
Greece
Portugal