An Empirical Study on Challenges for OpenAI Developers - arXiv

In contrast to traditional software development practices, OpenAI’s development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, plugins, and APIs), requiring developers to navigate unique methodologies and considerations specific to large language model development. ... … However, during OpenAI development, developers often encounter various challenges. For instance, correctly configuring and invoking OpenAI’s API can be difficult, including setting parameters, managing rate limits, and handling errors. For those unfamiliar with AI and LLMs, developing plugins and applications based on OpenAI’s technology can be daunting, involving integration, performance optimization, and ensuring security. Ensuring data privacy and security while handling user data is crucial. Developers must comply with relevant regulations and implement necessary security measures. … RQ3: What specific challenges for OpenAI developers? Result. We perform a manual analysis on 2,364 sampled questions and construct a taxonomy of challenges consisting of 27 categories. For example, *Prompt Design*, *Integration with Custom Applications*, and *Token Limitation*. In addition, based on this taxonomy, we summarize findings and actionable implications for stakeholders (such as developers and the OpenAI organization). … While the integration of OpenAI’s APIs and the development of plugins and GPTs offer significant advantages, they also present several challenges, such as: (1) Cost Management: Managing the computational and financial costs associated with training, fine-tuning, and deploying large AI models, which can be significantly higher than those for traditional software systems. … We summarize potential reasons as follows. First, the new and complex technological domains of OpenAI require deep expertise and skills from developers. This scarcity of qualified professionals results in limited responses. Second, the rapid evolution of OpenAI’s technology often leaves issues unresolved, leading to fewer available answers. Finally, the lack of comprehensive documentation and tailored support resources makes it difficult to address diverse developer needs, prolonging the resolution process for many questions. … Finding 3. The challenges faced by OpenAI developers are multifaceted and diverse, encompassing 27 distinct categories. These range from *Conceptual Questions* to *API Usage*, from *Prompt Design* to *Text Generation*, and from *Rate Limitation* to *Regulation*. … These include API integration methods, performance issues, output reproducibility issues, interpretability of output content, and so on. These challenges highlight the diverse and intricate nature of API integration and the need for clearer guidelines and examples from OpenAI to assist developers in these areas. As integration with custom applications is a major challenge, OpenAI could develop dedicated resources and support mechanisms to streamline this process. … Faults in API (B.1). When calling OpenAI APIs, developers frequently encounter a variety of issues, such as low-quality generated content, limitations in model comprehension, text coherence problems. These issues often result in outcomes that do not meet developers’ expectations. The majority of these issues are related to unsatisfactory output, such as the presence of extraneous information (like spaces and newlines) in the API’s responses^17^^17^17https://community.openai.com/t/578701, as well as phrase repetition in answers^18^^18^18https://community.openai.com/t/54737. … Discussion and Implication: The analysis of the various subcategories within the ”Generation and Understanding” category reveals several insights into the challenges developers face and the implications for improving OpenAI’s models and their usage. (1) API Usage Issues. A significant portion of the challenges are related to the practical use of APIs. Issues such as repeated responses from the Davinci model, errors in embedding API usage, and problems with the Whisper API during audio processing are common. … For example, developers encounter *RateLimitError* when calling gpt-3.5-turbo-0301^48^^48^48https://community.openai.com/t/566696. Additionally, developers inquire whether rate limits are shared among different APIs^49^^49^49https://community.openai.com/t/360331. Furthermore, developers ask about methods to increase the API rate limits^50^^50^50https://community.openai.com/t/248374. These types of questions account for 3.2% of the total challenges. … Discussion and Implication: Challenges such as API call costs, rate limitations, and token limitations are tightly linked to the development and usage of OpenAI’s services. Developers often express concerns about the costs associated with API calls, which are influenced by the choice of model and the number of tokens used in each request. Similarly, rate limitations are put in place to ensure service stability, but developers need to understand these limits and manage their API call frequencies accordingly.