Building a Receipt scanning agent

Learn how to automate multi-step technical processes by building a simple agent.

In this tutorial, you will learn how to build a Receipt Scanning agent. This agent will receive images of receipts, read the text and send you an email with the receipt details such as date, merchant and total cost.

In order to execute this function, this agent will utilize two tools: OCR Plugin to read text through images and the Email Management tool to send out the emails. This agent will also be connected to a Telegram bot to allow easier communication.

Engine - Build Section

Step 1: Create your first bot and set up configurations

Step-by-step instructions

Create a node

Create a new workflow by clicking "+ New" on the left panel. Your first node will appear on the canvas on the right.

Click "Set Bot" to configure bot settings.

Engine - Build Section

Configurations

Set Basic and Advanced configurations

Bot Settings - Configurations Section

Prompt:

Act as a personal receipt scanning and expense tracking assistant. You should receive images of receipts and send date, merchant and total cost to an email address specified by the client. You should also save the receipt information into a csv file and send a summary for a specified period to the user upon request.

Follow these steps:

1. Greet the user and introduce yourself as the receipt scanning and expense tracking assistant, explain that the agent can process photos of receipts, extract the date and cost, and automatically send it to the user's specified email address.

2. Explain that you can also send a summary for a specified date range.

3. Ask the user for an email address.

4. Instruct the user to take a photo of their most recent receipt and send it to the agent.

5. Utilize the OCR to extract the date and total amount from the receipt image.

6. Utilize Email Management to send an email to the user's designated email address, summarizing the purchase details (date, merchant, total).

7. Send a confirmation message after you've sent the email.

This prompt gives a general description of the purpose of this bot as well as specific step by step instructions. It also mentions neccessary tools (OCR and Email Management)

Bot Settings - Configurations Section

Fault tolerance is set to 5 and Auto Retry to 1, which means that if an errors occurs the bot will try 5 x 1 times before terminating.

This number is enough to tolerate minor errors and support bot's usage without using up too many tokens.

Learn more about bot configurations.

AI Model & Memory

Choose a suitable AI Model

Bot Settings - AI Model Section

For this bot, the default option - GPT-4O is suitable.

This model has a lower context window, can take images as input and has average pricing, it's good for simple tasks that don't require a lot of tools and memory and need to process images.

Learn more about AI Model configurations.

Choose memory settings

Bot Settings - Memory Section

Enabling memory for this bot would allow it to be more flexible when processing different requests.

Custom RAG is a suitable and more affordable option. For more complicated bots, Open AI Assistant would be more reccommended.

Text Embeding Ada is a well performing model with a medium price suitable for this agent.

Learn more about bot configurations.

Other

Choose a Tool usage mode

Bot Settings - Tool Section

Auto mode is a default option for bot implementation, allowing the bot to use all connected, active tools.

Learn more about tool configurations.

Connect a Telegram bot

Bot Settings - Channel Section

Paste your Telegram bot token to connect a Telegram bot.

Read detailed instructions on Telegram bot connection.

Learn more about channel configurations.

Bot Settings - Resrouces Section

This bot will not be using any additional resources.

Learn more about resources configurations.

Step 2: Add Email Management Tool

Step-by-step instructions

Connect a new node

Drag the point on the first bot to connect a new node. Press "Set Tool" to configure tool settings.

Engine canvas

Choose a Template

Search for "Email Management" and install the tool

Tool Settings - Script Section

You can find descriptions of available templates to connect more suitable tools. Change permission to access public or private tools.

Learn more about script settings.

Configurations

Set configuration details

Tool Settings - Configurations Section

Describe how your bot should use this tool in the description.

Learn more about configurations settings.

Step 3: Add OCR Tool

Step-by-step instructions

First Tab

Drag the point on the first bot to connect a new node. Press "Set Tool" to configure tool settings.

Engine canvas

Choose a Template

Search for "OCR" and install the tool

Tool Settings - Script Section

You can find descriptions of available templates to connect more suitable tools. Change permission to access public or private tools.

Learn more about script settings.

Configurations

Set configuration details

Tool Settings - Configurations Section

Describe how your bot should use this tool in the description.

Learn more about configurations settings.

Step 4: Test your bot

Sending a receipt to the agent through Telegram

Receipt Details sent by the agent:

Email sent by the bot

Step-by-step Instructions

Telegram Chat

Open the connected Telegram Bot to start testing!

Bot Configurations - Channel section

To try out your agent through the Telegram channel, you can access the bot by searching it directly, or by following the link and the QR code inside the bot vonfigurations, Channel section.

Learn more about Telegram bot connection.

Testing

Send messages to test your bot's functionality

Telegram bot convesation

Log Records of the Email Management Tool

As you send message to your bot either thorugh the build in console or the connected Telegram bot, the Log Records of each node will show live updates of internal processing.

Learn more about testing with the use of the console.

Step 5: Publish

Follow our next tutorial to learn how to publish this agent privately, within all members of your organization.

Publish an agent