Dotch.app: Testing
Dotch works! Here’s a video — still buggy, but it works.
You can try it out yourself at https://dotch.app. If you have any feedback, please send to dotch.app@gmail.com.
Technical Design
I built the frontend with Vue 3 and the backend with Node JS. The API has two routes: POST /scan and GET /receipt/:id.
It all starts from the frontend. A WebCam component takes a photo using HTML5 navigator and prints that image onto a canvas.
The frontend sends the image to the server as a Base64 file, which is then formatted and passed to the Mindee Node JS client for parsing.
const getDocument = async (request) => {
let base64String;
const parts = request.parts();
for await (const part of parts) {
if (part.value) base64String = part.value;
}
if (base64String.includes(';base64,')) {
base64String = base64String.split(';base64,')[1];
}
try {
const inputSource = mindeeClient.docFromBase64(base64String, "receipt-data");
const apiRequest = mindeeClient.parse(mindee.product.ReceiptV5, inputSource);
const apiResponse = await apiRequest;
const { document } = apiResponse;
return document.inference;
} catch {
return null;
}
}
Mindee returns the document type, labels and line items. I use this to decide if the document is a receipt. If it is, I filter out the important information and send it back to the client to display.
const isReceipt = (document) => {
const { documentType, lineItems } = document.prediction;
const docIsReceipt = documentType.value.toLowerCase().includes('receipt');
const hasLineItems = lineItems.length > 0;
const docIsConfident = documentType.confidence > 0.6;
return docIsReceipt && hasLineItems && docIsConfident;
}
const filterReceiptFields = (prediction) => ({
merchant: prediction.supplierName.value,
date: prediction.date.value,
currency: prediction.locale.currency,
items: prediction.lineItems,
taxes: prediction.taxes,
tax: prediction.totalTax.value,
total: prediction.totalAmount.value,
});
Notes
- There’s no database yet. I’m hoping to implement lowdb in the future but for now I use a variable to store uploaded receipts. This means every time the server is restarted, all receipts are refreshed. You can see the code on Glitch here: https://glitch.com/edit/#!/dotch
- The line items are often wrong. For best results, put the receipt on a flat surface and clean your camera. I need to improve the isReceipt logic to aggregate the confidence score per line item and if it’s too low, request a better image from the client.
- ChatGPT and Copilot were such a great help! I still used Google and Stack Overflow for specific issues, but sparingly. Here’s a log of all the requests I asked ChatGPT for: https://chat.openai.com/c/071b04c2-553f-4a64-af1b-b74b95020ff4
What next?
I’m testing, getting feedback and improving the app in my spare time. You can try it out yourself at https://dotch.app. If you have any feedback, please send to dotch.app@gmail.com.