Osemu Finance

Flowchart planning for Go app architecture

TLDR;

The rate retrieval was so fast we had to add artificial delay on the frontend.

Background

If you held a remote role from Ghana, or had USD to spare in May 2024, you’re probably aware of, if not affected by, Wise’s trading license being seized by the Bank of Ghana. A $2bn remittance market stopped without a clear reason. This made converting USD to GHS quite the pain. Families couldn’t access money sent from abroad. Workers could not pay bills with their hard earned money.

Sebastian , who has a flawless track record of handling dollar exchanges for people, sat with me on the balcony and we brainstormed how to get around the solution. Problem solving is our favorite activity.

In a perfect world, there would be APIs for the current rates which could be used as a base for exchanges. Unfortunately at the time of building this, there weren’t any to be found, specific to local currency. After a lot of searching, we went with Stanbic’s. Even though they do not provide an API, they do have a PDF file they update every working day.

This was my first fully fledged Go project ever since I took on the language. This was a chance for me to not just build a CRUD app, but use the tool for what it can be used for.

Getting the PDF

The best we could get was a PDF that’s posted on Stanbic’s website and is updated every working day. All I need to do now is to treat it like a newspaper: get one everyday and read to get my information. Using a combination of Go’s http.Get method to get the file along with io.ReadAll to read the body, there was no stopping me from getting that data 😆.

An interesting thing here to note is Go is a recursive and explicit language. Getting the file, reading the file and saving the file were separate processes that were handled, and for very good reason. I talk about that in the next section

func getPdf(hasBeenUpdatedToday *bool) (error error) {
	resp, err := http.Get("https://www.stanbicbank.com.gh/static_file/ghana/Downloadable%20Files/Rates/Daily_Forex_Rates.pdf")
	...
	defer resp.Body.Close()
	...
	file, readErr := io.ReadAll(resp.Body)
	...
	writeErr := os.WriteFile("Daily_Forex_Rates.pdf", file, 0644)
	...
}

Validating current data

There were a couple of considerations with data validity. How do I know the PDF was the current day’s? I considered the following:

Download the PDF and check the metadata to see when it was created or updated.
I could also download the PDF, check the date in the PDF, if it has the current date.

These two approaches involve downloading the PDF. That many round trips feel pretty excessive. and resource intensive, considering . It also helps to note that Stanbic has rate limiting in place (found out the hard way).

Thinking about this problem, I drew an analogy to visiting a physical store. When you go shopping, you don’t immediately walk inside and start browsing. First, you check if the store is actually open. If it’s closed, you save yourself time and effort by coming back later, rather than wandering around inside. Similarly, I realized I could check if the PDF had been updated before committing to downloading it.

Drawing from my experience building HTTP servers, I remembered that response headers contain valuable metadata. I could inspect the Last-Modified header first. If it showed the current date, I’d know the data was fresh and worth downloading. This approach would save unnecessary round trips and resources while respecting Stanbic’s rate limits.

func getPdf(hasBeenUpdatedToday *bool) (error error) {
	resp, err := http.Get("https://www.stanbicbank.com.gh/static_file/ghana/Downloadable%20Files/Rates/Daily_Forex_Rates.pdf")
	if err != nil {
		logWithFileLine(err)
		return err
	}

	last_modified := resp.Header.Get("Last-Modified")

	// parse last modified date
	last_modified_time, _ := time.Parse(time.RFC1123, last_modified)

	res := DateEqual(time.Now(), last_modified_time)

	defer resp.Body.Close()

	if res {
		fmt.Println("New data")

		if *hasBeenUpdatedToday {
			fmt.Println("Already updated today")
			return nil
		}
	}

	file, readErr := io.ReadAll(resp.Body)
	if readErr != nil {
		logWithFileLine(readErr)
		return readErr
	}

	writeErr := os.WriteFile("Daily_Forex_Rates.pdf", file, 0644)
	if writeErr != nil {
		logWithFileLine(writeErr)
		return writeErr
	}

	return nil
}

Reading the PDF

I went down the rabbit hole of PDF encoding and how delicate the work can be, thinking it would be less than an hour’s work. I was very wrong. It’s a complex topic and I can see why Adobe has such an upper hand when it comes to PDF processing. I (smartly) resorted to using a dependency. This is what the code ended up looking like

func readPdf(path string, currency map[string]string, currencies_list *[]currency) error {
	f, r, err := pdf.Open(path)
	defer func() {
		_ = f.Close()
	}()
	if err != nil {
		logWithFileLine(err)
		return err
	}
	totalPage := r.NumPage()

	for pageIndex := 1; pageIndex <= totalPage; pageIndex++ {
		p := r.Page(pageIndex)
		if p.V.IsNull() {
			continue
		}

		rows, _ := p.GetTextByRow()
		for _, row := range rows {
			whole_word := ""
			for _, word := range row.Content {
				whole_word += word.S
			}

			for _, pre := range currency {
				if s.Contains(whole_word, pre) {
					index := s.Index(whole_word, pre)
					// +1 to remove whitespace in front
					cut_range := index + len(pre) + 1
					rates := s.Split(whole_word[cut_range:], " ")

					buying, _ := strconv.ParseFloat(rates[0], 64)
					selling, _ := strconv.ParseFloat(rates[1], 64)

					curr := newCurrency(pre, rate{buying, selling})

					*currencies_list = append(*currencies_list, curr)

				}
			}
		}
	}
	return nil
}

Storing rates

We would need a way to keep previous rates collected in several cases.

Maybe things were being updated and we weren’t noticing. We would need a backlog to be able to check that.
Comparing the current rate to that of previous days to monitor changes

We could get it from the list of PDFs taken from the page. But why do that when databases exist? The choice of SQLite reflects the application’s requirements: single-node operation, moderate data volumes, and file-based portability.

CREATE TABLE rates (
    id INTEGER PRIMARY KEY, 
    currency TEXT, 
    buying FLOAT, 
    selling FLOAT, 
    created_at DATETIME, 
    updated_at DATETIME
);

For having to go back and check if the file had been updated, I used Go’s Ticker library in a Goroutine for asynchronicity. With consideration for business hour restrictions, I had it only work between 10 AM and 3 PM GMT. And during that time, it would go if it has been updated, if it hasn’t been updated, then just do nothing. And then if it has been updated, get the info, download the PDF, save it, and then parse it. Once everything has gone through successfully, then set the flag for if it has been updated to date or true. This set flag resets every day at midnight so it can start off the whole process the next day.

func isWithinBusinessHours() bool {
	now := time.Now().UTC()
	hour := now.Hour()
	return hour >= 10 && hour <= 15
}

...
	go func() {
		for range t.C {
			if time.Now().Hour() == 0 {
				updateUpdatedTodayFlag(&hasBeenUpdatedToday, false)
			}

			fmt.Println("has been updated today: ", hasBeenUpdatedToday)

			if !hasBeenUpdatedToday && isWithinBusinessHours() {
				getRatesFromPDF(&hasBeenUpdatedToday, db)
			}
		}
	}()

This was quite the exercise. Using Go was a great choice for a number of reasons. It had concurrency as a feature meant I didn’t have to break pattern. I would’ve had to use something like RabbitMQ if that wasn’t the case. It was an option but jumping straight to a dependency without understanding my tools wouldn’t have helped much with my learning.

go build returned a 12.6mb binary. Everything was pretty lightweight and the whole app was just a few kilobytes. The result was that the front end was so quick, I had to place an artificial timer in there because people thought the data wasn’t live! Users thought it was a flash of unstyled content but that was actually a feature. That was something.

Extra

Additional stuff that we worked on was SMS subscriptions so you can have daily SMS sent to you when the new rates are in. That was going great, unfortunately we had to discontinue that because getting a good SMS provider in Ghana tends to be an issue. Hubtel APIs tend to not be secure or straightforward. The other ones were just far too unstable to rely on. If anyone knows any SMS provider, please do let me know. I hope you enjoyed that reading. See you later.