Reading Image Metadata

Reading Image Metadata

Playing around with reading binary encoded Exif data

2021-06-13

I’ve recently got back from a trip and like everyone, I have a lot of pictures. I thought to myself that, it would be nice to arrange the pictures by date. Why not try and write some code to do that? Should be easy.

Let’s put the R in R&D

Quick google search taught me that this metadata I was after is called “Exif” data.
Why “Exif” you wonder? Well, I did of course, and it stands for “Exchangeable image file format”.

Alright, so a quick apt-get install got me a command line exif util. NOICE.


sudo apt-get install exif

This is what I got by feeding this cli tool a random pic I had on my disk.


EXIF tags in '~/Pictures/11.10.17/20120349_235641.jpg' ('Intel' byte order):
--------------------+----------------------------------------------------------
Tag                 |Value
--------------------+----------------------------------------------------------
Image Width         |3264
Image Length        |2448
Manufacturer        |samsung
Model               |SM-G5700
Orientation         |Bottom-right
X-Resolution        |72
Y-Resolution        |72
Resolution Unit     |Inch
Software            |G5700ZHU1APL1
Date and Time       |2017:09:29 23:33:42
YCbCr Positioning   |Centered
Exposure Time       |1/8 sec.
F-Number            |f/1.9
Exposure Program    |Normal program
ISO Speed Ratings   |1000
Exif Version        |Exif Version 2.2
Date and Time (Origi|2018:01:22 03:42:42
Date and Time (Digit|2018:01:22 03:42:42
Maximum Aperture Val|1.85 EV (f/1.9)
Metering Mode       |Center-weighted average
Flash               |Flash did not fire
Focal Length        |2.5 mm
Maker Note          |98 bytes undefined data
Color Space         |sRGB
Pixel X Dimension   |3264
Pixel Y Dimension   |2448
Exposure Mode       |Auto exposure
White Balance       |Auto white balance
Focal Length in 35mm|24
Scene Capture Type  |Standard
FlashPixVersion     |FlashPix Version 1.0
--------------------+----------------------------------------------------------

Super cool!
So I can basically arrange my pictures by date, time, camera, size and all sorts of stuff.
Next I need to see what libraries I can use in order to extract that precious metadata. I chose Go as my weapon of choice for that expedition, since I’ve been using it quite a lot lately and I can actually remember a lot of syntax.

Another google search showed this github library with a native go implementation for reading Exif data.

Hey, let’s wait a minute.. Do I really need this whole library just to read that “Exif” format…?
Well, not having to answer to anyone why I should do it myself and not “reinvent the wheel” is quite refreshing! I’ll arrange my photos later… That github repo is probably good for reference plus whatever I’ll find online. Ready? Set! Go!

I’m not running anywhere or trying to do this really fast

Finding where it starts

According to this document we need to find the APP1 marker. After the APP1 marker there will be the “APP1 data area” that I am looking for. Alright, so in this document it’s written that ‘The marker 0xFFE0~0xFFEF is named “Application Marker”'. What do I need to look for?

I’ve set up my code like this so I could go byte by byte and search for this “APP1” marker. BTW, I took some inspiration from the goexif library. The peek and discard technique is pretty nice.


package main

import (
	"bufio"
	"fmt"
	"os"
)

func compBytes(a []byte, b []byte) bool {
	lena := len(a)
	if lena != len(b) { // if they are not equal in length they cannot be equal
		return false 
	}
	// check if any byte is different
	for i := 0; i < lena; i++ {
		if a[i] != b[i] {
			return false
		}
	}
	return true 
}


func main () {
	fname := "~/Pictures/11.10.17/20120349_235641.jpg"
	fd, err := os.Open(fname) // open file read only
	if err != nil {
		fmt.Println("Error: ", err)
		os.Exit(1)
	}
	defer fd.Close()
	marker := []byte{0xF, 0xF, 0xE, 0x1}
	br := bufio.NewReader(fd)
	start := 0
	bytesRead := 0
	for {
		bts, err := br.Peek(len(marker))
		if err != nil {
			fmt.Println(err)
			break
		}
		if compBytes(bts, marker) {
			fmt.Printf("found %s at %d\n", bts, start)
			break
		}
		start++
		bytesRead++
		_, err = br.Discard(1)


	}


	fmt.Println("Bytes read: ", bytesRead)
	fmt.Println("Start: ", start)

}


Well, that didn’t work… The file was read through to the end until getting EOF.
I guess I made some fundamental mistake because I cannot really grok this whole thing…

Maybe this is a byte containing a value and not a sequence of bytes?

...

func main () {
	fname := "~/Pictures/11.10.17/20120349_235641.jpg"
	fd, err := os.Open(fname) // open file read only
	if err != nil {
		fmt.Println("Error: ", err)
		os.Exit(1)
	}
	defer fd.Close()
	// marker := []byte{0xF, 0xF, 0xE, 0x1}
	APP1 := 0xFFE1
	br := bufio.NewReader(fd)
	start := 0
	bytesRead := 0
	for {
		// bts, err := br.Peek(len(marker))
		bts, err := br.Peek(1)
		if err != nil {
			fmt.Println(err)
			break
		}
		if bts[0] == byte(APP1) {
			fmt.Println("eureka")
		}
		// if compBytes(bts, marker) {
		// 	fmt.Printf("found %s at %d\n", bts, start)
		// 	break
		// }
		start++
		bytesRead++
		_, err = br.Discard(1)


	}

	fmt.Println("Bytes read: ", bytesRead)
	fmt.Println("Start: ", start)

}

Here’s what I got..


eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
EOF
Bytes read:  2315185
Start:  2315185

Man… There are a lot of those. So maybe I’m right maybe I’m wrong. I mean.. Maybe there are a lot of those (?) The document says that after that header there’s ‘SSSS’ marking the size. Wish I knew what that means, pretty cryptic for all I know. Is it a byte array marking the size? is it just one byte as an int? Does S stands for size? OOooooofff… So much fun being a noob at something.

Ok, I’ll try printing the first few bytes. That might give me a hint.


ff
d8
ff
e1

Aha! It’s 2 bytes… OK, let’s keep going with this theory. Now my code looks like this:

...

func main () {
	fname := "~/Pictures/11.10.17/20120349_235641.jpg"
	fd, err := os.Open(fname) // open file read only
	if err != nil {
		fmt.Println("Error: ", err)
		os.Exit(1)
	}
	defer fd.Close()
	marker := []byte{0xFF,  0xE1}
	// APP1 := 0xFFE1
	br := bufio.NewReader(fd)
	start := 0
	bytesRead := 0
	for {
		bts, err := br.Peek(len(marker))
		if err != nil {
			fmt.Println(err)
			break
		}
		if compBytes(bts, marker) {
			fmt.Printf("found %x at %d\n", bts, start)
			break
		}
		start++
		bytesRead++
		_, err = br.Discard(1)


	}

	fmt.Println("Bytes read: ", bytesRead)
	fmt.Println("Start: ", start)

}

And returns this:


found ffe1 at 2
Bytes read:  2
Start:  2

Great! I can just found the FFE1 marker. Let’s see if I can find the following “Exif” string like the doc says: “Exif data is starts from ASCII character “Exif” and 2bytes of 0x00, and then Exif data follows.”

So this is where I’m at now, a bit ugly but we’ll make it nicer later.

....

	for {
		bts, err := br.Peek(len(marker))
		if err != nil {
			fmt.Println(err)
			break
		}
		if compBytes(bts, marker) {
			fmt.Printf("found %x at %d\n", bts, start)
			br.Discard(2)
			bts, _ := br.Peek(2)
			// should be SSSS
			vint := binary.BigEndian.Uint16(bts)
			fmt.Println("vint: ", vint)
			fmt.Printf("SSSS is %d\n", bts)
			br.Discard(2)
			bts, _ = br.Peek(4)
			fmt.Println(fmt.Sprintf("Exif marker is %s", bts))

			break
		}
...


And here’s what I got:


found ffe1 at 2
vint:  638
SSSS is [2 126]
Exif marker is Exif

Extracting the data

Looks like we are on the right track! So, the size of our encoded data is 638 (determined by turning ‘SSSS’ to uint16). I’ll try to get that and turn it into a string.

...

			bts, _ = br.Peek(4)
			fmt.Println(fmt.Sprintf("Exif marker is %s", bts))
			br.Discard(4)
			bts, _ = br.Peek(2)
			fmt.Println("should be 2 zeros", bts)
			br.Discard(2)
			exifData, _ := br.Peek(628) // minus 10 we already read
			fmt.Printf("Exif data: %s\n", exifData)

...

We can already see there’s something there:


$ go run main.go 
found ffe1 at 2
vint:  638
SSSS is [2 126]
Exif marker is Exif
should be 2 zeros [0 0]
Exif data: II*
             �
              ��	���(1�2�i��samsungSM-G5700G5700ZHU1APL12017:09:29 23:33:42HH��V��^"�'���0220�.�B�f�	�
�n|�b����
         ��	����0100 
                         Z@P2017:09:29 23:33:422017:09:29 23:33:4�d�d�d
Bytes read:  2
Start:  2


When I look at it now I see that the data starts with II which according to the doc means it’s “Intel byte code”. I have been interpreting the size bytes as Big Endian but maybe it’s the other one.. So is our data 32238 or 638 bytes long? Since the document doesn’t state which byte order to use in which occurrence I’ll go and dig into go-exif's code and see how they solved it.

Found this bit:


	BigEndianBoBytes    = [2]byte{'M', 'M'}
	LittleEndianBoBytes = [2]byte{'I', 'I'}

	ByteOrderLookup = map[[2]byte]binary.ByteOrder{
		BigEndianBoBytes:    binary.BigEndian,
		LittleEndianBoBytes: binary.LittleEndian,
	}

	ByteOrderLookupR = map[binary.ByteOrder][2]byte{
		binary.BigEndian:    BigEndianBoBytes,
		binary.LittleEndian: LittleEndianBoBytes,
	}

	ExifFixedBytesLookup = map[binary.ByteOrder][2]byte{
		binary.LittleEndian: {0x2a, 0x00},
		binary.BigEndian:    {0x00, 0x2a},
	}

and this (inside a function):



	byteOrderBytes := [2]byte{data[0], data[1]}

	byteOrder, found := ByteOrderLookup[byteOrderBytes]
	if found == false {
		// exifLogger.Warningf(nil, "EXIF byte-order not recognized: [%v]", byteOrderBytes)
		return eh, ErrNoExif
	}

Looks like the first two bytes in a header determine the byte order. Cross referencing the two amazing resources I have (for real, not to be taken for granted) I understand that Intel code is little endian and Motorola is big.

The difference between big and small endian is the order the bytes are written at. Big endian is human readable - big numbers go last, but it depends where you start from…. Anyway if you want a detailed explanation, you can get one here

Before I continue I have to make the code a little bit more organized (this has turned into a mess).

Here I wrap the bufio.Reader I used before to give a more likable API and also added some nice utils:

exif/utils.go


package exif

import (
	"bufio"
	"fmt"
	"io"
)





func compBytes(a []byte, b []byte) bool {
	lena := len(a)
	if lena != len(b) { // if they are not equal in length they cannot be equal
		return false 
	}
	// check if any byte is different
	for i := 0; i < lena; i++ {
		if a[i] != b[i] {
			return false
		}
	}
	return true 
}

func forceASCII(s []byte) string {
	rs := make([]byte, 0, len(s))
	for _, r := range s {
	  if r <= 127 {
		rs = append(rs, r)
	  }
	}
	return string(rs)
  }




type WindowReader struct {
	br *bufio.Reader
	curr int
}


func (wr *WindowReader) NextWindow(winLength int) ([]byte, error) {
	bts, err := wr.br.Peek(winLength)
	if err != nil {
		return nil, err
	}
	_, err = wr.br.Discard(winLength)
	if err != nil {
		return nil, err
	}
	wr.curr += winLength
	return bts, nil
}

func (wr *WindowReader) ReadUntil(when int) ([]byte, error) {
	readTo := make([]byte, when - wr.curr)
	n, err := io.ReadFull(wr.br, readTo); if err != nil { return nil, err}
	fmt.Println("Written: ", n)
	wr.curr += n
	return readTo, nil
}

func (wr *WindowReader) ScanUntil(toFind []byte) (bool, int, error) {
	scanned := 0
	for {
		bts, err := wr.br.Peek(len(toFind))
		if err != nil { return false, scanned, err } // Check for EOF?
		if compBytes(bts, toFind) {
			wr.br.Discard(len(toFind))
			wr.curr += len(toFind)
			return true, scanned, nil
		}
		_, err = wr.br.Discard(1)
		if err != nil {
			return false, scanned, err
		}
		scanned++
		wr.curr++
	}
}


func NewWindowReader(reader io.Reader) *WindowReader {
	br := bufio.NewReader(reader)
	return &WindowReader{br: br, curr: 0}
}


Copied everything from main to use my new fancy WindowReader:

exif/reader.go


package exif

import (
	"encoding/binary"
	"fmt"
	"os"
)

const SizeDataLen = 2
const ExifStringDataLen = 4

var ExifMarker = []byte{0xFF,  0xE1}


func GetExifBytes(fname string) ([]byte, error) {
	fd, err := os.Open(fname) // open file read only
	if err != nil {
		return nil, err
	}
	defer fd.Close()

	wr := NewWindowReader(fd)
	found, scanned, err := wr.ScanUntil(ExifMarker)
	if err != nil { return nil, err }
	if !found {
		return nil, fmt.Errorf("could not read exif data from %s", fname)
	}
	sizeBytes, err := wr.NextWindow(SizeDataLen); if err != nil { return nil, err }
	totalExifSize := binary.BigEndian.Uint16(sizeBytes)
	fmt.Println("Exif size: ", totalExifSize)

	exifStringBytes, err := wr.NextWindow(ExifStringDataLen); if err != nil { return nil, err }
	exifString := forceASCII(exifStringBytes)
	fmt.Println("Exif string: ", exifString)
	if exifString != "Exif" {
		return nil, fmt.Errorf("Unexpected non exif identifier")
	}

	twoZeros, err := wr.NextWindow(2); if err != nil { return nil, err } // should be two zeroes
	if !compBytes(twoZeros, []byte{0, 0}) {
		return nil, fmt.Errorf("Couldn't find the two zeros after the exif string")
	}
	fmt.Printf("Two Zeros: %x\n", twoZeros)

	return nil, nil // TBD
}


Let’s try and adjust the code to handle different byte orders.


....

	companyIDBts, err := wr.NextWindow(2); if err != nil { return nil, err } // should tell us if it's MM or II
	fmt.Printf("CompanyID: %s\n", companyIDBts)
	companyID := forceASCII(companyIDBts)

	bom, err := wr.NextWindow(2); if err != nil { return nil, err } // either 0x2a00 or 0x002a 

	if companyID == "MM" && compBytes(bom, []byte{0x00, 0x2a}){
		fileEndian = binary.BigEndian
	} else if companyID == "II" && compBytes(bom, []byte{0x2a, 0x00}) {
		fileEndian = binary.LittleEndian
	} else {
		return nil, fmt.Errorf("Could not figure out file byte order")
	}

	totalExifSize := fileEndian.Uint16(sizeBytes)
	fmt.Println("Exif size: ", totalExifSize)

 	dataLenToRetrieve := int(totalExifSize) - wr.curr // total data includes the header

....

Alright! Markers, identifier, byte order.. All sorted! We now have a function that reads and returns the chunk of the file that holds our data.

Now we need to figure out how TIFF works. This document is a bit unclear about how exactly the data should be interpreted. There’s a table showing how the data is structured:

the image APP1 data structure

Next thing we have to do is to interpret the tiff header, which should give us the offset of IFD.

Figuring out IFD

After a long session of trial and error I finally understood what’s going on.

IFD or Image File Directory, holds the actual data (or keys and offsets to it)

Right after the Exif header there are 4 additional bytes that tell you the offset of the start of the IFD. That number is usually 8 since it starts after the Exif header.

Here’s our structure so far:

Jpeg marker and Exif header

jpeg markerExif markerExif sizeTwo 0
Bytes2242
Value0xFFD8 (const)0xFFE1 (const)Size of Exif data in bytes00 (const)

Then the Tiff header

Company IDByte Order (Endians)Offset to first IFD
Bytes224
ValueIntel or Motorolla (MM/II)0x2a00 or in reverse32 bit int

The offset to the first IFD as stated is probably 8.

The IFD

So here’s it gets a bit messy, but I’ll explain.
First let’s look at the IFD structure.

The first 2 bytes determine how many entries we have. | | Number of entries |
|– | :—: |
| Bytes | 2 |
| Value | 16 bit uint |

After that each entry is divided like that:

Tag IDData FormatNumber of components/unitsOffset/Data
Bytes2244
Valuehex (to be matched in a table)16 bit int32 bit int32 bit int or *Other

And now for the explanation.

The first 2 bytes of the IFD are representing the number of entries, so look at the entry table and imaging they appear one after the other without any breaks. For example, if ‘number of entries’ is interpreted to 3 - there will be 3 entry “rows”. Each row will be consisted of 12 bytes as you can easily calculate.

The entry fields are used as following:

  • Tag ID: Identifies the tag, there’s a table describing each tag and it’s description.
    • 0x0110 for example is ‘Model’
  • Data Format: Is an int between 1 and 12 determining the data type.
    • For example: 2 is ascii, 12 is double float. There is a table for that too.
  • Number Of components: The number of units of that data type.
    • For example: we have an ascii entry (data type no. 2) and number of components is 4 (We have 4 ascii units).
    • Since ascii has size of 2, means the size of our data is 8 bytes.
  • Offset/Data: This field has two options
    1. When the data size in bytes is less than 4, it holds the data itself.
    2. When the data size in bytes is more than 4, it holds the offset to the data from the jpeg marker.

After the entries there are 4 zero bytes.

Now we can add code to our function to read all that information (we stopped at total exif size).

We should add a necessary translation table first.


type DataFormat struct {
	Name string
	BtsPerComponents uint32
}

var DataFormats = map[uint16]DataFormat{
	1: {Name: "unsigned byte", BtsPerComponents: 1},
	2: {Name: "ascii strings", BtsPerComponents: 1},
	3: {Name: "unsigned short", BtsPerComponents: 2},
	4: {Name: "unsigned long", BtsPerComponents: 4},
	5: {Name: "unsigned rational", BtsPerComponents: 8},
	6: {Name: "signed byte", BtsPerComponents: 1},
	7: {Name: "undefined", BtsPerComponents: 1},
	8: {Name: "signed short", BtsPerComponents: 2},
	9: {Name: "signed long", BtsPerComponents: 4},
	10: {Name: "signed rational", BtsPerComponents: 8},
	11: {Name: "single float", BtsPerComponents: 4},
	12: {Name: "double float", BtsPerComponents: 8},
}


Then we can continue with the IFD number of entries.

...

	totalExifSize := fileEndian.Uint16(sizeBytes)
	fmt.Println("Exif size: ", totalExifSize)

	offsetToFirstBts, err := wr.NextWindow(4); if err != nil { return nil, err }
	fmt.Println("offset to first ifd: ", fileEndian.Uint32(offsetToFirstBts)) // is 8

	fmt.Println("Current offset: ", wr.curr)

	// Start IFD
	entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
	fmt.Println("Num of entries: ", fileEndian.Uint16(entryNumber))
	numOfEntries := fileEndian.Uint16(entryNumber)


...

Now that we have the number of entries we can use a for loop to iterate them.


	entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
	fmt.Println("Num of entries: ", fileEndian.Uint16(entryNumber))
	numOfEntries := fileEndian.Uint16(entryNumber)
	// For loop entries
	for i := 0; i < int(numOfEntries) ; i++ {
		// get Tag ID
		tagID, err :=  wr.NextWindow(2) ; if err != nil { return nil, err }
		fmt.Printf("tagID: %x\n", tagID)
		
		// Data Format
		dataFormatBts, err :=  wr.NextWindow(2) ; if err != nil { return nil, err }
		dataFormat := fileEndian.Uint16(dataFormatBts)
		fmt.Println("dataFormat: ", dataFormat)
		
		// Num of components
		NumOfComponentsBts, err :=  wr.NextWindow(4) ; if err != nil { return nil, err }
		NumOfComponents := fileEndian.Uint32(NumOfComponentsBts)
		fmt.Println("NumOfComponents: ", NumOfComponents)

		// Offset/Data
		offset, err :=  wr.NextWindow(4) ; if err != nil { return nil, err }

		// use our data formats table
		format := DataFormats[dataFormat]
		fmt.Println("Format type is: ", format.Name)

		// calculate the data size
		nextDataSize := NumOfComponents * format.BtsPerComponents

		if nextDataSize <= 4 { // in case the data is smaller than 4 bytes we can get it from here
			switch dataFormat {
			case 4:
				fmt.Printf("Value is: %d\n\n",  fileEndian.Uint16(offset))
			case 3:
				fmt.Printf("Value is: %d\n\n",  fileEndian.Uint16(offset))
			// and 10 more types....
			}
			continue // continue to the next entry
		}

		// read the data from the file descriptor 
		dataBts := make([]byte, nextDataSize)
		fd.ReadAt(dataBts, int64(fileEndian.Uint32(offset) + uint32(2))) // the offset, as stated in the offset field 
		                                                                 // plus 2 bytes for the jpeg marker 

		if format.Name == "ascii strings" { 
			fmt.Println("ascii: ", forceASCII(dataBts))
		}
		// and other types....


	}
	
	lastDirBytes, err := wr.NextWindow(4); if err != nil { return nil, err }
	fmt.Printf("lastDirbytes %x\n", lastDirBytes) // should be 4 zeros



Alright!

The code is a bit ugly and only partially functional, but I bet you have a basic understanding of how this things work. I for sure had fun messing around with this encoded data. I just love to see a bunch of numbers and letters, that almost seem random, come together into logical data structures.

But don’t worry, there’s still work to be done! I would still like to have a nice and efficient implementation, that can read Exif data. Let’s see what I can come up with…

Reorganizing the code

Instead of having a big chunk of code, I decided to break the decoding into functions. Started with decoding the header:

exif/header.go


package exif

import (
	"encoding/binary"
	"fmt"
)

type CompanyID string

const Intel CompanyID = "II"
const Motorolla CompanyID = "MM"

const EXIF = "Exif"

var BigEndianOrder = []byte{0x00, 0x2a}
var LittleEndianOrder = []byte{0x2a, 0x00}

type ExifHeader struct {
	Size uint16                   // 2 b
	ExifString string             // 4 b 6
	TwoZeros []byte				  // 2 b 8
	CompanyID CompanyID           // 2 b 10
	ByteOrder binary.ByteOrder    // 2 b 12
	Offset  uint32                // 4 b 16
}

// DecodeHeader accepts 16 bytes []byte and returns ExifHeader
func DecodeHeader(bts []byte) (ExifHeader, error) {
	header := ExifHeader{}
	if len(bts) != 16 {
		return header, fmt.Errorf("expecting header length of 18 but got %d", len(bts))
	}
	sizeBts := bts[:2]
	exifStrBts := bts[2:6]
	twoZeros := bts[6:8]
	companyIDBts := bts[8:10]
	bomBts := bts[10:12]
	offsetBts := bts[12:16]
	
	var decodesResults []error

	// Start with validations
	err := validateExifString(exifStrBts); if err != nil { goto onError }
	err = validateTwoZeros(twoZeros); if err != nil {  goto onError }
	// Continue on to determine byte order
	header.CompanyID =  CompanyID(forceASCII(companyIDBts))
	header.ByteOrder, err = determineByteOrder(header.CompanyID, bomBts); if err != nil {  goto onError }

	// Now we can decode all those numbers
	decodesResults = []error{
		decode(header.ByteOrder, sizeBts, &header.Size), 
		decode(header.ByteOrder, offsetBts, &header.Offset), 
	}
	for _, r := range decodesResults {
		if r != nil { 
			err = r
			goto onError 
		}
	}
	onError:
		if err != nil{
			return header, err
		}

	return header, nil

}


func determineByteOrder(cID CompanyID, bom []byte) (binary.ByteOrder, error) {

	if cID == Motorolla && compBytes(bom, BigEndianOrder){
		return binary.BigEndian, nil
	} else if cID == Intel && compBytes(bom, LittleEndianOrder) {
		return binary.LittleEndian, nil
	} 
	return nil, fmt.Errorf("Could not figure out file byte order")
}

func validateExifString(bts []byte) error {
	exifStr := forceASCII(bts)
	if exifStr != EXIF {
		return fmt.Errorf("Unexpected Exif identifier %s", exifStr)
	}
	return nil
}

func validateTwoZeros(bts []byte) error {
	if !compBytes(bts, []byte{0, 0}) {
		return fmt.Errorf("Expected 0x0000 but got %x", bts)
	}
	return nil
}



As you can probably see, I don’t really need all the fields in Header… But it helps me understand the byte structure so I keep it there. This makes it possible to take the header from the file and decode it. Note that I added a decode utility function (to utils.go), it looks like that:


func decode(bo binary.ByteOrder,b []byte, i interface{}) error {
	switch i.(type) {
	case *uint, *uint8, *uint16, *uint32, *uint64, *int, *int8, *int32, *int64, *float32, *float64:
		return binary.Read(bytes.NewReader(b), bo, i)
	case *string:
		asc := forceASCII(b)	
		conv := i.(*string)
		*conv = asc
	default:
		return fmt.Errorf("Unsupported type %T", i)
	}
	return nil
}

In my reader.go file I can now do:

....

	wr := NewWindowReader(fd)
	found, _, err := wr.ScanUntil(ExifMarker)
	if err != nil { return nil, err }
	if !found {
		return nil, fmt.Errorf("could not read exif data from %s", fname)
	}
	headerBts, err := wr.NextWindow(16); if err != nil { return nil, err }
	exifHeader, err := DecodeHeader(headerBts); if err != nil { return nil, err}

....

Already the code is more clear. Great!

Next, I added a decode for number of entries, even though it’s pretty small.

numentries.go

package exif

import "encoding/binary"


func DecodeNumEntries(byteOrder binary.ByteOrder, bts []byte) (int, error) {
	var entries uint16
	return int(entries), decode(byteOrder, bts, &entries)
}

And continue accordingly in our reader function:

....

	fileEndian = exifHeader.ByteOrder
	if int(exifHeader.Offset) > wr.curr {
		wr.NextWindow(int(exifHeader.Offset) - wr.curr)
	}

	// Start IFD
	entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
	numOfEntries, err  := DecodeNumEntries(fileEndian, entryNumber); if err != nil { return nil, err }

....

After that was done, all that was left is to decode the entry.

entry.go


package exif

import (
	"encoding/binary"
	"fmt"
	"io"
)

type Entry struct {
	Tag  string            // 2
	DataFormat uint16      // 2 4
	NumOfComponents uint32 // 4 8
	Offset uint32		   // 4 12
	offsetBts []byte
	byteOrder binary.ByteOrder
}

type DataFormat struct {
	Name string
	BtsPerComponents uint32
	ID uint16
}

func (ent Entry) GetDataFormat() DataFormat {
	return dataFormats[ent.DataFormat]
}

func (ent Entry) DataSize() int {
	df := ent.GetDataFormat()
	return int(ent.NumOfComponents * df.BtsPerComponents)
}

func (ent Entry) DecodeData(rd io.ReaderAt, i interface{}) error {
	dataSize := ent.DataSize()
	if dataSize <= 4 { // in case the data is smaller than 4 bytes we can get it from the offset bytes
		return decode(ent.byteOrder, ent.offsetBts, i)
	}
	dataBts := make([]byte, dataSize)
	_, err := rd.ReadAt(dataBts, int64(ent.Offset) + 12 ) // the offset, as stated in the offset field 
	if err != nil {											   // plus 12 bytes for the header
		return err
	}
	return decode(ent.byteOrder, dataBts, i)
}


const (
	UByte uint16 =  1
	AsciiString uint16 = 2
	UShort uint16 = 3
	ULong uint16 = 4
	Urational uint16 = 5
	Byte uint16 = 6
	Undefined uint16 = 7
	Short uint16 = 8
	Long uint16 = 9
	Rational uint16 = 10
	Float uint16 = 11
	DoubleFloat uint16 = 12
)


var dataFormats = map[uint16]DataFormat{
	1: {ID: 1, Name: "unsigned byte", BtsPerComponents: 1},
	2: {ID: 2, Name: "ascii strings", BtsPerComponents: 1},
	3: {ID: 3, Name: "unsigned short", BtsPerComponents: 2},
	4: {ID: 4, Name: "unsigned long", BtsPerComponents: 4},
	5: {ID: 5, Name: "unsigned rational", BtsPerComponents: 8},
	6: {ID: 6, Name: "signed byte", BtsPerComponents: 1},
	7: {ID: 7, Name: "undefined", BtsPerComponents: 1},
	8: {ID: 8, Name: "signed short", BtsPerComponents: 2},
	9: {ID: 9, Name: "signed long", BtsPerComponents: 4},
	10: {ID: 10, Name: "signed rational", BtsPerComponents: 8},
	11: {ID: 11, Name: "single float", BtsPerComponents: 4},
	12: {ID: 12, Name: "double float", BtsPerComponents: 8},
}


// DecodeEntry accepts a 12 byte array and returns an Entry
func DecodeEntry(bo binary.ByteOrder, bts []byte) (Entry, error) {
	entry := Entry{
		byteOrder: bo,
	}
	if len(bts) != 12 {
		return entry, fmt.Errorf("Expected a 12 bytes long slice but got %d", len(bts))
	}
	if bo == binary.BigEndian {
		entry.Tag = fmt.Sprintf("0x%x",string(bts[:2]))
	} else {
		entry.Tag = fmt.Sprintf("0x%x",string([]byte{bts[1], bts[0]})) // need to flip it
	}
	dataFormatBts := bts[2:4]
	numOFComponentsBts := bts[4:8]
	entry.offsetBts = bts[8:12]
	errs := []error {
		decode(bo, dataFormatBts, &entry.DataFormat),
		decode(bo, numOFComponentsBts, &entry.NumOfComponents),
		decode(bo, entry.offsetBts, &entry.Offset),
	}
	for _, err := range errs {
		if err != nil {
			return entry, err
		}
	}

	return entry, nil
}

Wooow! That’s a lot of code… So, I started by decoding the entry just like I did with the header. After that I did some grunt work and added the data format list and consts. And finally I added the DecodeData function.
By the way I’m not very sure about this:

	_, err := rd.ReadAt(dataBts, int64(ent.Offset) + 12 ) 

The header is actually 16 bytes. So the offset field is not very clear and the question to be asked is - “offset from where?", because it’s not from the beginning of the file or from the beginning of exif data…
Anyway, at the moment I can live with 12 being the magic number.

I also did some more grunt work for the tags (copied some from the doc, of course not all of them)

tags.go

package exif

type Tag struct {
	ID  string
	Name string
}

// names are in big endian
var Tags = map[string]Tag{
	"0x0100": {ID: "0x0100", Name: "ImageWidth"}, // ImageWidth has double tag
	"0x0001": {ID: "0x0001", Name: "ImageWidth"},
	"0x0101": {ID: "0x0101", Name: "ImageLength"},
	"0x9003": {ID: "0x9003", Name:"DateTimeOriginal"},
	"0x0132": {ID: "0x0132", Name:"DateTime"},
	"0x010f": {ID: "0x010f", Name: "Make"},
	"0x0110": {ID: "0x0110", Name: "Model"},
	"0x0112": {ID: "0x0112", Name: "Orientation"},
	"0x0131": {ID: "0x0131", Name: "Software"},
}


And now the final version of my reader function looks like that:

reader.go


package exif

import (
	"encoding/binary"
	"fmt"
	"os"
)

const SizeDataLen = 2
const ExifStringDataLen = 4

var ExifMarker = []byte{0xFF,  0xE1}
var fileEndian binary.ByteOrder



func GetExifData(fname string) (map[string]interface{}, error) {
	res := map[string]interface{}{}
	fd, err := os.Open(fname) // open file read only
	if err != nil {
		return nil, err
	}
	defer fd.Close()

	wr := NewWindowReader(fd)
	found, _, err := wr.ScanUntil(ExifMarker)
	if err != nil { return nil, err }
	if !found {
		return nil, fmt.Errorf("could not read exif data from %s", fname)
	}
	headerBts, err := wr.NextWindow(16); if err != nil { return nil, err }
	exifHeader, err := DecodeHeader(headerBts); if err != nil { return nil, err}
	fileEndian = exifHeader.ByteOrder
	if int(exifHeader.Offset) > wr.curr {
		wr.NextWindow(int(exifHeader.Offset) - wr.curr)
	}

	// Start IFD
	entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
	numOfEntries, err  := DecodeNumEntries(fileEndian, entryNumber); if err != nil { return nil, err }
	// For loop entries
	for i := 0; i < numOfEntries ; i++ {
		entryBts, err := wr.NextWindow(12); if err != nil { return nil, err}
		entry, err := DecodeEntry(fileEndian, entryBts); if err != nil { return nil, err}
		df := entry.GetDataFormat()
		// uint16, *uint32, *int64, *int, *int32, *float32, *float64
		tag, ok := Tags[entry.Tag] 
		if ok {
			var d interface{}
			switch df.ID {
			case UByte:
				var v uint8
				err = entry.DecodeData(fd, &v)
				d = v
			case AsciiString: 
				var v string
				err = entry.DecodeData(fd, &v)
				d = v
			case UShort:
				var v uint16
				err = entry.DecodeData(fd, &v)
				d = v
			case ULong:
				var v uint32
				err = entry.DecodeData(fd, &v)
				d = v
			case Urational:
				var v uint64
				err = entry.DecodeData(fd, &v)
				d = v
			case Byte:
				var v int8
				err = entry.DecodeData(fd, &v)
				d = v
			case Undefined:
				// leave it interface
			case Short:
				var v int16
				err = entry.DecodeData(fd, &v)
				d = v
			case Long:
				var v int32
				err = entry.DecodeData(fd, &v)
				d = v
			case Rational:
				var v int64
				err = entry.DecodeData(fd, &v)
				d = v
			case Float:
				var v float32
				err = entry.DecodeData(fd, &v)
				d = v
			case DoubleFloat:
				var v float64
				err = entry.DecodeData(fd, &v)
				d = v
			}
			if err != nil { return nil, err}
			res[tag.Name] = d
		}


	}
	
	// lastDirBytes, err := wr.NextWindow(4); if err != nil { return nil, err }
	// fmt.Printf("lastDirbytes %x\n", lastDirBytes) // should be 4 zeros	
	return res, nil
}



It’s fairly readable.. Would maybe move that ugly switch case, but I think we’re good for now.

Now my main function:

func main () {

	fname := "~/Pictures/11.10.17/20120349_235641.jpg"
	props, err := exif.GetExifData(fname)	
	if err != nil {
		fmt.Println("Error: ", err)
		return
	}
	for k, v := range props {
		fmt.Println(k,":   ", v)
	}
}

And the output is something like that:


$ go run main.go
Software :    J7800GUOPAPL1
DateTime :    2021:06:01 66:66:66
ImageWidth :    3264
ImageLength :    2448
Make :    xiaomi
Model :    UX-8866

That’s all for today! You can check out the complete solution at https://github.com/asafg6/exif-reader

Hope you enjoyed!