GIS – Uhurulabs

OpenDroneMap and Uhurulabs do Kerela

16th February 201817th February 2018
by Frederick Mbuya

Okay so I am finally writing this post. My initial target was to have it done by the end of January, but that slipped so next target was mid Feb, and that has slipped but better late than never!

Last year I together with Stephen Mather of OpenDroneMap were privileged to be invited to Kerala, India by International Centre for Free and Open Source Software (ICFOSS) to run a workshop on open source solutions for processing drone imagery and open hardware drone solutions. Checkout Stephen’s blog post here. As usual I was nervous, although I have been diving deep into the topics the past few years, I still get nervous when I am going to do a presentation or workshop.

Was a little hard leaving home, I don’t like leaving my girls for more than a couple of days, and even though this was only going to be 5 days, India felt like a long way away.

I left very early from home, but as usual the traffic in Dar es Salaam is totally unpredictably. By about half way it was clear I was not going to make it, so I started considering my options. Only one really, get a Bodaboda, or rather two Bodaboda’s, one for me and one for my luggage.

IMG_20180115_090923

IMG_20180115_090917

IMG_20180115_090727

Got to the airport in time and was able to board the flight to first Dubai and then India.

Arrived in India quite late at night around 4am, as usual I was a little concerned about getting through customs with all the gear but it was no problem at all, even with the big ebee case(link), and the tons of drone bits in my bags. Once outside the airport the person who was there to collect me seemed to recognize me immediately, (maybe he was told to look for the black guy with dreads!). He took me straight to the guest house which was on the Trivandrum Technopark campus, and within 20 mins I was in my room and about to get some rest, (by then it was 5am), when there was a knock on the door! I opened and there was a guy standing, telling me that they are ready to leave in 30 mins…

Although the organization that had invited us was based in Trivandrum Technopark, India’s Largest IT Park, the actual workshop was to be held at Vagamon some 190 km a hill station at least 5 hours drive.I have to admit I was a little rough saying “NO I have to get some rest”, he went away for a while and then came back to let me know it was okay, I would go later with another group! Nice, I could get some sleep.

Woke up around 10am, and of-course top priority was INTERNET! Unfortunately the guest house did not have internet, which seemed kinda strange as it was on a science park, so went off wondering to see if I could get a sim card. After wondering around for about an hour and talking to a number of people it seemed that you need an Indian ID to get a sim card, not so sure that was true, but figured I ought to get back to the guest house in-case my host was looking for me. Lucky I did as someone from my host turned up a few minutes later and we headed together to the ICFOSS offices.

I spent a couple of hours at the ICFOSS office, got online, checked in with my family back home and checked my emails etc. Then headed back to the guest house because the plan was to drive up into the mountains at around 17:00.

The drive was great, a little long, but it was so nice to see some of the Kerala countryside, arrived pretty late partly because we had stopped along the way to eat, FYI what people in Kerala consider junk food, many would consider a healthy snack!

DSC_0035

IMG_20180117_173947

DCIM100MEDIADJI_0417.JPG

IMG_20180118_130740

Once at the place we were spending the night Stephen and I spent some time planning the following day and then everyone got some much needed rest.

In the morning we headed to the place where we were going to hold the classroom part of the workshop. Before the rest of the attendees showed up we had some time to work on the Ranger, ahead of its maiden flight that was planned for the afternoon. We had good attendance, Arun M the lead from ICFOSS together with the core drone team, Deepthi, Vaisakh, and Asishwith, plus other ICFOSS staff, members of the Kerala electricity board, Veenus and Manoj from 1074 Vectors and others. All in all we had people from different sectors of government, startups and some more mature private sector companies.

IMG_20180117_100809

IMG_20180117_093523

IMG_20180117_105155

Stephen started off the workshop with introductions and then it was over to me to talk about some of the work we have been doing in Tanzania. For the most part I covered how drones had helped us with the mapping of several flood prone areas of Dar es Salaam in the Ramani Huria Project, and how the biggest project yet the mapping of Zanzibar with low cost drones, had gone on to win awards and really helped validate all the work we were doing. It was great to have representatives not only from government, but also from the private sector, all keen and interested in how drones could help their particular needs. I left them with some thoughts about how to engage with the relevant government stakeholders to ensure that not only were they building technical expertise, but also ensuring a regulatory framework that would encourage innovation and be supportive of local businesses. The afternoon was supposed to be a field session were the group would actually see some drones in action. One of the guys from ICFOSS and I went hunting for a good place for take-off and landing.

We were lucky and found a great spot, so we started with out field setup, we actually had quite a few drones with us…

Hobby King Duet. Small fixed wing, basically a park flier that can take a beating and keep flying making it perfect for training.
SenseFly Ebee, a great platform even if its proprietary.
DJI Mavic, easy to use consumer drone.
Homemade quad-copter build around a pixhawk.
3DR Solo
Ranger RX

DSC_0061

DSC_0060

DSC_0088

DSC_0086

DSC_0083

The aim for the afternoon session was to demonstration how both a multi-rotor, (3DR SOLO), and a fixed wing (Ebee), can be used for mapping highlighting the difference and when each is most appropriate. Best case we were hoping to do some actual mapping with both platforms so that we could have some data that could be used for the OpenDroneMap processing demonstrations. Everything went perfectly, both the ebee and the 3dr solo were used to gather some data, and we even had some time to play around with the Mavic and really show the group how far consumer drones had come.

DCIM100MEDIADJI_0427.JPG

DCIM100MEDIADJI_0419.JPG

DCIM100MEDIADJI_0425.JPG

DCIM100MEDIADJI_0428.JPG

DCIM100MEDIADJI_0424.JPG

DSC_0040

DSC_0045

DSC_0039

DSC_0051

You may be asking yourself why we had a SenseFly Ebee at a workshop that was about OpenSource and OpenHardware? As it stands I still heavily recommend someone who is exploring the use of UAVs for mapping and other survey activities to get an Ebee IF their budget allows. I say this because the Ebee is a platform that just works, it comes with both the flight planning software, processing software, (Pix4D) and an excellent support infrastructure that is critical when you are getting into this field. Things will go wrong, and it’s nice in the beginning to minimize the things that could go wrong, and to have an organization that provides excellent support available, should you need it. There are many times when I have been out in the field doing a client survey and I have had to call SenseFly support to help get on some technicality or the other. Another reason why I recommend the ebee especially in countries where regulations are still in their infancy if they exist at all, is because of the extremely low risk the low weight of the ebee poses to people or property in the case of an accident, although I would not advise it there is even a video of a Danish firm doing a “test”.

That evening we all huddled together around the table planning for the next day. I processed the data we had collected during the day, Stephen and Deepthi planned the flights they expected to do the following day, and even decided to use the elevation data from my flights to plan other flights what we would do the following day. This was a great opportunity to show how valuable data generated from a quick drone survey could be, after just 2 hours in the field we already had elevation data that was more accurate than we could get from Google.

The following day we planned to do some quick flights in the morning and then head back to the city so that on the last day we could do some classroom data processing work. The flights went almost too perfectly, in my opinion you have not flown till you have crashed. The last flight of the 3DR Solo was done by Deepthi, and unfortunately, (or fortunately), there was a small issue during landing which broke 2 props, no huge issue and props are “consumables” anyway! This was also the last chance to test the , the area was not perfect, but we gave it a try anyway, Stephen on the controls, (the only one with real manual fixed wing experience), fact is though the runway was just not long enough and he had to abort at the last moment, one broken prop but the UAV still in perfect condition.

DSC_0052

DSC_0053

DSC_0054

DSC_0087

DSC_0086

DSC_0085

DSC_0084

DSC_0083

DSC_0053

Friday was the last day of the workshop held at the ICFOSS offices. Usually such workshops start with lots of time being wasted installing software and actually getting the environment ready. Not at ICFOSS, they had everything ready, a server already setup with OpenDroneMap, and a room setup with laptops for the participants. I wish every workshop I was involved with went this well. I won’t go into detail but some highlights were:

1) Everyone was able to connect to the ODM server and create their account without issue.
2) Everyone after a little bit of instruction was able to upload their data-set, and get the processing done.

But that was not the most impressive thing, while Stephen was driving the workshop I was preparing a 3D model of the work that I had made from the data using Pix4D. I have to admit was I was quite looking forward to showing it off, until I turned around and saw something very similar on one of the participants screen. I wandered over to ask what it was and it turns out, it was the 3D mesh from the work he had just done with OpenDroneMap, being rendered on windows using meshlab. I was humbled, excited and impressed all at once.

We closed the workshop, with clear plans on how we would like to move forward, and exciting possibilities of future collaboration, and a very strong feeling that there is so much we can learn from each-other. That evening we had a great tour of Kerala with Arun and Jucar, which ended with a great meal on the beach. All in all I really enjoyed this trip, and looking forward to my next, (hopefully with my family this time), and to the great collaborations that I hope will come from it between Uhurulabs, OpenDroneMap and ICFOSS.

IMG_20180119_193218

IMG_20180119_202635

GeoTIFF Transparency

31st March 20172nd January 2019
by Frederick Mbuya

I have spent way too much time messing with this, and certainly too many CPU cycles processing only to get the wrong output. The problem…

“Two irregular shaped geotiffs that you need to merge into one and not have any stupid gaps”

My first attempts resulted in things like…

Not the desired output, but the fix was easy really, first fix the nodata value with

gdal_translate -a_nodata 0 \
-of GTiff \
file1.tif \
file1_nodata0.tif

gdal_translate -a_nodata 0 \
-of GTiff \
file2.tif \
file2_nodata0.tif

Then merge the two

gdal_merge.py -n 0 \
-a_nodata 0 \
file1_nodata0.tif file2_nodata0.tif \
-o merged_file

That results in a nicely merged GeoTIFF

Working with GeoTIFF’s from Pix4D

28th March 20172nd January 2019
by Frederick Mbuya

We have been doing a lot of work with drones over the past couple of years. Much of it proof of concept for the use of small, under 1Kg drones to capture aerial imagery as alternative to both manned aircraft and satellite, the premise being deploying small drones is much easier and cheaper than the alternative and that the acquired data would be of equal if not superior quality. For the most part we have used senseFly ebee drones, and done our post processing work in Pix4D Mapper, both being professional grade survey hardware and software. There are many possible outputs, but one of the main ones is a Orthophoto, according to Wikipedia

An orthophoto, orthophotograph or orthoimage is an aerial photograph or image geometrically corrected (“orthorectified”) such that the scale is uniform: the photo has the same lack of distortion as a map. Unlike an uncorrected aerial photograph, an orthophotograph can be used to measure true distances, because it is an accurate representation of the Earth’s surface, having been adjusted for topographic relief,^[1] lens distortion, and camera tilt.

The format of the orthophoto as output from Pix4D is GeoTiff and using gdalinfo we can extract the following info:

Driver: GTiff/GeoTIFF
Files: oysterbay.tif
Size is 34565, 37051
Coordinate System is:
PROJCS["WGS 84 / UTM zone 37S",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",39],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","32737"]]
Origin = (529846.625330000068061,9252232.345700001344085)
Pixel Size = (0.049060000000000,-0.049060000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=pix4dmapper
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  529846.625, 9252232.346) ( 39d16'12.33"E,  6d45'53.64"S)
Lower Left  (  529846.625, 9250414.624) ( 39d16'12.36"E,  6d46'52.83"S)
Upper Right (  531542.384, 9252232.346) ( 39d17' 7.57"E,  6d45'53.60"S)
Lower Right (  531542.384, 9250414.624) ( 39d17' 7.61"E,  6d46'52.80"S)
Center      (  530694.505, 9251323.485) ( 39d16'39.97"E,  6d46'23.22"S)
Band 1 Block=34565x1 Type=Byte, ColorInterp=Red
  NoData Value=-10000
Band 2 Block=34565x1 Type=Byte, ColorInterp=Green
  NoData Value=-10000
Band 3 Block=34565x1 Type=Byte, ColorInterp=Blue
  NoData Value=-10000
Band 4 Block=34565x1 Type=Byte, ColorInterp=Alpha
  NoData Value=-10000

This is for a sample 2 Gigabyte GeoTIFF. Now this might not sound too big but this is for an area of about 1.5sq/km. As you can imagine when you’re doing projects that are in the 100’s if not 1000’s of sq/km you will quickly need huge amounts of storage space. And that is not the only problem, as far as I know the most popular open source platform for hosting such data is currently a combination of geonode and geoserver, however after some experimentation I found that serving these images as is was simply not an option it was taking the geonode too long to render them, resulting in an awful user experience.

I looked into many options, and one of the best seemed to be mbtiles, however in the end it was not a suitable .. blog for another day…

After much research online it seemed like the best option was to first compress the GeoTIFF with gdal using gdal_translate. So far the best options I have been able to find are:

“-b 1 -b 2 -b 3”, which specifies that we only want the first three bands, if you refer to the gdalinfo output above you will see that for some reason Pix4D includes an alpha band. We don’t need it, I think! and its critical to only have the three bands or the PHOTOMETRIC flag won’t work. The next option is

“-a_nodata”, which sets the nodata, (transparency), value to 0 allowing the generated file to have transparency.

COMPRESS=JPEG is quite self-explanatory this is the instruction to gdal to use JPEG compression. This is better than the

PHOTOMETRIC=YCBCR, this changes the used color space from RGB to YCbCr, from what I have been able to find out. (thanks google), the eye is more sensitive to changes in luminance (Y, brightness) than to changes in chroma (Cb, Cr, color). Thus, it is possible to erase some chroma information while retaining image quality, thus allowing for better compression without any visible loss in image quality.

TILED=Yes again is quite self explanatory, it stores the image data in a tiled format which makes for a much better user experience when viewing the data using something like geonode or QGIS.

time gdal_translate \
-b 1 -b 2 -b 3 \
-a_nodata 0 \
-co COMPRESS=JPEG \
-co PHOTOMETRIC=YCBCR \
-co TILED=YES \
sample.tif \
sample_JPEG_YCBCR

After this the we have two files

151M Mar 28 17:54 oysterbay_JPEG_YCBCR.tif
2.0G Mar  2 07:49 oysterbay.tif

As you can see this has reduced our file from 2Gto 151M this is a 13 fold reduction is size, it took my workstation 39 seconds to do the compression, you can see details of my workstation here.

The next step is to add some zoom scales to the image, this will result in a slight increase in the file size but will result in a much better user experience when zooming into and out of the image.

time gdaladdo \
--config COMPRESS_OVERVIEW JPEG \
--config PHOTOMETRIC_OVERVIEW YCBCR \
--config INTERLEAVE_OVERVIEW PIXEL \
-r average oysterbay_JPEG_YCBCR.tif 2 4 8 16

This on my workstation took 28 seconds and added 50M to the file resulting in

220M Mar 28 18:29 oysterbay_JPEG_YCBCR.tif

And if we again check the file info using gdalinfo we get

Driver: GTiff/GeoTIFF
Files: oysterbay_JPEG_YCBCR.tif
Size is 34565, 37051
Coordinate System is:
PROJCS["WGS 84 / UTM zone 37S",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",39],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","32737"]]
Origin = (529846.625330000068061,9252232.345700001344085)
Pixel Size = (0.049060000000000,-0.049060000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=pix4dmapper
Image Structure Metadata:
  COMPRESSION=YCbCr JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (  529846.625, 9252232.346) ( 39d16'12.33"E,  6d45'53.64"S)
Lower Left  (  529846.625, 9250414.624) ( 39d16'12.36"E,  6d46'52.83"S)
Upper Right (  531542.384, 9252232.346) ( 39d17' 7.57"E,  6d45'53.60"S)
Lower Right (  531542.384, 9250414.624) ( 39d17' 7.61"E,  6d46'52.80"S)
Center      (  530694.505, 9251323.485) ( 39d16'39.97"E,  6d46'23.22"S)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  NoData Value=0
  Overviews: 17283x18526, 8642x9263, 4321x4632, 2161x2316
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  NoData Value=0
  Overviews: 17283x18526, 8642x9263, 4321x4632, 2161x2316
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  NoData Value=0
  Overviews: 17283x18526, 8642x9263, 4321x4632, 2161x2316

Note the GeoTIIFF is now compressed using YCbCr JPEG, and uses the YCbCr color space.
Bellow on the left is the section of the original Orthophoto and the right the compressed version, can you see a difference?